Multi-source heterogeneous data fusion method and system, storage medium and equipment

Abstract

The invention provides a multi-source heterogeneous data fusion method and system, a storage medium and equipment. The method comprises the steps of obtaining multi-source heterogeneous data, and executing a specified theme library fusion rule through a corresponding theme library fusion engine to obtain theme library data; and executing a specified thematic library fusion rule on the thematic library data through a corresponding thematic library fusion engine to obtain the thematic library data. According to the multi-source heterogeneous data fusion method, the data processing efficiency and the analysis accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a multi-source heterogeneous data fusion method, system, storage medium and device. Background Art

[0002] As the global demand for air travel continues to grow, the number of passengers carried by airports has increased dramatically. The core business of airports covers the entire process from passenger ticket booking to final boarding, including but not limited to: integrated information system, flight reservation system, departure control system, data backup system, passenger security inspection system, boarding gate management system, automatic baggage sorting system, cargo loading system, and specific host systems serving foreign airlines and low-cost airlines. Each system operates independently, generating massive and diverse data, forming a typical multi-source heterogeneous data environment. Due to the uniqueness of the data in each business system, it is not only reflected in the data format (such as structured database records and unstructured image and video information, etc.), but also in the differences in data models and storage methods. Therefore, in the process of ensuring smooth travel for passengers, there are huge challenges in the accurate management and efficient use of passenger and baggage data.

[0003] Due to the complexity and variety of actual application scenarios, sometimes only a single data source is needed to complete a specific task, while in other cases, it is necessary to comprehensively analyze the information from multiple data sources to achieve deep integration and application of data to support efficient decision-making and optimize service processes. However, due to the diversity of data sources, inconsistency of formats, and lack of effective intercommunication between systems, traditional data processing methods are difficult to meet the requirements of airport business for data immediacy, accuracy, and comprehensiveness. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a multi-source heterogeneous data fusion method, system, storage medium and device. Based on fusion rules, the acquired multi-source heterogeneous data is subjected to the fusion of subject libraries and special library in stages through a designated fusion engine, which can realize the full-chain automated processing from raw data to high-quality, structured information, and then to customized special data analysis, thereby improving data processing efficiency and analysis accuracy.

[0005] The present invention is achieved through the following technical solutions:

[0006] Obtain multi-source heterogeneous data, execute the specified theme library fusion rules through the corresponding theme library fusion engine, and obtain theme library data;

[0007] The specified thematic library fusion rules are executed on the subject library data through the corresponding thematic library fusion engine to obtain the thematic library data.

[0008] Optional,

[0009] The subject library fusion rules include: cleaning rules, structured processing rules and subject library priority fusion rules; and / or,

[0010] The subject library fusion engine includes: a data cleaning engine, a structured processing engine and a subject library priority fusion engine.

[0011] Optional,

[0012] The method of acquiring multi-source heterogeneous data and executing a specified theme library fusion rule through a corresponding theme library fusion engine to obtain theme library data includes:

[0013] Setting data cleaning rules, and executing the data cleaning rules on the multi-source heterogeneous data through a data cleaning engine;

[0014] Set structured processing rules and convert the cleaned data into corresponding entity structures through the structured processing engine;

[0015] Set the theme library priority fusion rules, build a data unique identification system through the theme library priority fusion engine, and obtain the theme library data.

[0016] Optional,

[0017] The structured processing engine comprises:

[0018] Logical units, execution core, business process operations, and data access layers.

[0019] Optional,

[0020] The thematic library fusion rules include: data fusion rules and thematic library priority fusion rules;

[0021] The thematic library fusion engine includes: a data fusion engine and a thematic library priority fusion engine.

[0022] Optional,

[0023] The data fusion engine comprises:

[0024] Data projection engine, data connection engine, data selection engine and data aggregation engine.

[0025] Optional,

[0026] The method further comprises:

[0027] The data fusion engine operates on the subject database data according to the data fusion rules to generate business tables that meet specific analysis or reporting requirements;

[0028] The thematic library priority fusion engine integrates data according to the thematic library priority rules to generate thematic library data.

[0029] The present invention also provides a multi-source heterogeneous data fusion system for implementing the above method, the system comprising:

[0030] The theme library fusion module is used to obtain multi-source heterogeneous data, execute the specified theme library fusion rules through the corresponding theme library fusion engine, and obtain the theme library data;

[0031] The subject library fusion module is used to execute the specified subject library fusion rules on the subject library data through the corresponding subject library fusion engine to obtain the subject library data.

[0032] Optional,

[0033] The subject library fusion rules include: cleaning rules, structured processing rules and subject library priority fusion rules; and / or,

[0034] The subject library fusion engine includes: a data cleaning engine, a structured processing engine and a subject library priority fusion engine.

[0035] Optional,

[0036] Setting data cleaning rules, and executing the data cleaning rules on the multi-source heterogeneous data through a data cleaning engine;

[0037] Set structured processing rules and convert the cleaned data into corresponding entity structures through the structured processing engine;

[0038] Set the theme library priority fusion rules, build a data unique identification system through the theme library priority fusion engine, and obtain the theme library data.

[0039] Optional,

[0040] The thematic library fusion rules include: data fusion rules and thematic library priority fusion rules; and / or,

[0041] The thematic library fusion engine includes: a data fusion engine and a thematic library priority fusion engine.

[0042] Optional,

[0043] The data fusion engine operates on the subject database data according to the data fusion rules to generate business tables that meet specific analysis or reporting requirements;

[0044] The thematic library priority fusion engine integrates data according to the thematic library priority rules to generate thematic library data.

[0045] The present invention also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned multi-source heterogeneous data fusion method.

[0046] The present invention also provides a device, comprising a processor, a communication interface, a computer-readable storage medium and a communication bus; wherein the processor, the communication interface and the computer-readable storage medium communicate with each other via the communication bus;

[0047] The processor is configured to execute a program stored in a computer-readable storage medium.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. The multi-source heterogeneous data fusion method proposed in the present invention implements the fusion of subject libraries and special library in stages for the acquired multi-source heterogeneous data through a designated fusion engine according to data fusion rules, which can realize the full-chain automated processing from raw data to high-quality, structured information, and then to customized special data analysis, thereby improving data processing efficiency and analysis accuracy.

[0050] 2. The specified theme library fusion rules are executed through the theme library fusion engine to perform data cleaning, structural processing and theme library priority fusion on the acquired multi-source heterogeneous data. It can process data entries that do not comply with the rules, ensure the quality of subsequent processed data, and improve the accuracy and effectiveness of the fusion results.

[0051] 3. By executing the specified thematic library fusion rules through the thematic library fusion engine, business tables that meet specific analysis or reporting needs can be generated, providing data service capabilities to the outside world, realizing the thematic application of data, meeting data needs in different business scenarios, and improving data analysis accuracy.

[0052] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A schematic diagram of the process of multi-source heterogeneous data fusion method is shown;

[0055] Figure 2 A schematic structural block diagram of a multi-source heterogeneous data fusion system is shown;

[0056] Figure 3 A schematic diagram of a multi-source heterogeneous fusion method based on a rule engine according to an embodiment of the present invention is shown;

[0057] Figure 4 A schematic diagram of a structured processing engine message receiving process according to an embodiment of the present invention is shown;

[0058] Figure 5 A schematic diagram of a structured processing engine message parsing process according to an embodiment of the present invention is shown;

[0059] Figure 6 It is a structural schematic diagram of a device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] See attached Figure 1 , the method of the present invention comprises:

[0062] S1. Obtain multi-source heterogeneous data, execute the specified theme library fusion rules through the corresponding theme library fusion engine, and obtain theme library data;

[0063] The subject library fusion rules include: cleaning rules, structured processing rules and subject library priority fusion rules; and / or,

[0064] The theme library fusion engine includes: a data cleaning engine, a structured processing engine and a theme library priority fusion engine.

[0065] wherein, setting data cleaning rules, and executing the data cleaning rules on the multi-source heterogeneous data through a data cleaning engine;

[0066] Set structured processing rules and convert the cleaned data into corresponding entity structures through the structured processing engine;

[0067] Set the theme library priority fusion rules, build a data unique identification system through the theme library priority fusion engine, and obtain the theme library data.

[0068] Wherein, the structured processing engine includes:

[0069] Logical units, execution core, business process operations, and data access layers.

[0070] S2. Execute the specified thematic library fusion rule on the subject library data through the corresponding thematic library fusion engine to obtain the thematic library data.

[0071] The thematic library fusion rules include: data fusion rules and thematic library priority fusion rules;

[0072] The thematic library fusion engine includes: a data fusion engine and a thematic library priority fusion engine.

[0073] Wherein, the data fusion engine includes:

[0074] Data projection engine, data connection engine, data selection engine and data aggregation engine,

[0075] Among them, the data fusion engine operates on the subject library data according to the data fusion rules to generate business tables that meet specific analysis or reporting requirements;

[0076] The thematic library priority fusion engine integrates data according to the thematic library priority rules to generate thematic library data.

[0077] Specifically,

[0078] The present invention proposes a multi-source heterogeneous fusion method based on a rule engine, comprising:

[0079] 1. Perform theme library fusion on the acquired multi-source heterogeneous data to obtain theme library data.

[0080] The subject library fusion rules include: cleaning rules, structured processing rules and subject library priority fusion rules.

[0081] The theme library fusion engine includes: data cleaning engine, structured processing engine and theme library priority fusion engine.

[0082] Obtain multi-source heterogeneous data, execute the specified theme library fusion rules through the corresponding theme library fusion engine, and obtain theme library data, including:

[0083] Setting data cleaning rules, and executing the data cleaning rules on the multi-source heterogeneous data through a data cleaning engine;

[0084] Set structured processing rules and convert the cleaned data into corresponding entity structures through the structured processing engine;

[0085] Set the data source fusion priority rules for each field in the subject library, build a data unique identification system through the subject library priority fusion engine, and obtain the subject library data.

[0086] 1. Data cleaning.

[0087] Set data cleaning rules, execute them through the data cleaning engine, identify and process data entries that do not conform to the rules. Improve data accuracy and reliability through operations such as deduplication, filling in default values, processing outliers, data standardization, data conversion, and data verification, ensure the quality of data for subsequent processing, and lay a solid foundation for subsequent data fusion.

[0088] 2. Structured processing.

[0089] After data cleaning, the cleaned data is structured. By setting structured processing rules, the structured processing engine parses the rules, identifies data patterns, and converts the cleaned data into a preset entity structure, including: receiving and storing original messages, message flow processing, and message parsing and preprocessing.

[0090] In this embodiment, a general framework for message processing for implementing structured processing is developed for the data types involved in the airport bus. The framework realizes the separation of the core layer and the business layer, which not only improves development efficiency and reduces risks, but also supports new businesses more efficiently and robustly. At the same time, the framework does not rely on a specific software platform and is highly versatile.

[0091] In this embodiment, the general framework for message processing includes a structured processing engine. The overall structure of the structured processing engine includes: a logic unit, an execution core, a business process operation, and a data access layer. Among them, the logic unit is responsible for the logic control of the entire framework, the execution core is the core to perform data processing tasks, the business process operation includes all business-related logic process operations, and the data access layer provides a unified access interface to data.

[0092] The general framework for message processing also includes general process processing for message reception and message parsing.

[0093] Among them, the general process of data message reception includes: message reception, pre-storage, pre-processing and storage of pre-processed messages;

[0094] The general process of data message parsing includes: obtaining the message to be parsed, message distribution, serial / parallel processing, parser parsing and data storage.

[0095] 3. Priority fusion.

[0096] After data structuring, the structured data is prioritized for fusion. Set the priority processing rules for the subject library, and use the subject library priority fusion engine to integrate data from different sources according to the set priority and logic to solve the problem of data inconsistency. By setting the business priority of the fusion of data sources in each field of the subject library and building a unique data identification system, it can be ensured that in the data fusion process, data sources with higher importance can be given priority, thereby ensuring the accuracy and effectiveness of the fusion results.

[0097] 2. Fusing the subject database data with the special database data to obtain the special database data.

[0098] The specified thematic library fusion rules are executed on the subject library data through the corresponding thematic library fusion engine to obtain the thematic library data.

[0099] Thematic library fusion rules include: data fusion rules and thematic library priority fusion rules.

[0100] The thematic database fusion engine includes: data fusion engine and thematic database priority fusion engine;

[0101] Among them, the data fusion engine includes: data projection engine, data connection engine, data selection engine and data aggregation engine.

[0102] The data fusion engine operates on the subject library data according to the data fusion rules to generate business tables that meet specific analysis or reporting requirements.

[0103] In this embodiment, for different business scenarios, according to the data fusion rules, the corresponding data fusion engine performs operations such as projection, selection, connection, aggregation, etc. on the data tables of the subject library to form special business tables. These business tables provide data service capabilities externally through query screening, realize thematic application of data, and meet the data needs in different business scenarios.

[0104] The thematic library priority fusion engine intelligently resolves multi-source data conflicts based on thematic library priority rules, ensures the consistency and accuracy of information, and integrates data according to business priority logic.

[0105] In this embodiment, the subject library priority fusion rule base defines the corresponding business priority rules for merging data from the theme library to the subject library, identifies the data association relationship of each subject library involved in the subject library according to the subject library unique identification item, and determines the subject library data unique identification.

[0106] The following is a detailed description with reference to specific embodiments.

[0107] This embodiment is a multi-source heterogeneous fusion method based on a rule engine in a civil aviation airport application scenario, taking passenger and baggage data as an example.

[0108] The platform aggregates passenger and baggage data from multiple sources, including:

[0109] (1) TravelSky PNR data

[0110] (2) TravelSky ET data

[0111] (3) Passenger check-in data and passenger outbound baggage data of the TravelSky departure system

[0112] (4) Passenger boarding pass data at the airport check-in front end

[0113] (5) In flight backup mode, the airport's local backup system passenger check-in data

[0114] (6) Security inspection system passenger security inspection data

[0115] (7) Boarding system passenger boarding data

[0116] (8) Passenger statistics of the loading system

[0117] (9) Passenger check-in and boarding data of foreign airlines and low-cost airlines, and passenger outbound baggage data

[0118] (10) Passenger inbound baggage data on the AirEasy platform

[0119] (11) Baggage location data of baggage sorting system

[0120] (12) Baggage reconfirmation system baggage location data

[0121] The above-mentioned multi-source heterogeneous data is aggregated and fused into a theme library. According to the theme library fusion rules, the corresponding theme library fusion engine is used to perform data cleaning, structural processing and priority fusion, and multiple theme data tables (i.e. theme library data) can be obtained:

[0122] (1) Passenger Reservation Table

[0123] (2) Passenger Check-in Form

[0124] (3) Passenger Security Checklist

[0125] (4) Passenger boarding list

[0126] (5) Flight passenger statistics

[0127] (6) Baggage location table

[0128] (7) Baggage security checklist

[0129] Data cleaning: Set data cleaning rules. For example, the PNR data of TravelSky is integrated into the passenger reservation table. The current business scenarios of the airport only focus on passengers with issued tickets. Therefore, the data with empty ticket numbers is set in the data cleaning rules not to be integrated. The data cleaning engine will discard the data with empty ticket numbers according to the rules.

[0130] Structured processing: Set structured processing rules to convert the cleaned data into corresponding entity structures. In this embodiment, a general framework for message processing of data types involved in the airport bus is used for structured processing, including:

[0131] 1) Receive and store original messages: Receive data messages from the outside (may include flight information, passenger information, baggage information, etc.), and encapsulate the received messages into an entity (Entity), which includes all the contents of the original message, which is conducive to maintaining the originality and integrity of the data and facilitates subsequent problem tracking and data analysis; the original Entity is stored as an original message table in a centralized data storage system (such as Datastore or database), which provides a reliable data source for the entire system as a basic layer.

[0132] 2) Message flow processing: Extract the original message from the data storage system and perform business flow processing. The message is classified according to its content and stored in the corresponding business table. For example, passenger information enters the passenger business table, flight information enters the flight business table, and baggage information enters the baggage business table. This classified storage helps data query efficiency and targeted business processing.

[0133] 3) Message parsing and preprocessing: Extract messages from the business table, select a suitable parser to perform in-depth parsing on the message based on the main and sub-types of the message (i.e., the structural characteristics of the message), and convert it into a preset data structure; while parsing, store the preprocessed data again in different business tables of the data to prepare for subsequent business logic processing or analysis.

[0134] Topic library priority fusion: In order to adapt to the characteristics of civil aviation business, the priority strategy of the subject library field data source fusion can be set to deal with the problem of multi-source data conflict.

[0135] 1) Set data fusion priority rules: Design a dynamic priority rule for the emergency backup mechanism of flight failures in China Aviation Information Network. In the standard operation mode, limited use of departure system data fusion is adopted; when switching to the backup mode, the airport local backup system data is used as the primary fusion source to ensure information consistency and avoid conflicts such as boarding number and baggage information.

[0136] 2) Build a unique data identification system: Implement a unique confirmation solution based on multi-source data for the integration of passenger and baggage information, including:

[0137] Passenger booking data: Integrate the ET and PNR data of TravelSky, and use "flight date + flight number + departure station + ticket number + PNR" as the unique identifier.

[0138] Passenger check-in information: Taking into account the departure data of TravelSky, local backup system and other sources, the unique check-in passenger identification is determined by "flight date + flight number + departure station + host number + ticket number".

[0139] Passenger security check records: Based on the security check system data, "flight date + flight number + departure station + boarding number" is used as the unique identifier of the security check passenger.

[0140] Passenger boarding management: Summarize the boarding system and foreign airline data, and use "flight date + flight number + departure station + host number + boarding sequence number" as the unique identification of boarding passengers.

[0141] The subject database data obtained through the above subject database fusion is further fused with the special database.

[0142] The thematic library fusion rule base includes: data fusion rule base and priority fusion rule base.

[0143] The data fusion rule base projects, selects, connects, and aggregates the data tables of the subject database for different business scenarios to form specialized business tables. The subject database data fusion rule base corresponds to four engine libraries, namely, data projection engine, data connection engine, data selection engine, and data aggregation engine:

[0144] Selection is to filter data by giving certain data in the data table. For example, in the passenger statistics table of the subject database, select all data with reservation status = ticketed in the passenger reservation table of the subject database;

[0145] Projection is to select data of certain attributes in the table. For example, the passenger security check status field in the passenger security check table of the subject database is projected into the passenger statistical data table of the subject database. The result is security check or not security check.

[0146] Connections are divided into various types of natural connections, external connections, internal connections, etc. Connections are mainly used for data queries between multiple tables;

[0147] Aggregation is to compress rows and merge attributes of subject database data according to the different dimension granularity, indicators and calculation elements.

[0148] The thematic database priority fusion rule base defines the corresponding business priority rules for merging data from the subject database to the thematic database, for example:

[0149] (1) Passenger statistics table: Provides the airport collaborative decision-making system with real-time passenger booking, check-in, security check, boarding, checked-in but not security check, and security check but not boarding statistics.

[0150] (2) Passenger and baggage check-in and security check data table: Provides passenger check-in, security check, and photo data as well as baggage security check and photo data to the boarding reconfirmation system, supports boarding passenger security check and baggage security check reconfirmation services including passengers who cancel or change their tickets in the quarantine area; provides passenger check-in, security check, and photo data to the boarding gate, supports facial recognition boarding.

[0151] In the process of subject database data being integrated into subject database data through thematic database, it is necessary to identify the association relationship of each subject database data involved in the subject database according to the unique identification item of the subject database, and finally determine the unique identification of the subject database data. Take the passenger statistical data table as an example: the passenger statistical data table identifies the unique identification of the data in the four subject database passenger tables of booking, check-in, security check, and boarding, and finally extracts the associated data items, with the flight date + flight number + departure station + ticket number + host number + boarding sequence number as the unique identification of the passenger statistical data table.

[0152] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0153] See attached Figure 2 , shows the structure of a multi-source heterogeneous data fusion system for implementing the above method, including a subject library fusion module and a special library fusion module.

[0154] The theme library fusion module is used to obtain multi-source heterogeneous data, and execute the specified theme library fusion rules through the corresponding theme library fusion engine to obtain theme library data;

[0155] The thematic library fusion module is used to execute a specified thematic library fusion rule on the subject library data through a corresponding thematic library fusion engine to obtain the thematic library data.

[0156] Figure 3 It is a schematic diagram of a multi-source heterogeneous fusion method based on a rule engine according to an embodiment of the present invention, which is mainly composed of a theme library fusion rule base, a special library fusion rule base, a theme library fusion engine and a special library fusion engine.

[0157] Theme library fusion rule base: It includes three stages: cleaning rule base, structured processing rule base, and theme library priority fusion rule base. Each stage of the rule base has a corresponding engine to parse and implement the rule base.

[0158] In this embodiment, data cleaning rules are set according to the business scenarios of the airport, and the data cleaning engine automatically executes the cleaning rules to identify and process data entries that do not comply with the rules; structured processing rules are set, and the structured processing engine parses the rules, identifies data patterns, and converts them into predefined entity structures, such as tables or JSON; theme library priority processing rules are set, and the theme library priority fusion engine integrates data from different sources according to the set priority and logic to solve data inconsistency problems.

[0159] Thematic library fusion rule base: includes data fusion rule base and thematic library priority fusion rule base. In this embodiment, the data fusion engine operates on thematic library data according to the fusion rules to generate business tables that meet specific analysis or reporting requirements; thematic library priority fusion engine uses priority rules to intelligently resolve multi-source data conflicts, ensure the consistency and accuracy of information, and integrate data according to business priority logic.

[0160] Figure 4 The schematic diagram of the message receiving process of the structured processing engine of the embodiment of the present invention is as follows. The general process of receiving data messages includes: message receiving, pre-storage, pre-processing and storing the pre-processed messages in the storage.

[0161] Receive data messages from the outside, encapsulate them into original message tables, and store them in the data storage layer;

[0162] Perform pre-storage and put it into the data storage layer;

[0163] The data is retrieved for preprocessing and stored in the data storage layer; wherein the preprocessing includes parsing the data through a message parser according to the message main type-subtype;

[0164] The data is called for processing at the business layer, and after passing through the persistence layer, the data is stored in the database (DB).

[0165] Figure 5 This is a schematic diagram of the message parsing process of the structured processing engine of an embodiment of the present invention. The general process of data message parsing includes: obtaining messages to be parsed, message distribution, serial / parallel processing, parser parsing and data storage.

[0166] Obtain the message to be parsed from the database, process it through the persistence layer and business layer, and store it in the data storage layer;

[0167] Extract data messages for message distribution and determine whether to process them serially: if they are processed serially, put them into the serial data storage layer for serial processing; if they are not processed serially, put them into the parallel data storage layer for parallel processing;

[0168] The messages after serial / parallel processing are parsed by the message parser according to the message main type-subtype;

[0169] The messages after serial / parallel processing are stored in the data storage service, and can be called by the business processing service, and stored in the database after persistence processing.

[0170] In addition, an embodiment of the present invention further provides a multi-source heterogeneous data fusion device, including:

[0171] The theme library fusion module is used to obtain multi-source heterogeneous data, and execute the specified theme library fusion rules through the corresponding theme library fusion engine to obtain the theme library data; wherein, the theme library fusion rules include: cleaning rules, structured processing rules and theme library priority fusion rules; and / or, the theme library fusion engine includes: data cleaning engine, structured processing engine and theme library priority fusion engine.

[0172] The thematic library fusion module is used to execute the specified thematic library fusion rules on the subject library data through the corresponding thematic library fusion engine to obtain the thematic library data, wherein the thematic library fusion rules include: data fusion rules and thematic library priority fusion rules; and / or, the thematic library fusion engine includes: data fusion engine and thematic library priority fusion engine.

[0173] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned multi-source heterogeneous data fusion method.

[0174] like Figure 6 As shown, an embodiment of the present invention further provides a device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus.

[0175] The memory is a computer-readable storage medium for storing one or more programs.

[0176] The processor is used to execute the program stored in the computer-readable storage medium.

[0177] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or may exist independently without being assembled into the device / apparatus.

[0178] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source heterogeneous data fusion method, characterized in that: include: Obtain multi-source heterogeneous data, execute the specified theme library fusion rules through the corresponding theme library fusion engine, and obtain theme library data; The specified thematic library fusion rules are executed on the subject library data through the corresponding thematic library fusion engine to obtain the thematic library data.

2. The method according to claim 1, characterized in that: The subject library fusion rules include: cleaning rules, structured processing rules and subject library priority fusion rules; and / or, The subject library fusion engine includes: a data cleaning engine, a structured processing engine and a subject library priority fusion engine.

3. The method according to claim 2, characterized in that The method of acquiring multi-source heterogeneous data and executing a specified theme library fusion rule through a corresponding theme library fusion engine to obtain theme library data includes: Setting data cleaning rules, and executing the data cleaning rules on the multi-source heterogeneous data through a data cleaning engine; Set structured processing rules and convert the cleaned data into corresponding entity structures through the structured processing engine; Set the theme library priority fusion rules, build a data unique identification system through the theme library priority fusion engine, and obtain the theme library data.

4. The method according to claim 3, characterized in that The structured processing engine comprises: Logical units, execution core, business process operations, and data access layers.

5. The method according to claim 1, characterized in that: The thematic library fusion rules include: data fusion rules and thematic library priority fusion rules; The thematic library fusion engine includes: a data fusion engine and a thematic library priority fusion engine.

6. The method according to claim 5, characterized in that The data fusion engine comprises: Data projection engine, data connection engine, data selection engine and data aggregation engine.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The data fusion engine operates on the subject database data according to the data fusion rules to generate business tables that meet specific analysis or reporting requirements; The thematic library priority fusion engine integrates data according to the thematic library priority rules to generate thematic library data.

8. A multi-source heterogeneous data fusion system, characterized in that: The system comprises: The theme library fusion module is used to obtain multi-source heterogeneous data, execute the specified theme library fusion rules through the corresponding theme library fusion engine, and obtain the theme library data; The subject library fusion module is used to execute the specified subject library fusion rules on the subject library data through the corresponding subject library fusion engine to obtain the subject library data.

9. The system according to claim 8, characterized in that: The subject library fusion rules include: cleaning rules, structured processing rules and subject library priority fusion rules; and / or, The subject library fusion engine includes: a data cleaning engine, a structured processing engine and a subject library priority fusion engine.

10. The system according to claim 8, characterized in that The theme library fusion module is also configured to: Setting data cleaning rules, and executing the data cleaning rules on the multi-source heterogeneous data through a data cleaning engine; Set structured processing rules and convert the cleaned data into corresponding entity structures through the structured processing engine; Set the theme library priority fusion rules, build a data unique identification system through the theme library priority fusion engine, and obtain the theme library data.

11. The system according to claim 8, characterized in that: The thematic library fusion rules include: data fusion rules and thematic library priority fusion rules; and / or, The thematic library fusion engine includes: a data fusion engine and a thematic library priority fusion engine.

12. The system according to any one of claims 8 to 11, characterized in that: The thematic library fusion module is also configured to: The data fusion engine operates on the subject database data according to the data fusion rules to generate business tables that meet specific analysis or reporting requirements; The thematic library priority fusion engine integrates data according to the thematic library priority rules to generate thematic library data.

13. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed, the multi-source heterogeneous data fusion method described in any one of claims 1 to 7 can be implemented.

14. An electronic device comprising a processor, a communication interface, the computer-readable storage medium of claim 13, and a communication bus; wherein: The processor, the communication interface, and the computer-readable storage medium communicate electronically with each other via a communication bus; characterized in that: The processor is configured to execute a program stored in a computer-readable storage medium.

Images