Star-type parallel single-theme multi-source data fusion method, medium, device and system

Abstract

The application relates to a star-type parallel single-theme multi-source data fusion method, medium, equipment and system, a total table corresponding to multi-source data is arranged, an extensible sub-table is arranged corresponding to the total table; total table and sub-table configuration information are matched through sub-table information; sub-table content is fused into the corresponding total table according to a strategy; the medium and the equipment are realized based on the method; the system stores the total table and total table information in a data server, a plurality of user terminals realize sub-table data reading, writing and total table data reading, a configuration unit configures total table information, issues sub-table information and allocates new sub-tables, and a data fusion unit fuses sub-table data into the total table after processing the sub-table data based on a strategy. Each source of the application independently operates and fuses sub-table data, and the latest data corresponding to the total table can be obtained; parallel fusion is realized, the fusion result is always available, the parallel degree and the efficiency of data fusion are significantly improved; different application services are processed in parallel, tasks based on requirements are realized, and the task amount is theoretically unlimited.

Description

Technical Field

[0001] This invention relates to the technical field of electronic digital data processing, and in particular to a star-parallel single-topic multi-source data fusion method, medium, device and system. Background Technology

[0002] Big data refers to data sets whose scale far exceeds the capabilities of traditional database software tools in terms of acquisition, storage, management, and analysis. It is characterized by four main features: massive data volume, rapid data flow, diverse data types, and low value density. With the continuous breakthroughs and development of technologies such as the Internet of Things, artificial intelligence, and 5G, the global data sources and data volume are growing exponentially, resulting in a significant demand for data reading, writing, and storage.

[0003] ETL (Extract-Transform-Load) is used to describe the process of extracting, transforming, and loading data from the source to the destination. It can extract various distributed and heterogeneous source data (such as relational data), clean up "dirty" data such as incomplete, duplicate, and erroneous data according to pre-designed rules, obtain "clean" data that meets the requirements, and load it into a data warehouse for storage, making this data the cornerstone of data analysis and mining.

[0004] However, traditional ETL often suffers from problems such as timing dependencies, chaotic fusion processes, and unusable fusion results when handling multi-source data fusion needs. This severely impacts the parallelism and efficiency of data fusion, leading to redundancy in some data and missing data in others, resulting in inconvenience for the caller and hindering the smooth application of the data. When multiple (e.g., a dozen) tasks read and generate data that are highly correlated and have a large data volume (hundreds of millions to tens of billions), improving the system's parallelism, throughput, and fusion efficiency is of great value, while reducing the dependencies in tasks and solving unresolved problems while ensuring data sharing. Summary of the Invention

[0005] This invention solves the problems existing in the prior art and provides a star-shaped parallel single-topic multi-source data fusion method, medium, device and system.

[0006] The technical solution adopted in this invention is a star-shaped parallel single-topic multi-source data fusion method. The method sets up a master table corresponding to the multi-source data for global fusion. The master table is expandable, and expandable sub-tables are set up corresponding to the master table. In conjunction with the master table configuration information, the sub-tables are configured with sub-table information, which is then matched with the master table. Sub-table data is merged into the corresponding master table according to the strategy.

[0007] Preferably, the master table information includes a first ID and a master table identifier unit corresponding to any master table, the master table is divided into several groups, and each group corresponds to a sub-table; each group corresponding to any sub-table is configured with several attributes, and the attributes of any group are not repeated.

[0008] Preferably, the master table identification unit includes an identification code corresponding to the master table and a first version number.

[0009] Preferably, the sub-table information includes a second ID corresponding to any group, several corresponding attributes, and a sub-table identifier unit, wherein the second ID is associated with the first ID through the corresponding group.

[0010] Preferably, the sub-table identification unit includes an identification code corresponding to the master table and a second version number.

[0011] Preferably, the data of the sub-tables are processed, and the data between different sub-tables is deduplicated using the second version number. The deduplicated sub-table data is then merged into the corresponding master table based on the identification code of the master table.

[0012] Preferably, the validity of data associated with the same second ID is determined by the second version number; After data fusion, the master table includes the master table identifier unit and the second version number of each sub-table during fusion. The final fusion content is determined based on the version number.

[0013] In this invention, the validity of data associated with the same ID is determined by the version. The larger the version number, the newer the data is and the more it is retained. The total table data contains the version number of the total table (first version number) and the second version number of each sub-table when it is merged. Therefore, the final merged content can be determined based on the second version number of each sub-table when it is merged in the ID and the latest version number of the current sub-table.

[0014] A computer-readable storage medium storing a star-parallel single-topic multi-source data fusion program, which, when executed by a processor, implements the aforementioned star-parallel single-topic multi-source data fusion method.

[0015] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the above-described star-parallel single-topic multi-source data fusion method.

[0016] A star-shaped parallel single-topic multi-source data fusion system, the system comprising: A data server is used to store the master table and master table information; Several client terminals are used to read and write data from sub-tables and read data from the main table. A configuration unit is used to configure master table information, distribute sub-table information, and allocate new sub-tables; A data fusion unit is used to process and merge data from sub-tables into the master table based on a strategy.

[0017] This invention relates to a star-topic parallel single-topic multi-source data fusion method, medium, device, and system. The method sets up a master table corresponding to the multi-source data, and expandable sub-tables corresponding to the master table. It configures master table information and sub-table information, matching the sub-table information with the master table. Sub-table content is fused to the corresponding master table according to a strategy. The method is implemented using a computer-readable storage medium and computer equipment. The system uses a data server to store the master table and its information, and several user terminals to read and write data from sub-tables and the master table. A configuration unit configures master table information, distributes sub-table information, and allocates new sub-tables. A data fusion unit processes the sub-table data and fuses it to the master table based on a strategy.

[0018] The beneficial effects of this invention are as follows: (1) Eliminate the problems of time dependence and unusable fusion results in the processing of multi-source fusion in traditional ETL; (2) Each source can operate independently on the data in the sub-table and achieve fusion, and can obtain the latest data of the corresponding master table; (3) Achieve parallel fusion, and the fusion result is always available, significantly improving the parallelism and efficiency of data fusion; (4) It enables parallel processing of different application services and allows for the addition of tasks based on demand, with no upper limit on the amount of tasks in principle. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method of the present invention; Figure 2 This is a schematic diagram of an embodiment of the present invention; Figure 3 This is a system structure block diagram of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figure 1As shown, the present invention relates to a star-shaped parallel single-topic multi-source data fusion method, wherein the method sets up a master table corresponding to the multi-source data, the master table is expandable, and expandable sub-tables are set up corresponding to the master table; In conjunction with the master table configuration information, the sub-tables are configured with sub-table information, which is then matched with the master table. Sub-table data is merged into the corresponding master table according to the strategy.

[0022] In this invention, the master table is used to integrate, summarize and store multi-source data. Expandable sub-tables are set up in conjunction with the master table. Each sub-table corresponds to a part of the data in the master table. Under the premise that the sub-tables are expandable, the data in the master table can be expanded. Obviously, for the data pool in the master table corresponding to the sub-table, new data is written and integrated through the sub-table, but this new data is not directly written to the master table.

[0023] In this invention, the master table and the sub-table are respectively configured with master table information and sub-table information. Through these configuration information, the sub-table and the master table are matched, thereby satisfying requirements including but not limited to the fusion of sub-table data into the master table and the reading of partial data from the master table through the sub-table.

[0024] In this invention, sub-table data is merged into the corresponding master table according to a strategy. This strategy is generally preset in the system by technical personnel and automatically triggered by the system based on factors including but not limited to time (segment) and content (quantity).

[0025] like Figure 2 As shown, based on the above embodiments, the master table information includes a first ID and a master table identifier unit corresponding to any master table. The master table is divided into several groups, and each group corresponds to a sub-table. Each group corresponding to any sub-table is configured with several attributes, and the attributes of any group do not overlap.

[0026] In this invention, master table information is configured for any master table, with the first ID as the index of the current master table, and a master table identifier unit is set to satisfy the fusion operation of reading and writing sub-table data relative to the master table.

[0027] In this invention, the master table is divided into several groups, namely SEGM-0, SEGM-1... SEGM-n-1, where n is a positive integer; each group corresponds to a task type, and each sub-table processes one task type. That is, when a sub-table reads or writes data for each task, it directly reads or writes data to one group in the master table; to prevent data confusion, each group is defined to allow only one application (task) to access and read / write.

[0028] In this invention, when a new application (task) is generated, a new group is directly added to the master table, corresponding to a new sub-table, to ensure the integrity of the data and the consistency of data attributes.

[0029] In this invention, after grouping the master table data, the data within each group corresponds to different attributes. Taking group SEGM-0 as an example, it includes the attribute {attr 0.0 attr 0.1 attr 0.... In other words, during the writing process of sub-table data, data is actually written to different attributes in each group of the master table, and during the data fusion stage, the updated sub-table data is also written to the corresponding attributes in the master table group.

[0030] Based on the above embodiments, the sub-table information includes a second ID corresponding to any group, several corresponding attributes, and a sub-table identifier unit. The second ID is associated with the first ID through the corresponding group.

[0031] In this invention, reading and writing sub-tables is actually based on processing a certain task based on the data in the main table. Therefore, the second ID is used as the index of the current sub-table, and this second ID is associated with the first ID through the corresponding group. For example, the first ID is ID, and the sub-table corresponds to the si-th group, so the second ID is ID:SI, thus achieving the correspondence.

[0032] Based on the above embodiments, the master table identification unit includes an identification code corresponding to the master table and a first version number.

[0033] The sub-table identifier unit includes the identification code corresponding to the main table and the second version number.

[0034] In this invention, the identification code corresponding to the master table refers to the mark symbol, which is consistent with the information in the sub-table identification unit and is generally used to satisfy data retrieval.

[0035] In this invention, the master table identifier unit also includes a first version number, which is VER in the embodiment. Correspondingly, the sub-table identifier unit also includes a second version number, which is also VER in the embodiment. By comparing the first version number and the second version number, the deduplication and updating of different version content are achieved. That is, the sub-table data under different version numbers will be stored under different version numbers of the master table to prevent data confusion and errors, and at the same time facilitate data reading. In the embodiment, the second version number and the first version number of the sub-table and the master table are both VER, which can correspond to each other. That is, the sub-table data of the current version can be merged into the master table data.

[0036] The data in the sub-tables is processed, and the data between different sub-tables is deduplicated using the second version number. The deduplicated sub-table data is then merged into the corresponding master table based on the identification code of the master table.

[0037] And / or, the validity of data associated with the same second ID is determined by the second version number; After data fusion, the master table includes the master table identifier unit and the second version number of each sub-table during fusion. The final fusion content is determined based on the version number.

[0038] In this invention, data fusion from sub-tables to master tables is achieved through a built-in data fusion unit, which is generally a built-in engine.

[0039] In this invention, data comparison between different sub-tables is performed using a second version number. That is, for a certain second version number, the data is merged by referring to the first version number of the main table. For two different sub-tables, if their second version numbers are the same, the data in the two sub-tables is deduplicated. In practical applications, the merging time of different sub-tables is different, ensuring that the time can determine which version will start merging, and ensuring that the merging of two sub-tables will not conflict.

[0040] In this invention, during implementation, the first version number of the master table and the second version number of each group (sub-table) are maintained by the global fusion daemon, and the version number of each group is maintained by the group facade.

[0041] In this invention, it is obvious that it is necessary to check the integrity of the data in the sub-table. This integrity check can be set at the stage of writing data into the sub-table or before the data is merged into the main table, as required by those skilled in the art.

[0042] The present invention also relates to a computer-readable storage medium storing a star-parallel single-topic multi-source data fusion program, which, when executed by a processor, implements the above-described star-parallel single-topic multi-source data fusion method.

[0043] The present invention also relates to a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the above-described star-parallel single-topic multi-source data fusion method.

[0044] This invention also relates to a star-shaped parallel single-topic multi-source data fusion system, the system comprising: A data server is used to store the master table and master table information; Several client terminals are used to read and write data from sub-tables and read data from the main table. A configuration unit is used to configure master table information, distribute sub-table information, and allocate new sub-tables; A data fusion unit is used to process and merge data from sub-tables into the master table based on a strategy.

[0045] In this invention, the data server can store the master table and its corresponding master table information, and can expand the grouping in each master table to meet the writing of sub-table data under new applications (tasks).

[0046] In this invention, the data fusion unit exists in the form of an engine. The data processing here includes, but is not limited to, cleaning erroneous data in sub-tables, supplementing missing data, and removing duplicate data within the sub-tables themselves.

[0047] In this invention, the system manages the master table and sub-tables based on their corresponding IDs and version numbers, which facilitates the implementation of manually defined query statements, such as viewing the latest sub-table or master table, or viewing sub-tables or master tables with different IDs.

[0048] In practical applications, the core components of this invention include three parts: a global fusion daemon, a group facade, and a management console. It supports NoSQL databases such as ClickHouse and Hive, and the entire system uses metadata management. like Figure 3 As shown, during implementation, tasks (task0, task1, task2...) interact with the group facade, the group facade reads and writes data with the NoSQL database, the NoSQL database is fused and stored by the global fusion process, and the management console manages the group facade and the global fusion process through the metadatabase.

[0049] The system of this invention can be further applied to fields such as clue generation, information release and feedback, AI autonomous services, and model evaluation. In particular, clue generation can meet the needs of rapid table building and industry chain updates for new industry chains. AI autonomous services can realize big data-based public opinion event extraction, industry chain clue evaluation, etc., which can be expanded into a large number of applications while ensuring the stability and availability of data.

[0050] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0053] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0054] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0055] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A star-shaped parallel single-topic multi-source data fusion method, characterized in that: The method sets up a master table corresponding to multi-source data, the master table is expandable, and expandable sub-tables are set up corresponding to the master table; In conjunction with the master table configuration, the sub-tables are configured with sub-table information, which is matched with the master table. The master table information includes a first ID corresponding to any master table and a master table identifier unit. The master table is divided into several groups, and each group corresponds to a sub-table. Each group corresponding to a sub-table is configured with several attributes, and the attributes of any group are unique. The sub-table information includes a second ID corresponding to any group, several corresponding attributes, and a sub-table identifier unit. The second ID is associated with the first ID through the corresponding group. The sub-table identifier unit includes a second version number. Sub-table data is merged into the corresponding master table according to the strategy; the data of the sub-table is processed, and the data between different sub-tables is deduplicated using the second version number. The deduplicated sub-table data is then merged into the corresponding master table based on the identification code of the corresponding master table.

2. The star-type parallel single-theme multi-source data fusion method according to claim 1, characterized in that: The master table identification unit includes the identification code of the corresponding master table and the first version number.

3. The star-type parallel single-theme multi-source data fusion method according to claim 1, characterized in that: The sub-table identifier unit also includes an identification code corresponding to the master table.

4. The star-type parallel single-theme multi-source data fusion method according to claim 1, characterized in that: The validity of data associated with the same second ID is determined by the second version number; After data fusion, the master table includes the master table identifier unit and the second version number of each sub-table during fusion. The final fusion content is determined based on the version number.

5. A computer-readable storage medium, characterized in that: It stores a star-parallel single-topic multi-source data fusion program, which, when executed by a processor, implements the star-parallel single-topic multi-source data fusion method as described in any one of claims 1 to 4.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the star-parallel single-topic multi-source data fusion method as described in any one of claims 1 to 4.

7. A star-parallel single-topic multi-source data fusion system employing the star-parallel single-topic multi-source data fusion method according to any one of claims 1 to 4, characterized in that: The system includes: A data server is used to store the master table and master table information; Several client terminals are used to read and write data from sub-tables and read data from the main table. A configuration unit is used to configure master table information, distribute sub-table information, and allocate new sub-tables; A data fusion unit is used to process and merge data from sub-tables into the master table based on a strategy.

Images