Data merging method and program for multiple data sources, and electronic program

Abstract

The invention relates to the technical field of data management, and discloses a data merging method and program for multiple data sources and an electronic program, and the method comprises the following steps: synchronizing data of multiple data sources to a KafKa platform by using multiple synchronization modes; merging the data consumption of the KafKa platform to obtain a merged wide table; wherein the merged wide table comprises a main table and a sub-table; and configuring a wide table merging rule through an interface operation so as to merge the wide table for data query. According to the method, the data of the association table is pre-merged into one wide table, so that the calculation amount during query is reduced, and the system performance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, for example, to a data merging method and program for multiple data sources, and an electronic program. BACKGROUND

[0002] Modern transaction systems usually use relational databases as the background, and the HASH JOIN algorithm is usually used in SQL (Structured Query Language). However, when the number of associated tables and the number of associated levels increase (there are seven to eight tables or three to four levels of association), the computing performance will decrease sharply, and the association complexity in the transaction business far exceeds this scale, so direct use of the JOIN of SQL cannot meet the needs of the front-end query. In order to avoid the performance problems caused by association, the association needs to be eliminated first, and the multi-table is associated in advance and stored in a single table by the unified query system, and then the association is not needed during the query, so as to improve the query performance.

[0003] The related technology discloses a data warehouse that collects data from multiple heterogeneous data sources and integrates the data into a consistent format through data cleaning, conversion and merging steps.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that at least the following problems exist in the related art:

[0005] The related technology data warehouse is usually batch imported, and it is difficult to support real-time data update. In addition, multiple JOINs are required during query, which is complex and has low performance. Frequent table association operations not only increase the complexity of the query, but also significantly reduce the query performance.

[0006] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] In order to have a basic understanding of some aspects of the disclosed embodiments, the following is a simple summary. The summary is not a general review, nor is it intended to determine the key / important constituent elements or delineate the scope of protection of these embodiments, but as a prelude to the detailed description below.

[0008] The embodiments of the present disclosure provide a data merging method and program for multiple data sources, and an electronic program, to reduce the amount of calculation during query and improve performance.

[0009] In some embodiments, the method for data merging of multiple data sources comprises: synchronizing data of multiple data sources to a Kafka platform by using multiple synchronization modes; merging data of the Kafka platform to obtain a merged wide table; wherein the merged wide table comprises a master table and a sub table; configuring a merged wide table rule by an interface operation to query data of the merged wide table.

[0010] Optionally, the merging of data of the Kafka platform to obtain a merged wide table comprises: judging whether there is data of the Kafka platform in an original wide table; updating data in the case of the data of the Kafka platform; inserting the data of the Kafka platform into the original wide table in the case of no data of the Kafka platform.

[0011] Optionally, the inserting the data of the Kafka platform into the original wide table in the case of no data of the Kafka platform comprises: judging whether a current data table is a master table in the case of no data of the Kafka platform; inserting the data of the Kafka platform into the master table in the case of the current data table being the master table; skipping in the case of the current data table not being the master table.

[0012] Optionally, the updating data in the case of the data of the Kafka platform comprises: locating a result set to be updated in the case of the data of the Kafka platform; judging whether there is an update time; updating data according to the located result in the case of the update time; wherein an update field of the data is less than the update time; directly updating the data in the case of no update time.

[0013] Optionally, the directly updating the data in the case of no update time comprises: covering current data.

[0014] Optionally, the configuring a merged wide table rule by an interface operation to query data of the merged wide table comprises: judging whether a current data table is a master table after updating data; judging whether there is a join operation in the case of the current data table being the master table; querying data of a sub table according to a query rule in the case of the join operation; skipping in the case of the current data table not being the master table or no join operation.

[0015] Optionally, the method for data merging of multiple data sources further comprises: updating data in the case of unsuccessfully inserting the data of the Kafka platform into the master table; judging whether there is a join operation to query data in the case of successfully inserting the data of the Kafka platform into the master table.

[0016] Optionally, the data merging method for multiple data sources further comprises: querying data of the data source according to the update time, comparing the data of the data source with data of the merged wide table to check the merged wide table, and checking whether the number of data sources and the number of the merged wide table are consistent according to the update time.

[0017] In some embodiments, the data merging program for multiple data sources comprises an interface configuration and processing program, and the program is characterized in that when consuming the specified data, the program executes the data merging method for multiple data sources as described above.

[0018] In some embodiments, the electronic program comprises an electronic program body and a data merging program for multiple data sources as described above, which is installed on a server to process data.

[0019] The data merging method and program for multiple data sources provided by the embodiments of the present disclosure can achieve the following technical effects:

[0020] The embodiments of the present disclosure first synchronize data from different databases to the Kafka platform by using multiple synchronization methods. Secondly, the data in the Kafka platform is processed to generate a merged wide table. During the merging process, the data is divided into a main table and a sub-table. By distinguishing the structures of the main table and the sub-table, the JOIN operation during querying is effectively reduced, and the querying performance is improved. Finally, the data querying of the merged wide table is realized through an interface operation. Users can configure the data source and field mapping relationship through a graphical interface without complex SQL statement writing, and can quickly complete data querying. In this way, the embodiments of the present disclosure pre-merge the data of the associated table into a wide table, reduce the calculation amount during querying, and improve the system performance.

[0021] The general description above and the following description below are exemplary and explanatory only and are not intended to be limiting of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, which are schematic and not intended to be limiting of the embodiments, and in which like reference numerals designate similar elements, the drawings are not necessarily to scale and wherein:

[0023] Figure 1 is an implementation environment schematic diagram of the data merging method for multiple data sources of the embodiments of the present disclosure;

[0024] Figure 2 is a schematic diagram of a data merging method for multiple data sources provided by the embodiments of the present disclosure;

[0025] Figure 3is a schematic diagram of a method for merging data of multiple data sources provided by an embodiment of the present disclosure;

[0026] Figure 4 is a schematic diagram of another method for merging data of multiple data sources provided by an embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram of a program for merging data of multiple data sources provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0029] The terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0030] Unless otherwise specified, the term "a plurality of" means two or more.

[0031] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B represents: A or B.

[0032] The term "and / or" is a description of the association relationship between objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, the three relationships.

[0033] The term "corresponding" can refer to an association relationship or a binding relationship. A and B correspond to each other means that there is an association relationship or a binding relationship between A and B.

[0034] Currently, in modern transaction systems, relational databases are widely used as background data storage. However, when the business logic needs to perform complex join queries on multiple tables, the traditional SQL query method (especially based on the HASHJOIN algorithm) will face significant performance problems.

[0035] The HASH JOIN algorithm implements the association between tables by constructing a hash table. Although this algorithm is efficient in some scenarios, the computational complexity and resource consumption increase significantly when a large number of tables and multiple levels of association are involved. The association between tables needs to be calculated dynamically every time a query is performed, which not only increases the calculation time but also can lead to excessive consumption of memory and computing resources. As the amount of data increases, the performance problems of the association operation will be further exacerbated.

[0036] Therefore, there is a need for a data merging method that supports real-time data merging, which can horizontally integrate multiple table data in different data sources into one table through simple interface configuration.

[0037] Figure 1 is an implementation environment diagram of the data merging method for multiple data sources according to the embodiments of the present disclosure. As shown in the figure, the implementation environment can include a wide table database 100, multiple databases 200, and a processing program 500. Figure 1

[0038] The multiple databases 200 are stored in different departments, systems or platforms, and are independent of each other, lacking sharing and cooperation. The isolated state limits the mobility and utilization value of data.

[0039] The processing program 500 aggregates all data into Kafka through a data synchronization technology. When business data is distributed in different databases 200, the processor 500 uses a wide table technology to aggregate multiple data sources into a larger wide table through a unique transaction number, reducing the amount of calculation when querying and improving performance. Finally, the wide table database 100 provides an interface-based way to configure the wide table, which is simple to operate and does not require complex modeling operations.

[0040] In combination with Figure 2 The embodiments of the present disclosure provide a data merging method for multiple data sources, which includes:

[0041] S201, the processing program synchronizes the data of multiple data sources to the Kafka platform using multiple synchronization methods.

[0042] S202, the processing program merges the data of the Kafka platform to obtain a merged wide table; wherein the merged wide table includes a main table and a sub table.

[0043] S203, the processing program configures the merged wide table rule through an interface-based operation to query the data in the merged wide table.

[0044] ​The data merging method provided in the embodiment of the present disclosure first synchronizes data from different databases to the Kafka platform using multiple synchronization methods. According to different database types, an adaptive synchronization tool is used to ensure that data flows into Kafka in real time and accurately. Second, the data in the Kafka platform is merged and processed to generate merged wide table data, which is divided into a main table and a sub-table: the main table is responsible for storing core data, and the sub-table stores associated data. Through this main-sub table structure, the JOIN operation during query is effectively reduced, and the query performance is improved. Finally, the data query of the merged wide table is realized through the interface operation. Users can configure the data source, field mapping relationship, etc. through the graphical interface without complex SQL statement writing, and can quickly complete the data query.

[0045] In this way, real-time synchronization of multiple data sources is realized through Kafka, ensuring the timeliness and consistency of data. The wide table design of the main-sub table structure greatly reduces the association operation during query, significantly improving the query efficiency. Finally, the interface operation reduces the use threshold, so that non-technical users can easily perform data query.

[0046] Specifically, as shown in Figure 3 The mysql database or tidb database is synchronized to Kafka using a general data collection device. The db2 type database is synchronized to Kafka using OGG (Oracle GoldenGate).

[0047] Optionally, the data of the Kafka platform is merged and processed to obtain a merged wide table, including: judging whether the original wide table has data of the Kafka platform; updating the data in the case of having data of the Kafka platform; inserting the data of the Kafka platform into the original wide table in the case of not having data of the Kafka platform.

[0048] In the embodiment of the present disclosure, data is obtained from the Kafka platform, which may come from multiple data sources, and after preliminary cleaning and formatting, enters the wide table processing stage. It is judged whether the original wide table already has a record corresponding to the data of Kafka, which is based on a preset unique identifier (such as a transaction number or a primary key). If the original wide table already has data of the Kafka platform, the update process is entered, and the fields of the corresponding record in the wide table are updated to ensure that the data in the wide table is always up-to-date. If the original wide table does not have data of the Kafka platform, the Kafka data will be inserted into the wide table as a new record, thereby ensuring the integrity of the wide table data.

[0049] In this way, efficient data merging and updating can be achieved, significantly improving data processing efficiency and accuracy. At the same time, by updating the wide table data in real time, the system can support faster query response, meeting the strict requirements of real-time business systems on data timeliness.

[0050] Optionally, in the case where there is no Kafka platform data, the Kafka platform data is inserted into the original wide table, including: in the case where there is no Kafka platform data, determining whether the current data table is a master table; in the case where the current data table is a master table, inserting the Kafka platform data into the master table; in the case where the current data table is not a master table, performing skip processing.

[0051] In the embodiments of the present disclosure, data is obtained from the Kafka platform and it is checked whether these data already exist in the original wide table. If not, it is further determined whether the current data table is a master table. The master table is the core part of the wide table, storing key business data, while the sub-table stores auxiliary data related to the master table. If the current data table is a master table, the data of the Kafka platform is inserted as a new record into the master table, ensuring the completeness and real-time nature of the master table data. However, if the current data table is not a master table (i.e., a sub-table), the system performs skip processing to avoid inserting data that does not exist in the master table into the sub-table, thereby preventing data redundancy and potential association errors.

[0052] Through this insertion strategy, efficient and accurate data integration can be achieved, significantly improving the quality and usability of wide table data. By distinguishing between the insertion operations of master tables and sub-tables, the system can avoid unnecessary data redundancy, reduce the waste of storage resources, and improve the efficiency of data processing.

[0053] Optionally, in the case where there is Kafka platform data, the data is updated, including: in the case where there is Kafka platform data, locating the result set that needs to be updated; determining whether there is an update time; in the case where there is an update time, updating the data according to the located result; wherein the update field of the data is less than the update time; in the case where there is no update time, directly updating the data.

[0054] In the embodiments of the present disclosure, after obtaining data from the Kafka platform, the unique identifier (such as transaction number) in the Kafka data is matched with the corresponding field in the wide table to locate the result set that needs to be updated in the wide table. Then, it is determined whether there is an update time field in the Kafka data. If there is an update time field, the data in the wide table is updated, and the update operation only involves fields whose values are less than the update time in the Kafka data, thereby ensuring the timeliness and consistency of the data. If there is no update time field in the Kafka data, the data in the wide table is directly updated.

[0055] In this way, the real-time performance and accuracy of the wide table data can be significantly improved. Through accurate update time judgment and field-level update operation, the system avoids unnecessary data coverage, reduces data redundancy and potential errors. At the same time, this method also improves the efficiency of data processing, especially when processing large-scale data, which can significantly reduce the consumption of system resources.

[0056] Optionally, in the absence of an update time, the current data is overwritten.

[0057] Optionally, the wide table merging rule is configured through the interface operation to merge the wide table for data query, including: after the data is updated, it is judged whether the current data table is a master table; if the current data table is a master table, it is judged whether there is a join operation; if there is a join operation, the data of the child table is queried according to the query rule; if the current data table is not a master table, or there is no join operation, the skip processing is performed.

[0058] In the embodiments of the present disclosure, if the current data table is a master table, it is further judged whether there is a JOIN operation requirement related to the master table. The JOIN operation is usually used to associate the master table with the child table for query to obtain a complete data view. If there is a JOIN operation, the related data is queried from the child table according to the preset query rule (such as the associated field, the query condition, etc.), and is merged with the master table data, and finally the complete query result is returned. This rule-based query method can ensure the accuracy and efficiency of the query. If the current data table is not a master table, or there is no JOIN operation requirement, the system performs skip processing to avoid unnecessary query operations, thereby saving system resources and improving query performance.

[0059] Through the interface operation, the convenience and flexibility of data query are significantly improved. Users do not need to write complex SQL query statements, but only need to configure query rules through a graphical interface to quickly complete data query. At the same time, through the query logic of distinguishing the master table and the child table, and the optimized processing of the JOIN operation, the system can effectively reduce unnecessary association query, and reduce the query complexity and resource consumption.

[0060] Optionally, the data merging method for multiple data sources further includes: in the case that the insertion of the data of the KafKa platform into the master table is unsuccessful, the data is updated; in the case that the insertion of the data of the KafKa platform into the master table is successful, it is judged whether there is a join operation to perform data query.

[0061] In the embodiments of the present disclosure, if the insertion operation fails, the update mode is automatically switched to, and the Kafka data is applied to the master table as an update operation to ensure the real-time and consistency of the data. If the insertion operation is successful, it is further determined whether there is a JOIN operation demand related to the master table. If there is a JOIN operation, a complete query result is generated according to the preset query rule.

[0062] By dynamically processing data insertion and update operations, the flexibility and reliability of data processing are significantly improved. When data insertion fails, automatically switching to the update operation can ensure that data is not lost due to insertion errors, and at the same time, the problem of data duplication is avoided. After successful insertion, by judging the demand of JOIN operation, it can be intelligently determined whether to perform associated query, thereby saving system resources and improving query performance.

[0063] Optionally, the data merging method for multiple data sources further comprises: querying the data of the data source according to the update time, comparing the data of the data source with the data of the merged wide table, and checking the merged wide table; checking whether the number of data sources and the number of merged wide tables are consistent according to the update time, to check the merged wide table.

[0064] In combination Figure 4 The embodiments of the present disclosure provide another data merging method for multiple data sources, which comprises:

[0065] S401, the processing program synchronizes the data of multiple data sources to the Kafka platform by using multiple synchronization modes.

[0066] S402, the processing program performs merging processing on the data of the Kafka platform to obtain a merged wide table; wherein the merged wide table comprises a master table and a sub-table.

[0067] S403, the processing program queries the data of the data source according to the update time, compares the data of the data source with the data of the merged wide table, and checks the merged wide table.

[0068] S404, the processing program checks whether the number of data sources and the number of merged wide tables are consistent according to the update time, to check the merged wide table.

[0069] S405, the processing program configures the merged wide table rule through the interface operation to perform data query on the merged wide table.

[0070] In the embodiments of the present disclosure, first, data is queried from data sources according to update time, which can include multiple heterogeneous databases, and the data is synchronized to the merged wide table through the Kafka platform. Then, the system compares the queried data source data with the data in the merged wide table piece by piece to check whether the field values are consistent. It can be found that the data may be lost, incorrect or inconsistent in the synchronization or processing process.

[0071] Specifically, the data from 10 minutes ago to 55 minutes ago is checked every 5 minutes (configurable), and the data from 1 point T+1 is checked at 1 o'clock in the morning every day (configurable).

[0072] In addition, the number of data in the data source and the number of data in the merged wide table are also checked according to the update time. If the numbers are inconsistent, further analysis of the specific time interval is performed to locate the specific position of the data loss or redundancy, thereby ensuring the integrity of the data.

[0073] Specifically, the number of updates from 6 minutes ago to 1 minute ago and the number of original tables are checked according to the update time. If they are consistent, the check is successful, and if they are inconsistent, the specific inconsistent time interval is queried according to the unit, and finally the specific inconsistent second is queried.

[0074] By introducing the data checking mechanism based on update time, the reliability and accuracy of data processing are significantly improved. The data field comparison can find and correct the inconsistency of data content, and the data number checking can ensure that there is no omission or duplication in the synchronization and processing process. This double checking mechanism not only improves the data quality, but also enhances the fault tolerance of the system, which can timely find and solve the problems in data synchronization and processing.

[0075] In combination with Figure 5 As shown in the figure, the embodiments of the present disclosure provide a data merging program 50 for multiple data sources, which includes a processing program 500 and a wide table database 501. Optionally, the data merging program 50 can also include a communication interface (Communication Interface) 502 and a data collection program 503. The communication interface 502 can be used for information transmission. The processing program 500 can call the logical instructions in the wide table database 501 to execute the data merging method for multiple data sources in the above-mentioned embodiments.

[0076] In addition, the logical instructions in the wide table database 501 described above can be realized in the form of a software function unit and sold or used as an independent product when used, which can be stored in a computer readable storage medium.

[0077] The wide table database 501 can be used to store software program data, such as data corresponding to the method in the embodiments of the present disclosure. The processing program 500 executes functions and data processing by running data in the data collection program 503, and finally stores the data in the wide table database 501, that is, the data merging method for multiple data sources in the above-mentioned embodiments is implemented.

[0078] The embodiments of the present disclosure provide an electronic program, which comprises an electronic device program and the above-mentioned data merging program for multiple data sources. The data merging program for multiple data sources is installed on a server to process data.

[0079] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0080] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0081] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to apparatuses, devices, etc.) can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. For example, the division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or among them, can be indirect couplings or communication connections through some interfaces, devices, or units, and can be in electric, mechanical, or other forms.

[0083] The flowcharts and block diagrams in the drawings show the possible implementation architectures, functions, and operations of the system, method, and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks can occur in an order different from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the drawings, the operations or steps corresponding to different blocks can also occur in an order different from that disclosed in the descriptions, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for merging data from multiple data sources, characterized in that, include: Utilize multiple synchronization methods to synchronize data from multiple data sources to the KafKa platform; The data consumption on the KafKa platform is merged to obtain a merged wide table; the merged wide table includes a main table and sub-tables. Configure rules for merging wide tables through a user interface to perform data queries using merged wide tables.

2. The data merging method according to claim 1, characterized in that, The data from the KafKa platform is merged to obtain a merged wide table, including: Determine if the original wide table contains data related to the KafKa platform; If data exists on the KafKa platform, update the data; If no data for the KafKa platform exists, insert the KafKa platform data into the original wide table.

3. The data merging method according to claim 2, characterized in that, If KafKa platform data does not exist, insert KafKa platform data into the original wide table, including: In the absence of data on the Kafka platform, determine whether the current data table is the primary table; With the current data table as the main table, insert data from the KafKa platform into the main table; If the current data table is not the primary table, skip the processing.

4. The data merging method according to claim 2, characterized in that, If data exists on the KafKa platform, update the data, including: Given the existence of data on the Kafka platform, locate the result set that needs to be updated; Determine if an update time exists; If an update time exists, update the data based on the location result; where the updated field of the data is less than the update time. If no update time exists, update the data directly.

5. The data merging method according to claim 2, characterized in that, Configure merge wide table rules through a graphical interface to perform data queries using merged wide tables, including: After updating the data, determine whether the current data table is the primary table; Given the current data table as the main table, determine if a join operation exists. When a join operation exists, retrieve data from the sub-table according to the query rules; If the current data table is not the primary table, or if there is no join operation, skip the process.

6. The data merging method according to claim 3, characterized in that, Also includes: If inserting data from the KafKa platform into the main table fails, update the data. If data from the KafKa platform is successfully inserted into the main table, check if a join operation exists in order to perform a data query.

7. The data merging method according to any one of claims 1 to 6, characterized in that, Also includes: Query the data source based on the update time, compare the data from the data source with the data in the merged wide table, and verify the merged wide table. Based on the update time, verify whether the number of rows in the data source matches the number of rows in the merged wide table to check the merged wide table.

8. A data merging program for multiple data sources, comprising an interface configuration and a processing program, characterized in that, When consuming specified data, the processing program executes the data merging method for multiple data sources as described in any one of claims 1 to 7.

9. An electronic program, characterized in that, include: Electronic program body; The data merging program for multiple data sources as described in claim 8 is installed on a server to process the data.

Images