Multi-source data fusion method and device

Abstract

The embodiment of the application provides a kind of multi-source data fusion method and device, comprising: obtaining the source data of multiple data sources respectively;Based on the primary key field and foreign key field of source data, the association operation between multiple source data is carried out, obtains the first primary key field corresponding to the association operation, and the first corresponding relationship of the association key field participating in the association operation;The second corresponding relationship between the primary key field of source data and the content field of source data is constructed;Based on the matching of the primary key field of source data in the second corresponding relationship and the association key field in the first corresponding relationship, construct fusion data.The application calculates link simple, compared with related art greatly reduces the overall join operation between source data, to reduce the network overhead when data flow and the frequent operation to state, and only once association can obtain complete data, the data flow under this fusion mode is more quickly, and performance is higher.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and machine-readable medium for multi-source data fusion. Background Technology

[0002] In the construction of a real-time data warehouse, since the field information in a single data source is limited and it is difficult to support multi-dimensional and complex analysis in operations, the data streams from multiple data sources can be associated and merged to form a large wide table to meet the needs of multi-dimensional and complex analysis.

[0003] In related technologies, data stream fusion can be achieved by performing a join operation between pairs of data streams in multiple data streams. Specifically, during the processing, two data streams can be selected for join at a time. The upstream operator will process the joined data by performing a hash join operation according to the foreign key of the data stream, and then transmit it to the downstream through the network. The downstream operator repeats the upstream operation and performs the join processing with the remaining data streams in sequence to obtain the final wide table data.

[0004] However, the inventors discovered through research that the current solution, which involves joining data streams from one another, results in excessively long processing links, making maintenance difficult and incurring excessive network and computational overhead, thus affecting system performance. Summary of the Invention

[0005] This application provides a multi-source data fusion method to address the problem in related technologies where joining operations between pairs of data streams results in excessively long processing links, difficult maintenance, and excessive network and computational overhead, thus affecting system performance.

[0006] Accordingly, embodiments of this application also provide a multi-source data fusion device, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0007] To address the aforementioned problems, this application discloses a multi-source data fusion method, the method comprising:

[0008] Obtain source data from multiple data sources; the source data includes primary key fields, foreign key fields, and content fields.

[0009] Based on the primary key field and foreign key field of the source data, a join operation is performed between multiple source data to obtain a first correspondence between the first primary key field corresponding to the join operation and the join key field participating in the join operation; the join key field includes the primary key field and foreign key field participating in the join operation.

[0010] Construct a second correspondence between the primary key field of the source data and the content field of the source data;

[0011] Based on the matching of the primary key field of the source data in the second correspondence with the association key field in the first correspondence, the target content field corresponding to the first primary key field and the target primary key field selected from the first primary key field are used to construct fused data.

[0012] This application discloses a multi-source data fusion apparatus, the apparatus comprising:

[0013] The acquisition module is used to acquire source data from multiple data sources; the source data includes primary key fields, foreign key fields, and content fields.

[0014] The first construction module is used to perform association operations between multiple source data based on the primary key field and foreign key field of the source data, to obtain a first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation; the association key field includes the primary key field and foreign key field participating in the association operation.

[0015] The second construction module is used to construct a second correspondence between the primary key field of the source data and the content field of the source data;

[0016] The merging module is used to match the primary key field of the source data in the second correspondence with the association key field in the first correspondence, and to construct the fused data by combining the target content field corresponding to the first primary key field and the target primary key field selected from the first primary key field.

[0017] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more methods as described in this application.

[0018] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform one or more methods as described in this application.

[0019] Compared with related technologies, the embodiments of this application have the following advantages:

[0020] In this embodiment, the primary and foreign key matching relationship between real-time multi-source data is utilized. First, based on the primary key and foreign key fields of the source data, association operations are performed between multiple source data to determine the first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation. Then, a standardization processing method is used to construct a second correspondence between the primary key field and the content field of the source data. Finally, the first two data streams are spliced ​​together to restore the final fused data through data restoration. This method can greatly reduce the overall join operation between source data in related technologies, thereby reducing network overhead and frequent state operations during data flow, thus improving the performance of real-time computing tasks and reducing resource consumption. The optimized multi-source data fusion method in this embodiment has a simple, clear, and easy-to-maintain chain. The data flow is processed uniformly, intermediate states are greatly reduced, and complete data can be obtained with only one association. The data flow under this fusion method is faster and has higher performance. Attached Figure Description

[0021] Figure 1 This is a system architecture diagram of an embodiment of this application;

[0022] Figure 2 It is a data link diagram of related technologies;

[0023] Figure 3 This is a schematic diagram of data flow in related technologies;

[0024] Figure 4 This is a schematic diagram illustrating the data state processing of a related technology;

[0025] Figure 5 This is a data processing diagram of a related technology;

[0026] Figure 6 This is another data processing diagram of related technologies;

[0027] Figure 7 This is a schematic diagram of multi-source data fusion in a logistics scenario according to an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of multi-source data fusion in an e-commerce scenario according to an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of multi-source data fusion in a recommendation scenario according to an embodiment of this application;

[0030] Figure 10 This is a flowchart illustrating the steps of a multi-source data fusion method according to an embodiment of this application;

[0031] Figure 11 This is a schematic diagram illustrating the association between keys according to an embodiment of this application;

[0032] Figure 12 This is another schematic diagram illustrating the association between keys according to an embodiment of this application;

[0033] Figure 13 This is another schematic diagram illustrating the association between keys according to an embodiment of this application;

[0034] Figure 14 This is a flowchart illustrating the specific steps of a multi-source data fusion method according to an embodiment of this application;

[0035] Figure 15 This is a schematic diagram of multi-source data fusion using related technologies;

[0036] Figure 16 This is a schematic diagram illustrating the processing of a primary and foreign key relationship construction layer for multi-source data fusion according to an embodiment of this application;

[0037] Figure 17 This is a schematic diagram of the processing of a fusion construction layer for multi-source data fusion according to an embodiment of this application;

[0038] Figure 18 This is a schematic diagram of the data restoration layer processing of multi-source data fusion according to an embodiment of this application;

[0039] Figure 19 It is a computational topology graph of related technologies;

[0040] Figure 20 This is a computational topology diagram according to an embodiment of this application;

[0041] Figure 21 This is a block diagram of a multi-source data fusion device according to an embodiment of this application;

[0042] Figure 22 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0043] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] To enable those skilled in the art to better understand this application, the concepts involved in this application are explained below:

[0045] Data source: refers to the source of data, specifically the device or original media that provides the required data. In the embodiments of this application, the data source may correspond to an overall service or a sub-service under the service, thereby storing the data generated by the service and providing this data to the outside world.

[0046] Source data: Real-time data streams provided by data sources. It is a type of data provided by a service that has not been further processed. Source data can specifically exist in the form of data tables.

[0047] Primary key: A primary key is a unique identifier that can determine a record. It can consist of one field or a combination of multiple fields. The definition of a primary key is to uniquely identify a record. There cannot be duplicates, and it cannot be null. Its purpose is to ensure data integrity. A data table can only have one primary key.

[0048] Foreign key: A foreign key is used to identify a record in another table. It can consist of a single field or a combination of multiple fields. It is a field that identifies a record in another table. For example, if a field in table A is the primary key of table B, then that field can be a foreign key in table A. Foreign keys can have duplicates and null values, and a table can have multiple foreign keys.

[0049] Content fields: In a data table (data stream), besides primary key fields and foreign key fields, content fields are specifically used to represent the content meaning of the data table.

[0050] Join operation: Used to associate data from two or more tables based on the common attributes of columns. For join operations on multiple tables (more than 2), the multiple tables can be split into pairs of tables and then joined in pairs. The result set of the first two tables involved in the join operation can then be used in the join operation of subsequent tables.

[0051] Association key: Since the join operation can achieve the association between the primary key and foreign key fields of two data tables, the primary key and foreign key of the join operation can be used as the association key.

[0052] Wide table: This can refer to a database table with a large number of fields. After merging source data from multiple data sources, the resulting table has a rich set of fields, which can meet the needs of subsequent complex analysis. Therefore, the resulting table after merging can be called a wide table.

[0053] Data container: A carrier that stores data in a predefined standard data format, such as a map data container that can store data in a key-value standard format.

[0054] Retract: This is a data update processing method in streaming computing scenarios. Since real-time computing is a message-triggered computing mechanism, in order to compensate for the problem of data error caused by inconsistent arrival times of multiple data streams, a retract mechanism is introduced inside real-time computing. This mechanism removes the previously erroneously sent data by sending "reverse" data.

[0055] State: In the field of real-time computing, a cached data model is introduced to save intermediate results of real-time computing. Unlike traditional offline computing, which can store intermediate computing results in Pangu files, real-time computing engines use state to save intermediate computing results in order to ensure the accuracy of the final result of real-time computing.

[0056] Backpressure: refers to the phenomenon where the amount of data input to a computing node is greater than the amount of data that the computing node can process, resulting in data blocking due to limited computing resources of the computing node.

[0057] Shuffle: In the field of big data, when dealing with large-scale data source computation, the "divide and conquer" approach is generally adopted, which means that a large amount of data is computed in parallel through multiple nodes. In the field of real-time computing, task execution is divided into different operators according to business logic. Each operator contains multiple running instances. After the upstream instance processes the data according to the user-defined business logic, it sends it to the downstream instance according to certain rules. This action of the upstream instance sending data to the downstream instance according to the rules is called the shuffle operation. The shuffle operation is generally a cross-node operation, which has a large network overhead.

[0058] The multi-source data fusion method in this application embodiment can be applied to data processing scenarios. These scenarios can include: analysis scenarios, statistical scenarios, recommendation scenarios, etc. Analysis scenarios can include: log analysis scenarios, traffic analysis scenarios, etc. Statistical scenarios can include: website access statistics scenarios, ranking scenarios, etc. Recommendation scenarios can include: advertising targeting scenarios, personalized recommendation scenarios, etc.

[0059] For example, in analytical scenarios, source data from multiple service data sources can be merged to obtain a wide-table merged data set, whose rich fields can meet the needs of complex analyses. Similarly, in statistical scenarios, source data from multiple data sources can be merged to obtain a wide-table merged data set, whose rich fields can also meet the needs of complex statistics, saving statistical time. Furthermore, in recommendation scenarios, source data generated by an object on various service platforms can be merged to obtain a wide-table merged data set, which can be used to accurately represent the object's profile, thereby improving recommendation accuracy. These scenarios are applicable to fields including, but not limited to, logistics and e-commerce. It is understood that the above application scenarios are merely examples, and this application does not limit specific application scenarios.

[0060] Reference Figure 1 The diagram illustrates a system architecture of a multi-source data fusion method provided in this application, including: a data source, a data pipeline, a fusion server, and a client. The fusion server includes: a primary-foreign key relationship construction layer, a fusion construction layer, and a data restoration layer.

[0061] The data source can be a data source corresponding to a service in a service cluster. The data source is used to store the data generated by that service, thus forming the corresponding source data. The source data can have primary key fields, foreign key fields (which may or may not be present), and content fields. The data pipeline can be a software process running on a hardware device. The data source can synchronize the source data to the data pipeline in real time, and the data pipeline can then send the source data to the fusion server in real time. The fusion server can then send the fused data obtained from the source data to the client for analysis and use.

[0062] In this embodiment, the relationship between the primary key and foreign key of the source data is an important attribute for defining the uniqueness of each data table and characterizing the association between data tables. It is also an important attribute for realizing multi-source data fusion in this embodiment.

[0063] For example, suppose there exists a student table containing: student ID, name, gender, and course number, and a course table containing: course number, course name, and credits. Each student's student ID is unique, so the student ID is the primary key of the student table. Each course number is also unique, so the course number is the primary key of the course table. The student table has a key named "course number" to represent the course number selected by each student. Since the course number is the primary key of the course table, it is also a foreign key in the student table. Through this foreign key, the student table can establish a relationship with the course table and import the course number data from the course table, making the data in the student table richer and more accurate.

[0064] Specifically, the fusion server can be divided into three data processing layers: a primary-foreign key relationship construction layer, a fusion construction layer, and a data restoration layer. The primary-foreign key relationship construction layer extracts the primary and foreign keys of multiple source data sets and defines the matching relationships between them. The core idea is to perform join operations (association operations) on each pair of source data sets according to their acquisition time, based on the primary key and foreign key fields. This generates two columns arranged in the order of data acquisition: one column is the primary key set consisting of the first primary key field corresponding to each join operation, and the other column is the first association key set consisting of the association key fields, which are the primary and foreign keys involved in the join operation. The number of rows in these two columns corresponds to the number of primary and foreign key types in the source data. The primary key set is used to represent the unique primary key specified in each join operation, while the first association key set is used to represent the association key involved in each join operation. The primary keys in the primary key set and the association keys in the first association key set can establish a first correspondence between them by being in the same layer. The subsequent data restoration layer can use this first correspondence to correctly and quickly merge the content fields of the source data under the corresponding primary key.

[0065] Since the data objects in these two data sets can be directly extracted from the original source data, and the amount of data in these two data sets is extremely small, the computational cost required for the primary-foreign key relationship construction layer is extremely low, and the resource consumption involved is almost negligible. As a lightweight front-end layer for processing source data, the primary-foreign key relationship construction layer has minimal negative impact on system load.

[0066] For example, Figure 1 In the given data, source data 1 includes: primary key a, foreign key b, and content field 1; source data 2 includes: primary key a, foreign key c, and content field 2; source data 3 includes: primary key b, content field 3… source data n includes: primary key d and content field n. The primary-foreign key relationship layer can be constructed by performing join operations between pairs of source data based on the primary key and foreign key fields, resulting in two data sets. One column is the primary key set consisting of primary key a, primary key a, primary key a… primary key a, and the other column is the first set of association keys consisting of primary key a, foreign key b, foreign key c… foreign key d.

[0067] Furthermore, the fusion construction layer can utilize a preset standard data structure to uniformly merge the content of source data from different data sources, thus departing from the fusion method of pairwise join operations in related technologies. Through a single unified fusion, the fusion construction layer can also obtain two data sets: one column is a second association key set composed of the primary key fields of the source data, and the other column is a content set composed of the content fields of the source data. Based on whether they are in the same row, there can be a second correspondence between the primary key fields of the source data in the second association key set and the content fields of the source data in the content set. Specifically, in this embodiment, the content fields of the source data can be written into a preset data container for storage, and the content fields of the source data can be standardized through the standard data structure defined by the data container itself. In this embodiment, the standardized content fields are aggregated into a unified data stream through a single fusion operation, which can reduce the network and system overhead of multiple join operations in related technologies (fusion method of pairwise join operations), and greatly reduce data computing costs.

[0068] For example, Figure 1 In the given data, source data 1 includes: primary key a, foreign key b, and content field 1; source data 2 includes: primary key a, foreign key c, and content field 2; source data 3 includes: primary key b, content field 3… source data n includes: primary key d and content field n. The fusion construction layer can write the content fields of the source data into data containers respectively, and then perform a unified fusion operation to construct two data sets. One column is a second set of related keys consisting of primary key a, primary key b… primary key d; the other column is a set of content fields consisting of data container 1, data container 2, data container 3… data container n.

[0069] Furthermore, the primary-foreign key relationship construction layer aims to utilize a lightweight, low-cost front-end stream to uniquely determine the primary key of the final output data during data restoration. The fusion construction layer aims to build a unified stream to address performance, cost, and data accuracy issues caused by multi-stream associations in related technologies. The data restoration layer can then leverage the data streams constructed in the first two layers, performing field matching between the first and second association key sets according to the acquisition time of the source data. This matches the source data content fields under the primary key of the final output fused data, thereby establishing a wide-table format of fused data. This wide-table format of fused data contains a unique target primary key field determined from the primary key set, and under the target primary key field are content fields arranged in chronological order of acquisition time, ensuring the correctness of the fused data. It should be noted that if new data sources are subsequently introduced, only corresponding expansions to the "primary-foreign key relationship construction layer" and "fusion construction layer" are needed, making the update method fast, convenient, and with extremely low overhead.

[0070] The effectiveness of the embodiments of this application in solving the technical problem will now be discussed through explanation and description of the implementation of related technologies:

[0071] Reference Figure 2 The diagram illustrates a data link diagram provided by related technologies. When performing multi-source data fusion, these technologies employ a pairwise join approach, where the result sets of the join operations between the two data sources initially participate in the join operation. This results in a unidirectional sequential computation link, which can become very long when there are many data sources, leading to poor stability and difficulty in maintenance.

[0072] To address the aforementioned issues, this application's embodiments develop a front-end primary and foreign key relationship construction layer and a fusion construction layer. These layers, respectively, use lightweight computational methods to construct data streams reflecting primary and foreign key matching relationships and data streams reflecting primary key and content field matching relationships. Finally, the final fused data is obtained through a single association by the data restoration layer. The computational chain in this application's embodiments is a simple three-layer network topology, independent of... Figure 2 The link shown solves the problems of complex and difficult-to-maintain links in related technologies.

[0073] Reference Figure 3 The diagram illustrates a data flow diagram provided by related technologies, where subtask is a computation operator. When performing pairwise join fusion of multi-stream data in related technologies, each join operation selects data tables from two data sources for join. The upstream operator transmits the join result to the downstream operator through network shuffle. The downstream operator repeats the operation of the upstream operator. This network shuffle has a large network overhead and seriously affects system performance.

[0074] To address the aforementioned issues, in the three-layer network topology of this application embodiment, only one association at the data restoration layer is needed to obtain the final fused data, which greatly reduces the number of network shuffles for large amounts of data in long links.

[0075] Reference Figure 4 The diagram illustrates a data state processing method provided by related technologies. When performing pairwise join fusion of multi-stream data, each operator in the dual-stream join operation records information from the data streams on both sides to the state backend. The state needs to undergo serialization and deserialization. When the data volume is large, serializing a large amount of data will seriously affect the computing performance.

[0076] To address the aforementioned issues, the three-layer network topology of this application abandons the unidirectional chain-style association and fusion method in related technologies, greatly reducing the serialization operations of states caused by join operations.

[0077] Reference Figure 5 The diagram illustrates a data processing method provided by related technologies, where source represents the source data of the data source, operator represents the operator, and sink represents the calculation result. When performing pairwise joins of multi-stream data in related technologies, the keys participating in the join are shuffled according to a hash algorithm. Data with the same primary key (e.g., Msg2: LP01, LBX01, XX warehouse, attr1; Msg1: LP01, NULL, XX warehouse, attr2) are sent to the same downstream instance for association calculation. However, in multi-stream joins, foreign keys are often not unique. Therefore, data with the same primary key will be sent to different downstream operators. The calculation speed and network transmission of different operators vary (e.g., operatorr2#1 and operator2#2 have different calculation speeds and network transmissions). In actual processing, network issues can lead to data out-of-order phenomena (Msg2 and Msg1 are reversed compared to their original order).

[0078] To address the aforementioned issues, the three-layer network topology of this application abandons the unidirectional chain-style association and fusion method in related technologies. Data fusion is achieved through a single overall fusion operation, which greatly reduces the data disorder caused by multiple join operations.

[0079] Reference Figure 6 The diagram illustrates a data processing method provided by related technologies. When performing pairwise join and fusion of multi-stream data, in order to ensure the final accuracy of the data, rollback operations are continuously triggered during the calculation process. That is, a message is sent to eliminate the previously erroneously sent data. This rollback increases the amount of data in each layer of operators. As the number of data streams increases, the computational cost increases significantly. When resources are insufficient, back pressure may occur due to untimely processing by operators.

[0080] To address the aforementioned issues, the three-layer network topology of this application abandons the unidirectional chain-style association and fusion method in related technologies, thereby greatly reducing the large number of rollback operations caused by multiple join operations.

[0081] In summary, the multi-source data fusion method of this application embodiment utilizes the primary and foreign key matching relationship between real-time multi-source data. First, a real-time primary and foreign key relationship construction layer is developed to obtain the primary key of the final data output. Then, the standardized processing method of the fusion construction layer is used to integrate the content fields of the real-time multi-source data. Finally, the data streams of the first two layers are spliced ​​together and restored to the final fused data through the data restoration layer. This "split-then-merge" method can greatly reduce the network overhead and frequent state operations during data flow to a certain extent. It can improve the performance of real-time computing tasks, reduce resource consumption, and avoid data disorder problems. The optimized multi-source data fusion method of this application embodiment has a simple and clear chain, the data stream is processed uniformly, intermediate states are greatly reduced, and complete data can be obtained with only one association. The data flow under this fusion method is faster and has higher performance.

[0082] It should be noted that the multi-source data fusion method provided in this application can be implemented in several specific scenarios, as follows:

[0083] In one implementation, refer to Figure 7 This illustration shows a schematic diagram of multi-source data fusion in a logistics scenario provided by an embodiment of this application. It assumes that a parcel sequentially undergoes collection, sorting, transportation, and delivery services. The collection stage is used for centralized storage of parcels; the sorting stage is used to transport parcels to a distribution center, and calculates and arranges loading and transportation plans according to the destination and transportation routes of the receiving end, before implementation; the transportation stage is used to realize joint (relay) transportation between different or the same transportation modes; and the delivery stage is used to deliver the parcel to the receiving end. These four services can be implemented by the collection, sorting, transportation, and delivery service ends, respectively. This application embodiment utilizes a fusion server, with user authorization and relevant access permissions, to acquire source data generated by the receiving, sorting, transportation, and delivery services. The fusion server then uses the multi-source data fusion method of this application embodiment to obtain fused data reflecting the logistics lifecycle of a package. This fused data integrates the data from the receiving, sorting, transportation, and delivery services, and the fusion server can send the fused data to the client for analysis. Furthermore, the fused data can also be directly displayed on the fusion server, and its rich field content can meet users' complex statistical and analytical needs.

[0084] In another implementation, refer to Figure 8This illustration shows a multi-source data fusion diagram in an e-commerce scenario provided by an embodiment of this application. It assumes that the purchase of a product sequentially involves order generation, inventory query, quality inspection, and logistics services. These four services can be implemented by the order server, inventory server, quality inspection server, and logistics server, respectively. This embodiment of the application, with user authorization and relevant access permissions, can acquire the source data generated by each of the order server, inventory server, quality inspection server, and logistics server. The fusion server then uses the multi-source data fusion method of this embodiment to obtain fused data reflecting the product lifecycle. This fused data integrates the data from the order generation service, inventory service, quality inspection service, and logistics service. The fusion server can send the fused data to the client for analysis. Furthermore, the fused data can also be directly displayed on the fusion server, and the rich field content within the fused data can meet the user's complex statistical and analytical needs.

[0085] In another implementation, refer to Figure 9 This illustration shows a multi-source data fusion diagram in a recommendation scenario provided by an embodiment of this application. It assumes a user frequently browses and operates in four different sections of an application: section A, section B, section C, and section D. The functions of these four sections are implemented by servers A, B, C, and D, respectively. This embodiment of the application, with user authorization and relevant access permissions, can obtain source data related to the user's profile generated by each of servers A, B, C, and D. The fusion server then uses the multi-source data fusion method of this embodiment to obtain fused data reflecting the user's profile. This fused data integrates the user's profile data from sections A, B, C, and D. The fusion server can then send this fused data to the recommendation server for use, thereby improving recommendation accuracy.

[0086] It should be noted that the process of obtaining source data, primary key fields, foreign key fields, content fields, and other information, signals, or data used in the embodiments of this application is carried out in compliance with the relevant data protection laws and regulations of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0087] Reference Figure 10 It illustrates a flowchart of the steps of a multi-source data fusion method provided in an embodiment of this application, including:

[0088] Step 101: Obtain source data from multiple data sources; the source data includes primary key fields, foreign key fields, and content fields.

[0089] In this embodiment, the primary key is a unique identifier in the source data that can determine a record, and the foreign key is a parameter in the source data used to determine the association with another source data. The relationship between the primary key and foreign key in the source data is an important attribute for defining the uniqueness of each data table and representing the association between data tables. It is also an important attribute for realizing multi-source data fusion in this embodiment. Content fields are fields in the data table (data stream) other than the primary key field and the foreign key field. Content fields are specifically used to represent the content meaning of the data table.

[0090] Now through Figure 11 , 12 Section 13 explains the relationship between the source data of multiple data sources. Here, table represents a source data table, and pk and id are the values ​​of the primary key fields.

[0091] Figure 11 This reflects that one of the source data tables in a multi-source data table contains the association key that links all the source data tables, specifically reflecting that multiple source data tables are linked through the primary key (pk). Figure 12 This reflects that the join keys in the multi-source data table are distributed across multiple source data tables. Table2, Table3, and Table4 are foreign keys in source data table 1, which specifically reflects that the multiple source tables are linked through foreign keys and primary keys. Figure 13 This also reflects that the association keys in the multi-source data tables are distributed across multiple source data tables. Table2 is a foreign key of source data table1, Table3 is a foreign key of source data table2, and Table4 is a foreign key of source data table3. Specifically, this reflects that multiple source data tables are associated through foreign keys and primary keys.

[0092] Step 102: Based on the primary key field and foreign key field of the source data, perform association operations between multiple source data to obtain a first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation; the association key field includes the primary key field and foreign key field participating in the association operation.

[0093] In the embodiments of this application, reference is made to Figure 1 This solution constructs a layer by dividing the primary and foreign key relationships in the front, which can extract the primary and foreign keys of multiple source data and define the matching relationship between the primary and foreign keys of multiple source data.

[0094] The specific idea is to perform join operations (association operations) on each pair of multiple source data according to the order of acquisition time, based on primary key and foreign key fields. This generates two sets of data arranged in the order of data acquisition. One column is the primary key set, consisting of the first primary key field corresponding to each join operation. The other column is the first association key set, consisting of the association key fields, which are the primary keys and foreign keys involved in the join operation. The number of rows in these two sets corresponds to the number of primary key and foreign key types in the source data. The primary key set represents the unique primary key specified in each join operation, while the first association key set represents the association keys involved in each join operation. The primary keys in the primary key set and the association keys in the first association key set can establish a primary correspondence between them through relationships at the same level. Subsequent data restoration layers can use this primary correspondence to correctly and quickly merge the content fields of the source data under the corresponding primary keys. Since the data objects in these two data sets can be directly extracted from the original source data, and the amount of data in these two data sets is extremely small, the computational cost required for the primary-foreign key relationship construction layer is extremely low, and the resource consumption involved is almost negligible. As a lightweight front-end layer for processing source data, the primary-foreign key relationship construction layer has minimal negative impact on system load.

[0095] Step 103: Construct a second correspondence between the primary key field of the source data and the content field of the source data.

[0096] Reference Figure 1 The fusion construction layer can use a preset standard data structure to uniformly merge the content of source data from different data sources, and obtain a second correspondence between the primary key field and the content field of the source data. The fusion construction layer is designed to build a unified stream to solve the performance, cost and data accuracy problems caused by multi-stream association in related technologies.

[0097] This application embodiment aggregates standardized content fields into a unified data stream through a single fusion operation, which can reduce the network and system overhead of multiple join operations in related technologies (fusion methods that involve joining two source data as a whole), and greatly reduce data computing costs.

[0098] Step 104: Based on the matching of the primary key field of the source data in the second correspondence with the association key field in the first correspondence, construct the fused data by combining the target content field corresponding to the first primary key field and the target primary key field selected from the first primary key field.

[0099] Reference Figure 1The data restoration layer can utilize the data flow constructed in the first two layers to match the primary key field of the source data in the second correspondence with the association key field in the first correspondence, matching the content fields of the source data under the primary key of the final output of the fused data, thereby realizing the establishment of fused data in the form of a wide table. This fused data in the form of a wide table has a unique target primary key field determined from the primary key set, and under the target primary key field are content fields arranged in order of acquisition time, ensuring the correctness of the fused data.

[0100] In summary, the multi-source data fusion method of this application embodiment utilizes the primary and foreign key matching relationship between real-time multi-source data. First, based on the primary key field and foreign key field of the source data, it performs association operations between multiple source data to determine the first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation. Then, it uses a standardization processing method to construct a second correspondence between the primary key field and the content field of the source data. Finally, it uses data restoration to splice and restore the first two data streams into the final fused data. This method can greatly reduce the overall join operation between source data in related technologies, thereby reducing network overhead and frequent state operations during data flow, thus improving the performance of real-time computing tasks and reducing resource consumption. The optimized multi-source data fusion method of this application embodiment has a simple, clear, and easy-to-maintain chain. The data flow is processed uniformly, intermediate states are greatly reduced, and complete data can be obtained with only one association. The data flow under this fusion method is faster and has higher performance.

[0101] Reference Figure 14 It illustrates a flowchart of the specific steps of a multi-source data fusion method provided in an embodiment of this application, including:

[0102] Step 201: Obtain source data from multiple data sources; the source data includes primary key fields, foreign key fields, and content fields.

[0103] For details, please refer to step 101 above; it will not be repeated here.

[0104] Step 202: Based on the primary key field and foreign key field of the source data, perform consecutive association operations on two source data in the order of the acquisition time of the source data to obtain a primary key set composed of the first primary key field corresponding to the association operation, and a first association key set composed of the association key field.

[0105] The first primary key field is arranged in the order of the acquisition time; the related key field is arranged in the order of the acquisition time.

[0106] In this embodiment, join operations between pairs of source data can be performed sequentially according to the acquisition time of the source data, thereby ultimately determining the primary key set that uniquely determines the final output data, and the first association key set corresponding to the first primary key field in the primary key set. The primary key set is used to represent the unique primary key specified in each join operation, while the first association key set is used to represent the association keys participating in each join operation. The primary keys in the primary key set and the association keys in the first association key set can establish a first correspondence between them through the relationship of being at the same level. The subsequent data restoration layer can use this first correspondence to correctly and quickly merge the content fields of the source data under the corresponding primary key.

[0107] Optionally, step 202 may specifically include:

[0108] Sub-step 2021: Sort the source data according to the acquisition time of the source data to obtain the source data sequence.

[0109] Sub-step 2022: Starting from one end of the source data sequence, select two target source data and perform an association operation on the two target source data according to the primary key field and foreign key field of the target source data to obtain merged data. The merged data includes a first primary key field and at least one association key field.

[0110] Sub-step 2023 involves associating the merged data with the next source data in the source data sequence to obtain new merged data, and then proceeding to the step of associating the merged data with the next source data in the source data sequence to obtain new merged data, until the association of at least a portion of the source data in the source data sequence is completed.

[0111] For sub-steps 2021-2023, a specific example comparing them with a related technology and an embodiment of this application is now described:

[0112] Suppose there are 5 source data: source data 1-5, source data 1 includes primary key a and foreign key c, source data 2 includes primary key a and foreign key b, source data 3 includes primary key b and foreign key d, source data 4 includes primary key c, and source data 5 includes primary key d.

[0113] Reference Figure 15 The diagram illustrates a multi-source data fusion technique that uses a pairwise join of source data. The final output is: primary key a, foreign key b, foreign key c, foreign key d... field n, field n+1, field n+2... As can be seen, the output contains many other fields in addition to the sequentially arranged primary and foreign keys. The join operation is computationally intensive and has a high system overhead.

[0114] Reference Figure 16 This illustration shows a schematic diagram of the primary and foreign key relationship construction layer for multi-source data fusion according to an embodiment of this application. This embodiment can perform join operations on multiple source data in pairs based on primary key fields and foreign key fields, according to the order of their acquisition time, to obtain two columns of data: a primary key set and a first association key set. Specifically, the source data can be sorted according to the order of their acquisition time to obtain a source data sequence. Starting from one end of the source data sequence, two target source data are selected. Based on the primary key fields and foreign key fields of the target source data, a join operation is performed on the two target source data to obtain merged data. The merged data includes a first primary key field and at least one association key field. The merged data is then joined with the next source data in the source data sequence to obtain new merged data. This process continues until at least a portion of the source data in the source data sequence is associated.

[0115] Specifically, firstly, source data 1 and source data 2 undergo a first join operation, with primary key 'a' used as the first primary key field for the first join operation. Then, primary key 'a' is used as the join key to merge the two source data, resulting in merged data 1 (primary key a, foreign key c, foreign key b). Next, merged data 1 and source data 3 undergo a second join operation, with primary key 'a' used as the first primary key field for the second join operation. Then, foreign key 'b' is used as the join key to merge the two, resulting in merged data 2 (primary key a, foreign key c, foreign key b, foreign key d). Then, a third join operation is performed between merged data 2 and source data 4, using primary key 'a' as the first primary key field corresponding to the third join operation, and merging the two data using foreign key 'c' as the association key, resulting in merged data 3 (primary key a, foreign key c, foreign key b, foreign key d); finally, a fourth join operation is performed between merged data 3 and source data 5, using foreign key 'd' as the first primary key field corresponding to the fourth join operation, and merging the two data using foreign key 'd' as the association key, resulting in merged data 4 (primary key a, foreign key c, foreign key b, foreign key d).

[0116] Based on the above process, it can be seen that the primary key set includes primary key a, primary key a, primary key a, primary key a, primary key a, primary key a, arranged in the order of the acquisition time of the source data; the first association key set includes primary key a, foreign key b, foreign key c, foreign key d, arranged in the order of the acquisition time of the source data. The number of rows (4) of the primary key set and the first association key set can be determined by the types of keys (a, b, c, d).

[0117] Optionally, sub-step 2022 can be implemented by determining the first primary key field in the merged data corresponding to the association operation from the same key fields in the two target source data, and using the associated foreign key field as the association key field of the merged data.

[0118] In this embodiment of the application, each join operation can determine the first primary key field in the merged data corresponding to the join operation from the same key fields in the two target source data, and use the foreign key field for the join as the join key field of the merged data.

[0119] Since the data objects in these two data sets can be directly extracted from the original source data, and the amount of data in these two data sets is extremely small, the computational cost required for the primary-foreign key relationship construction layer is extremely low, and the resource consumption involved is almost negligible. As a lightweight front-end layer for processing source data, the primary-foreign key relationship construction layer has minimal negative impact on system load.

[0120] Step 203: Construct a second association key set consisting of the primary key fields of the source data, and a content set consisting of the content fields of the source data, wherein the primary key fields of the source data in the second association key set and the content fields of the source data in the content set have a second correspondence relationship.

[0121] In this context, the primary key fields in the second set of associated keys are arranged in the order of the acquisition time, and the content fields in the content set are arranged in the order of the acquisition time.

[0122] In this embodiment of the application, the fusion construction layer can use a preset standard data structure to uniformly merge the content of source data from different data sources to obtain two data sets. One column is a second association key set composed of the primary key fields of the source data, and the other column is a content set composed of the content fields of the source data. Based on whether they are in the same row, there can be a second correspondence between the primary key fields of the source data in the second association key set and the content fields of the source data in the content set.

[0123] The above process will now be described through a specific example:

[0124] Combination and Figure 16 The example uses the same source data: Source data 1-5, where source data 1 includes primary key a and foreign key c, source data 2 includes primary key a and foreign key b, source data 3 includes primary key b and foreign key d, source data 4 includes primary key c, and source data 5 includes primary key d. (See reference...) Figure 17This document illustrates a processing diagram of a fusion construction layer for multi-source data fusion according to an embodiment of this application. The second association key set can arrange the primary key fields of the source data according to the order of their acquisition time. The primary key fields in the second key set are used to match the key fields in the first association key set during subsequent data restoration. Similarly, the content set can also arrange the content fields of the source data according to the order of their acquisition time. The content fields are used to be associated with the unique primary key field of the fused data output in the primary key set through matching the primary key fields in the second key set and the key fields in the first association key set, thereby achieving the construction of fused data in a wide table format.

[0125] Optionally, step 203 can be implemented through sub-steps 2031-2033:

[0126] Sub-step 2031: Write the content fields of the source data into a preset data container to obtain a data container set.

[0127] Sub-step 2032: Construct a second set of association keys consisting of the primary key fields of the source data.

[0128] Sub-step 2033: Establish a second correspondence between the data containers in the data container set and the primary key fields in the second association key set.

[0129] In this embodiment of the application, the content fields of the source data can be written into a preset data container for storage, a data container set consisting of the source data data containers can be constructed, and a second correspondence relationship between the data containers in the data container set and the primary key fields in the second association key set can be established. In this way, the content fields of the source data can be standardized through the standard data structure defined by the data container itself, thereby reducing the difficulty of subsequent data processing.

[0130] Optionally, sub-step 2031 may specifically include:

[0131] Sub-step A1: Extract the field identifier and field value of the content field.

[0132] Sub-step A2: Write the field identifier of the source data into the key of the map data container, and write the field value of the source data into the corresponding key value of the map data container to obtain a data container set.

[0133] In a preferred implementation of this application, a map data container can be used to store the content fields of the source data. The map data container stores data in a key-value structure, where the key is the field identifier of the content field, and the value corresponding to the key is the field value of the content field. Storing content fields in this way can improve the standardization of content field storage and processing, and reduce the difficulty of subsequent processing of content fields. It should be noted that other forms of data containers can also be used to store content fields in this application embodiment, and this application embodiment is not limited to these.

[0134] Step 204: Based on the first correspondence and the second correspondence, and in the order of the acquisition time, match the primary key field of the source data in the second association key set with the association key field in the first association key set to obtain the target content field corresponding to the first primary key field.

[0135] Step 204 will now be described using a specific example:

[0136] Combination and Figure 16 , 17 The example uses the same source data: Source data 1-5, where source data 1 includes primary key a and foreign key c, source data 2 includes primary key a and foreign key b, source data 3 includes primary key b and foreign key d, source data 4 includes primary key c, and source data 5 includes primary key d. (See reference...) Figure 18 This illustration shows a processing diagram of a data restoration layer for multi-source data fusion according to an embodiment of this application. After matching the primary key field of the source data in the second association key set with the association key field in the first association key set, the content field of the source data can be mapped to the primary key field 'a' in the primary key set, obtaining multiple records corresponding to the primary key field 'a' (the content fields are refined and divided into content fields 1, 2, 3, 4, 5, 6, 7... according to the acquisition time of the source data), and obtaining the target content field corresponding to the first primary key field:

[0137] Record 1: Primary key a (corresponding associated key a) — Content field 3 — Content field 6 — Content field 7…

[0138] Record 2: Primary key a (corresponding associated key a) — Content field 1 — Content field 4 — Content field 7…

[0139] Record 3: Primary key a (corresponding to associated key b) — Content field 2 — Content field 3 — Content field 5 — Content field 6…

[0140] Record 4: Primary key a (corresponding associated key c) — Content field 1 — Content field 5 — Content field 7…

[0141] Record 5: Primary key a (corresponding associated key d) — Content field 1 — Content field 5 — Content field 7…

[0142] Step 205: Select the target primary key field from the first primary key field, and merge the target content field into the content field of the fused data according to the order of acquisition time to obtain the fused data.

[0143] In this step, refer to Figure 18 After obtaining the above 5 records, the target primary key field (primary key a) can be selected from the first primary key field (primary key a). That is, primary key a is used as the unique primary key of the merged data in the form of a wide table. According to the order of the acquisition time of the source data (from early to late or from late to early), the target content fields of each record are merged into the content fields of the merged data, resulting in the merged data: primary key a—content field 1—content field 2—content field 3—content field 4—content field 5—content field 6—content field 7….

[0144] Optionally, the method may further include:

[0145] Step 206: If the target primary key field corresponds to multiple duplicate target content fields, remove at least one duplicate target content field.

[0146] In this embodiment of the application, during the matching process between the primary key field of the source data in the second association key set and the association key field in the first association key set, a one-to-many situation may occur, which will cause data bloat, such as... Figure 18 In the five records, content fields 1, 3, 5, 6, and 7 appear repeatedly, resulting in inaccurate fused data. To solve the above problem, in this embodiment of the application, when the target primary key field corresponds to multiple duplicate target content fields, at least one duplicate target content field can be removed, that is, only one record of each of content fields 1, 3, 5, 6, and 7 needs to be kept.

[0147] Optionally, the method may further include:

[0148] Step 207: Sort the target content fields in the fused data according to a preset order.

[0149] In this embodiment of the application, preferably, the target content fields in the fused data can be sorted according to the order of the acquisition time of the source data. Of course, the target content fields can also be sorted according to a preset order, such as sorting according to factors such as data size and acquisition location, so as to realize the personalized display requirements of the fused data through the custom order arrangement.

[0150] Optionally, the method may further include:

[0151] Step 208: When the source data of the new data source is obtained, in response to the first update operation, add the first primary key field corresponding to the new association operation in which the new source data participates, and the association key field participating in the new association operation to the first correspondence relationship.

[0152] Step 209: If the source data of the new data source is obtained, in response to the second update operation, add the correspondence between the primary key field of the new source data and the content field of the new source data in the second correspondence.

[0153] In this embodiment, for steps 208-209, since this embodiment constructs a simple three-layer network topology (primary and foreign key relationship construction layer, fusion construction layer, and data restoration layer), and stores the primary and foreign key relationships and the relationship between keys and content fields of the source data through the lightweight foreign key relationship construction layer and fusion construction layer respectively, the overall design chain is simple, with strong reusability and ease of maintenance. If a new data source is introduced later, it is only necessary to establish the correspondence between the first primary key field corresponding to the new association operation in which the new source data participates and the association key field participating in the new association operation in the foreign key relationship construction layer, and to add the correspondence between the primary key field of the new source data and the content field of the new source data in the fusion construction layer, thereby incorporating the new data source into the fusion process. The entire process has a small amount of computation and system overhead.

[0154] Furthermore, by comparing related technologies with the computational topology of the present application's solution, the outstanding effects of the present application's solution are further demonstrated:

[0155] Reference Figure 19 It illustrates a computational topology diagram of the related technology, with reference to Figure 20 The diagram illustrates a computational topology diagram according to an embodiment of this application. Node 10 records the state of upstream real-time source data storage, and node 20 records the state of two upstream source data. Analysis and comparison show that the computational topology in related technologies has long and complex links and many state storage nodes, resulting in significant overhead. In contrast, the computational topology in this embodiment has simple and clear links and a large number of state storage nodes, making it more stable, easier to maintain, and with better performance.

[0156] Finally, a comparison of Tables 1 and 2 further demonstrates the superior advantages of this application's solution compared to related technologies:

[0157] Table 1 shows the computational costs required to implement the relevant technologies, and Table 2 shows the computational costs required to implement the solution in this application.

[0158] 202.98 Blocks / s 2536.14 messages / s 457.68 messages / s 4307181.02 bits / s 76.31

[0159] Table 1

[0160] 208.88 Blocks / s 3499.23 messages / s 658.78 messages / s 4572015.47 bits / s 29.41

[0161] Table 2

[0162] As can be seen, the proposed solution outperforms related technologies in terms of input system throughput, input speed, output speed, and input bit rate, while consuming only 29.41 quantization units, a reduction of 61.5% compared to related technologies, demonstrating a remarkable optimization effect.

[0163] In summary, the multi-source data fusion method of this application embodiment utilizes the primary and foreign key matching relationship between real-time multi-source data. First, based on the primary key field and foreign key field of the source data, it performs association operations between multiple source data to determine the first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation. Then, it uses a standardization processing method to construct a second correspondence between the primary key field and the content field of the source data. Finally, it uses data restoration to splice and restore the first two data streams into the final fused data. This method can greatly reduce the overall join operation between source data in related technologies, thereby reducing network overhead and frequent state operations during data flow, thus improving the performance of real-time computing tasks and reducing resource consumption. The optimized multi-source data fusion method of this application embodiment has a simple, clear, and easy-to-maintain chain. The data flow is processed uniformly, intermediate states are greatly reduced, and complete data can be obtained with only one association. The data flow under this fusion method is faster and has higher performance.

[0164] Reference Figure 21 It shows a block diagram of a multi-source data fusion device provided in an embodiment of this application, including:

[0165] The acquisition module 301 is used to acquire source data from multiple data sources; the source data includes primary key fields, foreign key fields, and content fields.

[0166] The first construction module 302 is used to perform association operations between multiple source data based on the primary key field and foreign key field of the source data, to obtain a first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation; the association key field includes the primary key field and foreign key field participating in the association operation.

[0167] The second construction module 303 is used to construct a second correspondence between the primary key field of the source data and the content field of the source data;

[0168] The merging module 304 is used to construct fused data by matching the primary key field of the source data in the second correspondence with the association key field in the first correspondence, and the target content field corresponding to the first primary key field and the target primary key field selected from the first primary key field.

[0169] Optionally, the first construction module 302 includes:

[0170] The association operation submodule is used to perform consecutive association operations on two source data in the order of their acquisition time, based on the primary key field and foreign key field of the source data.

[0171] Optionally, the associated operation submodule includes:

[0172] A sorting unit is used to sort the source data according to the acquisition time of the source data to obtain a source data sequence;

[0173] The association operation unit is used to select two target source data starting from one end of the source data sequence, and perform an association operation on the two target source data according to the primary key field and foreign key field of the target source data to obtain merged data. The merged data includes a first primary key field and at least one association key field.

[0174] The loop unit is used to associate the merged data with the next source data in the source data sequence to obtain new merged data, and then enter the step of associating the merged data with the next source data in the source data sequence to obtain new merged data, until the association of at least a portion of the source data in the source data sequence is completed.

[0175] Optionally, the associated operation unit includes:

[0176] The association subunit is used to determine the first primary key field in the merged data corresponding to the association operation from the same key fields in the two target source data, and to associate the foreign key field as the association key field of the merged data.

[0177] Optionally, in the primary key set consisting of the first primary key fields corresponding to the association operation, the first primary key fields are arranged in order of acquisition time;

[0178] In the first set of association keys composed of the association key fields, the association key fields are arranged in order of acquisition time.

[0179] Optionally, the second building module 303 includes:

[0180] A construction submodule is used to construct a second association key set consisting of the primary key fields of the source data, and a content set consisting of the content fields of the source data, wherein the primary key fields of the source data in the second association key set and the content fields of the source data in the content set have a second correspondence relationship;

[0181] In this context, the primary key fields in the second set of associated keys are arranged in the order of the acquisition time, and the content fields in the content set are arranged in the order of the acquisition time.

[0182] Optionally, the merging module 304 includes:

[0183] The matching submodule is used to match the primary key field of the source data in the second association key set with the association key field in the first association key set based on the first correspondence relationship and the second correspondence relationship, according to the order of the acquisition time, to obtain the target content field corresponding to the first primary key field;

[0184] The fusion submodule is used to select a target primary key field from the first primary key field, and merge the target content field into the content field of the fused data according to the order of the acquisition time, so as to obtain the fused data.

[0185] Optionally, the second building module 303 includes:

[0186] The write submodule is used to write the content fields of the source data into a preset data container to obtain a set of data containers;

[0187] The collection construction submodule is used to construct a second set of association keys composed of the primary key fields of the source data;

[0188] The relationship building submodule is used to establish a second correspondence between the data containers in the data container set and the primary key fields in the second association key set.

[0189] Optionally, the writing submodule includes:

[0190] An extraction unit is used to extract the field identifier and field value of the content field;

[0191] The writing unit is used to write the field identifier of the source data into the key of the map data container, and to write the field value of the source data into the corresponding key value of the map data container, so as to obtain a data container set.

[0192] Optionally, the device further includes:

[0193] The deduplication module is used to remove at least one duplicate target content field when the target primary key field corresponds to multiple duplicate target content fields.

[0194] Optionally, the device further includes:

[0195] The sorting module is used to sort the target content fields in the fused data according to a preset order.

[0196] Optionally, the device further includes:

[0197] The first addition module is used to, in response to the first update operation, add the first primary key field corresponding to the new association operation in which the new source data participates, and the association key field participating in the new association operation to the first correspondence in the first correspondence when the source data of the new data is obtained;

[0198] The second addition module is used to, in response to the second update operation, add the primary key field of the new source data and the correspondence between the content field of the new source data to the second correspondence when the source data of the new data source is obtained.

[0199] In summary, the multi-source data fusion method of this application embodiment utilizes the primary and foreign key matching relationship between real-time multi-source data. First, based on the primary key field and foreign key field of the source data, it performs association operations between multiple source data to determine the first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation. Then, it uses a standardization processing method to construct a second correspondence between the primary key field and the content field of the source data. Finally, it uses data restoration to splice and restore the first two data streams into the final fused data. This method can greatly reduce the overall join operation between source data in related technologies, thereby reducing network overhead and frequent state operations during data flow, thus improving the performance of real-time computing tasks and reducing resource consumption. The optimized multi-source data fusion method of this application embodiment has a simple, clear, and easy-to-maintain chain. The data flow is processed uniformly, intermediate states are greatly reduced, and complete data can be obtained with only one association. The data flow under this fusion method is faster and has higher performance.

[0200] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0201] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and server (cluster) devices.

[0202] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including terminal devices, server (cluster) and other electronic devices. Figure 22 An exemplary apparatus 1000 is schematically shown that can be used to implement the various embodiments described in the embodiments of this application.

[0203] In one embodiment, Figure 22 An exemplary device 1000 is shown, which includes one or more processors 1002, a control module (chipset) 1004 coupled to at least one of the processors 1002, a memory 1006 coupled to the control module 1004, a non-volatile memory (NVM) / storage device 1008 coupled to the control module 1004, one or more input / output devices 1010 coupled to the control module 1004, and a network interface 1012 coupled to the control module 1004.

[0204] Processor 1002 may include one or more single-core or multi-core processors, and processor 1002 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1000 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0205] In some embodiments, apparatus 1000 may include one or more computer-readable media (e.g., memory 1006 or NVM / storage device 1008) having instructions 1014 and one or more processors 1002 that are combined with the one or more computer-readable media and configured to execute instructions 1014 to implement modules and thereby perform the actions described in this disclosure.

[0206] In one embodiment, the control module 1004 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1002 and / or any suitable device or component communicating with the control module 1004.

[0207] The control module 1004 may include a memory controller module to provide an interface to the memory 1006. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0208] Memory 1006 may be used, for example, to load and store data and / or instructions 1014 for device 1000. In one embodiment, memory 1006 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 1006 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0209] In one embodiment, the control module 1004 may include one or more input / output controllers to provide interfaces to the NVM / storage device 1008 and (one or more) input / output devices 1010.

[0210] For example, NVM / storage device 1008 may be used to store data and / or instructions 1014. NVM / storage device 1008 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0211] NVM / storage device 1008 may include storage resources that are physically part of a device on which device 1000 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 1008 may be accessed via a network via one or more input / output devices 1010.

[0212] One or more input / output devices 1010 may provide an interface for device 1000 to communicate with any other suitable device. Input / output devices 1010 may include communication components, audio components, sensor components, etc. Network interface 1012 may provide an interface for device 1000 to communicate via one or more networks. Device 1000 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0213] In one embodiment, at least one of the processors 1002 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1004. In one embodiment, at least one of the processors 1002 may be logically packaged with one or more controllers of the control module 1004 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1002 may be integrated with the logic of one or more controllers of the control module 1004 on the same die. In one embodiment, at least one of the processors 1002 may be integrated with the logic of one or more controllers of the control module 1004 on the same die to form a system-on-a-chip (SoC).

[0214] In various embodiments, device 1000 may be, but is not limited to, a server, a desktop computing device, or a mobile computing device (e.g., a laptop, a handheld computing device, a tablet, a netbook, etc.). In various embodiments, device 1000 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1000 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0215] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0216] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0217] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0218] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0219] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0220] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0221] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0222] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0223] The foregoing has provided a detailed description of the multi-source data fusion method, lower body posture prediction method, modeling and display method and device in virtual reality scenes, electronic device, and machine-readable medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-source data fusion method, characterized in that, include: Retrieve source data from multiple data sources; The source data includes primary key fields, foreign key fields, and content fields; Based on the primary key field and foreign key field of the source data, perform association operations between multiple source data to obtain a first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation; The association key field includes the primary key field and foreign key field that participate in the association operation; Construct a second correspondence between the primary key field of the source data and the content field of the source data; Based on the matching of the primary key field of the source data in the second correspondence with the association key field in the first correspondence, the target content field corresponding to the first primary key field and the target primary key field selected from the first primary key field are used to construct fused data. The association operation includes: determining the same key field from the two source data, using the same key field as the association key, and merging the two source data according to the association key to obtain merged data.

2. The method according to claim 1, characterized in that, The operation of associating multiple sets of source data based on the primary key and foreign key fields of the source data includes: Based on the primary key and foreign key fields of the source data, the association operations of the two source data are performed sequentially according to the acquisition time of the source data.

3. The method according to claim 2, characterized in that, The method of performing consecutive association operations on two sets of source data based on the primary key and foreign key fields of the source data, in chronological order of the acquisition time of the source data, includes: The source data is sorted according to the acquisition time to obtain a source data sequence; Starting from one end of the source data sequence, select two target source data. Based on the primary key field and foreign key field of the target source data, perform an association operation on the two target source data to obtain merged data. The merged data includes a first primary key field and at least one association key field. The merged data is associated with the next source data in the source data sequence to obtain new merged data, and then the process continues until the association of at least a portion of the source data in the source data sequence is completed.

4. The method according to claim 3, characterized in that, The step of performing a join operation on the two target source data according to the primary key field and foreign key field of the target source data to obtain merged data includes: From the same key fields in the two target source data, determine the first primary key field in the merged data that corresponds to the association operation, and use the associated foreign key field as the association key field of the merged data.

5. The method according to claim 2, characterized in that, In the primary key set consisting of the first primary key fields corresponding to the association operation, the first primary key fields are arranged in order of acquisition time; In the first set of association keys composed of the association key fields, the association key fields are arranged in order of acquisition time.

6. The method according to claim 5, characterized in that, The construction of the second correspondence between the primary key field and the content field of the source data includes: Construct a second association key set consisting of the primary key fields of the source data, and a content set consisting of the content fields of the source data, wherein the primary key fields of the source data in the second association key set and the content fields of the source data in the content set have a second correspondence relationship; In this context, the primary key fields in the second set of associated keys are arranged in the order of the acquisition time, and the content fields in the content set are arranged in the order of the acquisition time.

7. The method according to claim 6, characterized in that, The matching of the primary key field of the source data in the second correspondence with the association key field in the first correspondence, and the resulting target content field corresponding to the first primary key field, along with the target primary key field selected from the first primary key field, are used to construct fused data, including: Based on the first and second correspondences, and in the order of acquisition time, the primary key field of the source data in the second association key set is matched with the association key field in the first association key set to obtain the target content field corresponding to the first primary key field; Select a target primary key field from the first primary key field, and merge the target content fields into the content fields of the fused data according to the order of acquisition time to obtain the fused data.

8. A multi-source data fusion device, characterized in that, include: The acquisition module is used to acquire source data from multiple data sources. The source data includes primary key fields, foreign key fields, and content fields; The first construction module is used to perform association operations between multiple source data based on the primary key field and foreign key field of the source data, and obtain a first correspondence between the first primary key field corresponding to the association operation and the association key field participating in the association operation; The association key field includes the primary key field and foreign key field that participate in the association operation; The association operation includes: determining the same key field from the two source data, using the same key field as the association key, and merging the two source data according to the association key to obtain merged data; The second construction module is used to construct a second correspondence between the primary key field of the source data and the content field of the source data; The merging module is used to match the primary key field of the source data in the second correspondence with the association key field in the first correspondence, and to construct the fused data by combining the target content field corresponding to the first primary key field and the target primary key field selected from the first primary key field.

9. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as described in any one of claims 1 to 7.

10. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method as claimed in any one of claims 1 to 7.

Images