Multi-source heterogeneous data fusion method, device and storage medium

Abstract

The present application discloses a multi-source heterogeneous data fusion method, device and storage medium, and relates to the field of data processing technology. The method includes: obtaining a multi-source heterogeneous data set of a business, and determining all data fields in the multi-source heterogeneous data set; for any data field, determining a first similarity score and a second similarity score between the data field and each base field of the business, and determining an associated base field associated with the data field according to each first similarity score and each second similarity score; wherein the first similarity score is a semantic similarity score between the data field and each base field, and the second similarity score is a field value similarity score between the data field and each base field; according to each data field and the corresponding associated base field, the multi-source heterogeneous data set is subjected to data fusion to obtain a data fusion result. In order to realize standardized and automated data fusion of multi-source heterogeneous data, the processing efficiency of multi-source heterogeneous data is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-source heterogeneous data fusion method, device and storage medium. Background Art

[0002] With the informatization construction of enterprises, data from different fields or entities are becoming more and more massive. However, due to the lack of unified specifications and standards, these data are often in various formats and structures, resulting in many problems such as information islands and low data interaction efficiency, which is not conducive to the informatization upgrade of enterprises.

[0003] At present, the processing methods of these multi-source heterogeneous data often rely on manual analysis and annotation of the data, and then develop corresponding data interfaces and other methods to carry out data fusion. However, this method of fusing multi-source heterogeneous data is inefficient, difficult to adapt to the environmental requirements of rapidly changing data standards, and unable to tap the potential of big data in the fields of intelligent decision-making and intelligent prediction. Summary of the invention

[0004] The main purpose of this application is to provide a multi-source heterogeneous data fusion method, device and storage medium to achieve standardized and automated data fusion of multi-source heterogeneous data and improve the processing efficiency of multi-source heterogeneous data.

[0005] To achieve the above objectives, the present application provides a multi-source heterogeneous data fusion method, comprising:

[0006] Acquire a multi-source heterogeneous data set of a business, and determine all data fields in the multi-source heterogeneous data set;

[0007] For any data field, determine a first similarity score and a second similarity score between the data field and each base field of the business, and determine an associated base field associated with the data field according to each of the first similarity scores and each of the second similarity scores; wherein the first similarity score is a semantic similarity score between the data field and each of the base fields, and the second similarity score is a field value similarity score between the data field and each of the base fields;

[0008] According to each of the data fields and the corresponding associated base fields, the multi-source heterogeneous data sets are fused to obtain a data fusion result.

[0009] Optionally, for any data field, determining a first similarity score between the data field and each base field of the business includes: inputting the data field into a pre-trained semantic similarity model to obtain the first similarity score between the data field and each base field, and the semantic similarity model is used to compare the semantic similarity of the data field with each base field to obtain the corresponding first similarity score.

[0010] Optionally, for any data field, determining the second similarity score between the data field and each basis field of the business includes: determining all field values ​​of the data field; randomly extracting a preset amount of field values ​​from all the field values, and inputting the extracted field values ​​into a pre-trained encoder model to obtain a first feature vector corresponding to each field value; for any field value, determining an intermediate similarity score between the data field value and each basis field value based on the first feature vector of the data field value and the second feature vector of each basis field value, the second feature vector being obtained by performing feature extraction on the basis field value using the pre-trained encoder model; for any basis field, averaging the intermediate similarity scores between each field value and the basis field value to obtain the second similarity score between the data field and the basis field.

[0011] Optionally, determining the associated base field associated with the data field according to each of the first similarity scores and each of the second similarity scores includes: for any base field, performing a weighted operation on the first similarity score and the second similarity score between the data field and the base field to obtain a total score of the data field and the base field; and determining the associated base field from all of the base fields according to the total score of the data field and each of the base fields.

[0012] Optionally, the multi-source heterogeneous data sets of the business include at least two groups. Before determining all data fields in the multi-source heterogeneous data sets, the method also includes: setting a standard field set for the business, the standard field set including at least one standard field information required by the business; determining a target multi-source heterogeneous data set from all the multi-source heterogeneous data sets according to the standard field set; determining all target fields in the target multi-source heterogeneous data set; for any target field, when the target field is semantically related to the corresponding standard field information, using the target field as the base field corresponding to the standard field information.

[0013] Optionally, after performing data fusion on the multi-source heterogeneous data sets to obtain a data fusion result, the method further includes: for any data field, determining the standard field information corresponding to the associated base field, and using the data field as the base field corresponding to the standard field information.

[0014] Optionally, the acquiring of a multi-source heterogeneous data set of a business includes: setting a data collection scheme and data standard specifications for the business; and extracting the multi-source heterogeneous data set from each system according to the data collection scheme and the data standard specifications.

[0015] Optionally, the multi-source heterogeneous data set is extracted from each system according to the data acquisition plan and the data standard specification, including: cleaning and preprocessing the data in the multi-source heterogeneous data set according to the data standard specification; converting the preprocessed multi-source heterogeneous data set into a common file format and storing it.

[0016] The present application also provides a multi-source heterogeneous data fusion device, including: a data acquisition module, used to acquire a multi-source heterogeneous data set of a business, and determine all data fields in the multi-source heterogeneous data set; a similarity comparison module, used to determine, for any data field, a first similarity score and a second similarity score between the data field and each base field of the business, and determine an associated base field associated with the data field according to each of the first similarity scores and each of the second similarity scores; wherein the first similarity score is a semantic similarity score between the data field and each of the base fields, and the second similarity score is a field value similarity score between the data field and each of the base fields; a data fusion module, used to perform data fusion on the multi-source heterogeneous data set according to each of the data fields and the corresponding associated base fields to obtain a data fusion result.

[0017] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-source heterogeneous data fusion method as described in any one of the above is implemented.

[0018] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-source heterogeneous data fusion method described above is implemented.

[0019] The multi-source heterogeneous data fusion method provided by the present application obtains a multi-source heterogeneous data set of a business and determines all data fields in the multi-source heterogeneous data set; then obtains a first similarity score by comparing the semantic similarity of the data field with each base field of the business, and obtains a second similarity score by comparing the field value of the data field with the field value of each base field of the business; then determines the associated base field associated with the data field through the first similarity score and the second similarity score; finally, the multi-source heterogeneous data is data-fused according to the association relationship between the data field and the associated base field, thereby unifying and fusion of the data corresponding to all similar data fields based on the associated base field, realizing standardized and automated data fusion of the multi-source heterogeneous data and improving the processing efficiency of the multi-source heterogeneous data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is one of the flow charts of the multi-source heterogeneous data fusion method according to an embodiment of the present application;

[0021] Figure 2 This is the second flowchart of the multi-source heterogeneous data fusion method of the embodiment of the present application;

[0022] Figure 3 This is the third flowchart of the multi-source heterogeneous data fusion method of the embodiment of the present application;

[0023] Figure 4 This is the fourth flowchart of the multi-source heterogeneous data fusion method of the embodiment of the present application;

[0024] Figure 5 This is the fifth flowchart of the multi-source heterogeneous data fusion method of the embodiment of the present application;

[0025] Figure 6 This is the sixth flowchart of the multi-source heterogeneous data fusion method of the embodiment of the present application;

[0026] Figure 7 It is a structural schematic diagram of a multi-source heterogeneous data fusion device according to an embodiment of the present application;

[0027] Figure 8 An example physical structure diagram of an electronic device is provided.

[0028] In the figure: 700, multi-source heterogeneous data fusion device; 710, data acquisition module; 720, similarity comparison module; 730, data fusion module; 810, processor; 820, communication interface; 830, memory; 840, communication bus.

[0029] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] Multi-source heterogeneous data refers to data that comes from multiple different data sets with different storage modes, logical structures, and standards. In the business work of the civil aviation industry, in order to solve specific business problems or meet the needs of specific business data statistics and viewing, it is usually necessary to extract relevant business data from different systems for viewing or processing. However, the data stored in different systems may have different characteristics such as data format, type, and structure. Therefore, most of the business data extracted from different systems are multi-source heterogeneous data.

[0032] At present, the main way to process this kind of multi-source heterogeneous data is to manually analyze and process the data, and then develop corresponding data interfaces to fuse and send the processed data. However, this manual processing method is inefficient, and data utilization requires customized development, which is costly; in addition, data standards are also in a dynamic state of change. Therefore, this processing method is also difficult to adapt to the basic needs of rapid changes in data standards, which is not conducive to data sharing within the industry to give full play to the role of big data in intelligent decision-making, intelligent prediction and other fields.

[0033] To this end, the embodiments of the present application provide a multi-source heterogeneous data fusion method, device and storage medium, which automatically learn and extract key features in multi-source heterogeneous data through a neural network algorithm, and realize adaptive fusion and intelligent analysis of data, thereby realizing standardized and automated multi-source heterogeneous data fusion and improving the efficiency and accuracy of multi-source heterogeneous data processing.

[0034] Figure 1 FIG. 1 is one of the flow charts of the multi-source heterogeneous data fusion method of the embodiment of the present application. Figure 1 As shown, the multi-source heterogeneous data fusion method may include the following steps:

[0035] Step 110: Acquire a multi-source heterogeneous data set of a business, and determine all data fields in the multi-source heterogeneous data set.

[0036] Step 120: For any data field, determine a first similarity score and a second similarity score between the data field and each base field of the business, and determine an associated base field associated with the data field according to each first similarity score and each second similarity score.

[0037] Step 130: According to each data field and the corresponding associated base field, the multi-source heterogeneous data sets are fused to obtain a data fusion result.

[0038] The first similarity score is a semantic similarity score between the data field and each base field, and the second similarity score is a field value similarity score between the data field and each base field.

[0039] In this embodiment, the required business data can be extracted from the databases of various systems according to actual business needs, and the extracted business data constitute a multi-source heterogeneous data set. For example, in the aircraft fault management business, the fault management business may need data related to the fault title, fault code, location, and stage of occurrence, and then the fault title data, fault code data, location data, and stage of occurrence data can be extracted from the databases of various systems according to the business needs.

[0040] It should be noted that a data field is an indivisible data unit in a relational database. It is used to describe the attributes of an entity object, which is equivalent to a "column" in a two-dimensional table. Therefore, when extracting relevant data from various system databases, the corresponding data and the data field corresponding to the data can be retrieved from the database according to the field information required by the business. For example, "fault code" is a field information, which only contains business meaning and does not contain actual data.

[0041] After extracting the multi-source heterogeneous data and the data fields corresponding to the data required for the business from the databases of various systems, the data fields are compared and analyzed with the base fields of the business one by one to determine the association between the data fields and the base fields, so as to fuse the data corresponding to the data fields with the same or similar meanings.

[0042] It should be noted that the basic fields of a business are a set of standard field examples determined based on the actual needs of the business. For example, the field information required for the fault management business is the fault code, location, occurrence stage, and fault handling status. The basic fields of the business may include "FaultCode", "Place", "Occurrence stage", and "Faulthandling status".

[0043] Taking a data field as an example, the process of comparing and analyzing the data field with the base fields of the business can be as follows: first, the data field is compared with each base field for semantic similarity to obtain semantic similarity scores between the multiple data fields and the base field (i.e., a first similarity score); then, all field values ​​corresponding to the data field are compared with the field values ​​of each base field for similarity to obtain field value similarity scores between the multiple data fields and the base field (i.e., a second similarity score).

[0044] It should be noted that the field value of a data field refers to the attribute value of the entity object corresponding to the data field. For example, in the "Address Book" database, "Name", "Contact Number", etc. are data fields, and the specific information contained in these data fields, such as a person's specific name or contact number, is the field value of these data fields. Therefore, by comparing the field value-related attributes of the data field and the base field (for example, field value length, field value content, etc.), it is also possible to assist in determining the association between the data field and the base field.

[0045] After obtaining the first similarity score and the second similarity score of the data field and each base field, the base field most relevant to the data field can be found in all base fields according to the first similarity score and the second similarity score, that is, the associated base field. After determining the associated base field corresponding to all data fields in the same way, the data corresponding to all data fields with the same associated base field can be fused. Because the associated base field is the same, it means that the business meanings of these data fields are the same, so the corresponding data can be fused to obtain a data fusion result.

[0046] It should be noted that the data fusion result may be a business data relationship table, which includes the base field of the business and the data of all data fields corresponding to the base field.

[0047] Therefore, based on the unified business base field, all data fields in the multivariate heterogeneous data group are compared with each base field for semantic similarity and field value similarity, so as to find the associated base field that is most associated with the data field; then the data corresponding to all data fields with the same associated base field are fused, realizing standardized and automated fusion of multi-source heterogeneous data, and improving the efficiency and accuracy of multi-source heterogeneous data processing.

[0048] Figure 2 This is the second flow chart of the multi-source heterogeneous data fusion method of the embodiment of the present application. Figure 2 As shown, in some implementations, obtaining a multi-source heterogeneous data set of a business in step 110 may include the following steps:

[0049] Step 210: Set the data collection plan and data standard specifications for the business.

[0050] Step 220: Extract multi-source heterogeneous data sets from various systems according to the data collection plan and data standard specifications.

[0051] Specifically, we can first clarify the specific scope of the business-related data, that is, which data in the database of each system is related to the business. Then, based on the actual demand for data in the enterprise's specific production business, we can determine the data types required by the current business among all relevant data types, so as to formulate a unified data collection plan for different business needs.

[0052] For example, the current business needs to solve the problem of aircraft fault management. To meet this demand, the data required by the business include fault title data, fault code data, occurrence location data and occurrence stage data. After clarifying the required data type, the data collection plan can be formulated as: collecting all data in the fields related to the fault title, fault code, occurrence location and occurrence stage.

[0053] Similarly, since the data comes from databases of different systems, the types, structures, and formats of these data are not uniform. Based on the required data types determined above, we can first find out which systems these data may come from, and the format, structure, and other characteristics of the data stored in the databases of these systems, and then refer to all the format characteristics and structural characteristics to formulate a unified data standard specification, so that all data formats can be unified during subsequent data fusion.

[0054] After determining the data collection plan for the business, data with the same or similar business meanings can be extracted from the databases of each system according to the data collection plan to obtain a multi-source heterogeneous data set. It should be noted that the data collection plans and data standard specifications formulated for different businesses may be different.

[0055] Figure 3 This is the third flow chart of the multi-source heterogeneous data fusion method of the embodiment of the present application. Figure 3 As shown, in some implementations, extracting a multi-source heterogeneous data set from each system according to a data collection scheme and data standard specifications in step 220 may include the following steps:

[0056] Step 310: Clean and preprocess the data in the multi-source heterogeneous data set according to data standard specifications.

[0057] Step 320: Convert the preprocessed multi-source heterogeneous data set into a common file format and store it.

[0058] Specifically, after obtaining a multi-source heterogeneous data set, data cleaning can be performed on the data in the multi-source heterogeneous data set to remove incomplete, inaccurate or incorrectly formatted data to ensure the quality and accuracy of the data. In this embodiment, the data cleaning method may include: processing missing values, deleting duplicates, processing outliers, format and type conversion, data normalization, etc. The specific data cleaning process can refer to the existing data cleaning process, which will not be repeated here.

[0059] Furthermore, the multi-source heterogeneous data set after data cleaning is preprocessed, and the format of the multi-source heterogeneous data can be unified according to the pre-established data standard specifications. For example, the data standard specifications can specify the field naming, data type, and data encoding specifications of the business-related data. Based on the data standard specifications, the field naming, data type, and data encoding of the multi-source heterogeneous data are uniformly modified, so that a set of multi-source heterogeneous data with a unified data format can be obtained.

[0060] It should be noted that the data used in the subsequent correlation comparison are all data that have been cleaned and preprocessed, so as to ensure the uniformity of the data format in the obtained data fusion results.

[0061] Furthermore, a set of multi-source heterogeneous data sets can be converted into a file format common to each system, and the multi-source heterogeneous data sets can be pre-stored in the form of files, so as to facilitate subsequent data retrieval. Here, the common file format can be: CSV file format, Excel file format, JSON file format, etc., which is not specifically limited here.

[0062] For example, the extracted multi-source heterogeneous dataset can be further converted into a common CSV file format through Python and other languages, and the multi-source heterogeneous dataset is stored in the form of a CSV file. When the multi-source heterogeneous dataset is subsequently retrieved, it is also retrieved in the form of a CSV file.

[0063] In order to reduce the difficulty of data fusion and improve the consistency of multi-source heterogeneous data, in this embodiment, corresponding base fields are also configured for different services. The specific configuration process is as follows:

[0064] Figure 4 This is the fourth flow chart of the multi-source heterogeneous data fusion method of the embodiment of the present application. Figure 4 As shown, in some implementations, multiple sets of multi-source heterogeneous data sets may be obtained. Before determining all data fields in the multi-source heterogeneous data sets in step 110, the multi-source heterogeneous data fusion method may further include the following steps:

[0065] Step 410: Set a standard field set for a service, where the standard field set includes standard field information required by at least one service.

[0066] Step 420: Determine a target multi-source heterogeneous data set from all multi-source heterogeneous data sets according to the standard field set.

[0067] Step 430: Determine all target fields in the target multi-source heterogeneous data set.

[0068] Step 440: For any target field, if the target field is semantically related to the corresponding standard field information, use the target field as the base field corresponding to the standard field information.

[0069] It should be noted that the obtained multiple sets of multi-source heterogeneous data sets can be stored in the form of CSV files respectively.

[0070] Specifically, the standard field set of the business can be set according to specific business needs. The standard field set can include all standard field information required by the business. The standard field information only represents the meaning of the relevant business, not the specific data. It should be noted that the standard field set of the business can be set manually or automatically according to the needs through deep learning, and no specific restrictions are made here.

[0071] As an example, if the business is aircraft fault management business, the set standard field set may include: fault title, fault code, occurrence location and occurrence stage; among which, the fault title, fault code, occurrence location and occurrence stage are all standard field information.

[0072] After setting the standard field set for the business, you can select a CSV file that is most similar to the standard field information in the standard field set from all CSV files based on the standard field set, and use the multi-source heterogeneous dataset in the CSV file as the target multi-source heterogeneous dataset. It should be noted that the process of selecting the CSV file that is most similar to the standard field set can be done manually or automatically through deep learning, and there is no specific restriction here.

[0073] Furthermore, after determining the target multi-source heterogeneous data set, all target fields in the target multi-source heterogeneous data set are determined, and the target fields can be extracted synchronously when extracting the target multi-source heterogeneous data set. After all target fields are determined, the semantic similarity model can be used to compare the semantic similarity of the target fields with each standard field information one by one, so as to make a preliminary correlation judgment between the target fields and the standard field information.

[0074] Specifically, the semantic similarity model can adopt the currently available semantic similarity models, such as the word vector cosine similarity model based on the Word2Vec algorithm, the semantic matching model based on BERT, etc. The semantic similarity of the target field is compared with each standard field information through the semantic similarity model, and the standard field information with the most similar semantics to the target field is found from all the standard field information.

[0075] In this embodiment, the target field and the standard field information most similar to it determined by the semantic similarity model can also be pushed to the user, and the user can further perform correlation judgment to ensure the accuracy of the configured base field. Specifically, the target field and the standard field information most similar to it can be sent to the user terminal, and the user can view the recommended target field and the standard field information most similar to it through the user terminal. If the user believes that the target field is related to the standard field information, then after the user confirms, a preliminary mapping between the target field and the standard field information is established, and the target field is used as the base field corresponding to the standard field information.

[0076] As an example, if the standard field information is the "fault code" field, the semantic similarity model is used to determine from the data fields in the target multi-source heterogeneous data set that the "FaultCode" field is most similar to the standard field information. Then the "fault code" field and the "FaultCode" field can be output to the user terminal. After the user confirms, a preliminary mapping relationship between the "fault code" field and the "FaultCode" field is established.

[0077] In some embodiments, determining the first similarity score between the data field and each base field of the business for any data field in step 120 may include: inputting the data field into a pre-trained semantic similarity model to obtain a first similarity score between the data field and each base field, the semantic similarity model being used to compare the semantic similarity of the data field with each base field respectively to obtain a corresponding first similarity score.

[0078] It should be noted that the semantic similarity model can be an existing semantic similarity model, for example, a word vector cosine similarity model based on the Word2Vec algorithm, a semantic matching model based on BERT, etc., and no specific restrictions are made here; in addition, the model structure and training method of the semantic similarity model can also refer to the existing model structure and training method, which will not be elaborated here.

[0079] Specifically, all data fields in the multi-source heterogeneous data set imported into the business are traversed, and the semantic similarity of each data field is compared one by one using the semantic similarity model. The following takes a data field as an example to introduce in detail the process of determining the first similarity score.

[0080] The data field is input into the semantic similarity model, and the semantic similarity model calculates the semantic similarity between the data field and each base field respectively, and then scores based on the semantic similarity. The higher the semantic similarity, the higher the first similarity score between the data field and the base field. In this embodiment, all first similarity scores of the same data field can be pre-saved in the form of a set.

[0081] As an example, the base fields of the business include base field A and base field B, and the multi-source heterogeneous data set includes data field 1. When data field 1 is input into the semantic similarity model, the first similarity score a between data field 1 and base field A can be obtained. （字段A-字段1） , and the first similarity score a between data field 1 and base field B （字段B-字段1） , a （字段A-字段1） with a （字段B-字段1） The first similarity score set a that can constitute data field 1 （字段1） .

[0082] Figure 5 This is the fifth flow chart of the multi-source heterogeneous data fusion method of the embodiment of the present application. Figure 5 As shown, in some implementations, determining the second similarity score between any data field and each base field of the business in step 120 may include the following steps:

[0083] Step 510: Determine all field values ​​of the data field.

[0084] Step 520: Randomly extract a preset amount of field values ​​from all field values, and input the extracted field values ​​into a pre-trained encoder model to obtain a first feature vector corresponding to each field value.

[0085] Step 530: For any field value, determine the intermediate similarity score between the data field value and the field values ​​of each basis field according to the first feature vector of the data field value and the second feature vector of the field values ​​of each basis field.

[0086] Step 540: For any base field, average the intermediate similarity scores of each field value and the base field to obtain a second similarity score between the data field and the base field.

[0087] It should be noted that the field value of a data field refers to the attribute value of the entity object corresponding to the data field. Under a data field, there may be several field values. Therefore, when determining the second similarity score, only a preset number of field values ​​can be extracted to compare the similarity with the field value of the base field, thereby reducing the amount of data calculation. In addition, the preset number can be set manually according to actual needs. If the performance of the computing hardware is sufficient, more field values ​​can be extracted for calculation to obtain a more accurate second similarity score.

[0088] In this embodiment, an encoder model can be constructed based on the transformer model framework, and the encoder model can perform feature extraction on the field value to obtain a first feature vector, wherein the feature parameter factors of the encoder model may include field value type, field value length, field value content, field value range, field value format, and field value uniqueness, etc.

[0089] In this embodiment, a training set may be obtained first, and the training set includes several field values, and the encoder model may be trained using the training set. Specifically, the field value is input into the encoder model, and the encoder model extracts the feature parameter factors in the field value and outputs a first feature vector. The encoder model is trained through a large amount of feature extraction training to obtain a trained encoder model; the trained encoder model can identify the feature parameter factors of each field value.

[0090] It should be noted that the model structure and specific training process of the encoder model can refer to the existing model structure and training process, which will not be repeated here.

[0091] In this embodiment, a trained encoder model may be used in advance to perform feature extraction on the field values ​​of each base field to obtain a second feature vector for subsequent use.

[0092] After obtaining the first feature vector of each field value of the data field and the second feature vector of each base field value, the similarity comparison model can be used to calculate the similarity score between the first feature vector of each field value and the second feature vector of each base field value. The following takes a field value in a data field as an example for a specific introduction:

[0093] The similarity comparison model is used to calculate the similarity between the first feature vector of the field value and the second feature vector of each base field value, and then the intermediate similarity score between the first feature vector and each second feature vector is obtained based on the similarity value.

[0094] After obtaining the intermediate similarity scores corresponding to all field values ​​of the data field and a certain base field, the average of all intermediate similarity scores is calculated, and the average is used as the second similarity score between the data field and the base field. In addition, all second similarity scores of the same data field can be pre-saved in the form of a set.

[0095] As an example, data field 1 includes n field values, namely, field value 11 To Field Value 1n The base field includes base field A and base field B. 11 To Field Value 1n The first eigenvector of is respectively calculated with the second eigenvector corresponding to the basis field A to obtain the intermediate similarity scores b （字段A-字段1）1 ……b （字段A-字段1）n ; In the same way, get the field values 11 To Field Value 1n The intermediate similarity score between the first eigenvector of and the second eigenvector corresponding to the basis field B, i.e., b （字段B-字段1）1 ……b （字段B-字段1）n ; Then b （字段A-字段1）1 To b （字段A-字段1）n Perform the average calculation to get the average value ‾ b （字段A-字段1） , similarly, calculate b （字段B-字段1）1 To b （字段B-字段1）n The average value of （字段B-字段1） ;‾b （字段A-字段1） This is the second similarity score between data field 1 and base field A, ‾b （字段B-字段1） This is the second similarity score between data field 1 and base field B, ‾b （字段A-字段1） With ‾b （字段B-字段1） A second similarity score set b that can constitute data field 1 （字段1） .

[0096] Figure 6 This is the sixth flow chart of the multi-source heterogeneous data fusion method of the embodiment of the present application. Figure 6 As shown, in some embodiments, determining the associated base field associated with the data field according to each first similarity score and each second similarity score in step 120 may include the following steps:

[0097] Step 610: For any base field, perform a weighted operation on the first similarity score and the second similarity score between the data field and the base field to obtain a total score of the data field and the base field.

[0098] Step 620: Determine the associated base field from all base fields according to the total score of the data field and each base field.

[0099] It should be noted that the weights in the weighted calculation process can be manually set in advance according to actual needs, and no specific restrictions are imposed on the weights here. For the convenience of description, the following takes a data field and a base field as an example to specifically introduce the calculation method of the total score.

[0100] Specifically, the first similarity score between the data field and the base field can be retrieved from the first similarity score set of the data field, and the second similarity score between the data field and the base field can be retrieved from the second similarity score set of the data field. Then, a weighted operation is performed on the first similarity score and the second similarity score according to a preset weight. Specifically, the operation can be performed using the following formula:

[0101] c （基底字段-数据字段) =W a ×a （基底字段-数据字段) +W b ×b (基底字段-数据字段）

[0102] Among them, c （基底字段-数据字段) is the total score of the data field and the base field, a （基底字段-数据字段) is the first similarity score between the data field and the base field, W a is the weight of the first similarity score, b （基底字段-数据字段) is the second similarity score between the data field and the base field, W b is the weight of the second similarity score.

[0103] The higher the total score of the data field and the base field, the higher the similarity between the data field and the base field, which further indicates that the business meaning represented by the data field and the base field is the same.

[0104] Therefore, after obtaining the total score of the data field and each base field, a preset base number of base fields can be screened out from all base fields according to the total score, and these base fields all have a high similarity with the data field. It should be noted that the preset base number can be manually set according to actual needs, and no specific restrictions are made here. In this embodiment, the preset base number of base fields and the data field can be recommended to the user through the user terminal, and then the user performs manual auxiliary judgment. The user selects the base field that is most similar to the data field from the recommended base fields as the associated base field, thereby further ensuring the accuracy of subsequent data fusion.

[0105] It is worth mentioning that when a preset base number of base fields and data fields are recommended to the user through the user terminal, the names, scores, scoring basis, data value samples, etc. of these fields can be output to the user for reference.

[0106] In step 130, after determining the associated base fields corresponding to each data field, an association relationship table between the associated base fields and the data corresponding to the data fields can be established in the database, and the association relationship table contains all the base fields of the business, and the multi-source heterogeneous data corresponding to the base fields in a unified format. In addition, a new data access layer can be created to encapsulate all database operations, including establishing an association relationship table, so as to realize the penetration logic of multi-table data and form a complete data link.

[0107] Thus, the data corresponding to the same association base field can be fused and formatted to obtain a data fusion result. The data fusion result here can be the above association relationship table, or can be partial fusion data extracted from the association relationship table.

[0108] Furthermore, a standard interface can be automatically generated based on the business and external applications to which the data fusion results are output. Through the standard interface, the data fusion results can be output to external applications or user terminals in a unified specification, thereby providing data services to the outside world.

[0109] In some embodiments, after performing data fusion on a multi-source heterogeneous data group to obtain a data fusion result, the multi-source heterogeneous data fusion method may further include: for any data field, determining the standard field information corresponding to the associated base field, and using the data field as the base field corresponding to the standard field information.

[0110] In this embodiment, after determining the associated base field corresponding to the data field, the data field can also be used as a new base field for use in the next data fusion. Specifically, the standard field information corresponding to the associated base field can be determined first, and then the data field can be included in the new training as the base field corresponding to the standard field information, so that the accuracy of data fusion can be continuously improved.

[0111] Therefore, the data required for the business is collected from the databases of various systems through a unified data collection scheme and data standard specifications, and the collected multi-source heterogeneous data sets are cleaned and preprocessed to unify the format of the data in the multi-source heterogeneous data sets; then, the semantic similarity and field value similarity comparison between the data fields and each base field in the multi-source heterogeneous data sets are performed to determine the most associated base field with the data field, thereby obtaining the association relationship between the data field and the associated base field; finally, according to the association relationship between each data field and the associated base field, all data corresponding to the same associated base field are fused. This achieves standardized and automated data fusion of multi-source heterogeneous data, and improves the processing efficiency and accuracy of multi-source heterogeneous data.

[0112] Based on the above embodiments, Figure 7 Schematic diagram of the structure of the multi-source heterogeneous data fusion device of the embodiment of the present application. Figure 7 As shown, the multi-source heterogeneous data fusion device 700 may include:

[0113] The data acquisition module 710 is used to acquire a multi-source heterogeneous data group of a business and determine all data fields in the multi-source heterogeneous data group;

[0114] The similarity comparison module 720 is used to determine, for any data field, a first similarity score and a second similarity score between the data field and each base field of the business, and determine an associated base field associated with the data field according to each first similarity score and each second similarity score; wherein the first similarity score is a semantic similarity score between the data field and each base field, and the second similarity score is a field value similarity score between the data field and each base field;

[0115] The data fusion module 730 is used to perform data fusion on the multi-source heterogeneous data set according to each data field and the corresponding associated base field to obtain a data fusion result.

[0116] Thus, the data acquisition module 710 collects the data required for the business from the databases of various systems through a unified data collection scheme and data standard specifications, and performs data cleaning and preprocessing on the collected multi-source heterogeneous data sets to unify the format of the data in the multi-source heterogeneous data sets; the similarity comparison module 720 then compares the semantic similarity and field value similarity of the data fields in the multi-source heterogeneous data sets with each base field to determine the associated base field most associated with the data field, thereby obtaining the association relationship between the data field and the associated base field; finally, the data fusion module 730 fuses all data corresponding to the same associated base field according to the association relationship between each data field and the associated base field. This realizes standardized and automated data fusion of multi-source heterogeneous data, and improves the processing efficiency and accuracy of multi-source heterogeneous data.

[0117] In some embodiments, the similarity comparison module 720 is specifically used to: input the data field into a pre-trained semantic similarity model to obtain a first similarity score between the data field and each base field, and the semantic similarity model is used to perform semantic similarity comparison between the data field and each base field to obtain a corresponding first similarity score.

[0118] In some embodiments, the similarity comparison module 720 is also specifically used to: determine all field values ​​of the data field; randomly extract a preset amount of field values ​​from all field values, and input the extracted field values ​​into a pre-trained encoder model to obtain a first feature vector corresponding to each field value; for any field value, based on the first feature vector of the data field value and the second feature vector of each basis field value, determine the intermediate similarity score between the data field value and the field value of each basis field, where the second feature vector is obtained by performing feature extraction on the basis field value using a pre-trained encoder model.

[0119] In some embodiments, the similarity comparison module 720 is further specifically used to: for any base field, perform a weighted operation on the first similarity score and the second similarity score between the data field and the base field to obtain a total score of the data field and the base field; and determine an associated base field from all base fields based on the total score of the data field and each base field.

[0120] In some embodiments, the multi-source heterogeneous data set of a business includes at least two groups. The multi-source heterogeneous data fusion device 700 may also include a base configuration module (not shown in the figure), which is used to: set a standard field set for the business, the standard field set includes at least one standard field information required by the business; determine the target multi-source heterogeneous data set from all multi-source heterogeneous data sets according to the standard field set; determine all target fields in the target multi-source heterogeneous data set; for any target field, if the target field is semantically related to the corresponding standard field information, use the target field as the base field corresponding to the standard field information.

[0121] In some implementations, the base configuration module is further used to: for any data field, determine the standard field information corresponding to the associated base field, and use the data field as the base field corresponding to the standard field information.

[0122] In some implementations, the data acquisition module 710 is specifically used to: set a data acquisition plan and data standard specifications for a business; and extract multi-source heterogeneous data sets from various systems according to the data acquisition plan and data standard specifications.

[0123] In some implementations, the data acquisition module 710 is further specifically used to: clean and preprocess the data in the multi-source heterogeneous data set according to data standard specifications; convert the preprocessed multi-source heterogeneous data set into a common file format and store it.

[0124] It should be noted that for details not disclosed in the multi-source heterogeneous data fusion device of this embodiment, please refer to the details disclosed in the embodiment of the multi-source heterogeneous data fusion method in the embodiments of this specification, and will not be repeated here.

[0125] Based on the above embodiments, Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 complete mutual communication through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the multi-source heterogeneous data fusion method, which includes: obtaining a multi-source heterogeneous data group of a business, and determining all data fields in the multi-source heterogeneous data group; for any data field, determining a first similarity score and a second similarity score between the data field and each base field of the business, and determining an associated base field associated with the data field according to each first similarity score and each second similarity score; wherein the first similarity score is a semantic similarity score between the data field and each base field, and the second similarity score is a field value similarity score between the data field and each base field; according to each data field and the corresponding associated base field, the multi-source heterogeneous data group is subjected to data fusion to obtain a data fusion result.

[0126] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0127] On the basis of the above embodiments, on the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-source heterogeneous data fusion method provided by the above methods, the method including: obtaining a multi-source heterogeneous data group of a business, and determining all data fields in the multi-source heterogeneous data group; for any data field, determining a first similarity score and a second similarity score between the data field and each base field of the business, and determining an associated base field associated with the data field according to each first similarity score and each second similarity score; wherein the first similarity score is a semantic similarity score between the data field and each base field, and the second similarity score is a field value similarity score between the data field and each base field; according to each data field and the corresponding associated base field, the multi-source heterogeneous data group is subjected to data fusion to obtain a data fusion result.

[0128] On the basis of the above embodiments, on another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the multi-source heterogeneous data fusion method provided by the above methods, the method comprising: obtaining a multi-source heterogeneous data group of a business, and determining all data fields in the multi-source heterogeneous data group; for any data field, determining a first similarity score and a second similarity score between the data field and each base field of the business, and determining an associated base field associated with the data field according to each first similarity score and each second similarity score; wherein the first similarity score is a semantic similarity score between the data field and each base field, and the second similarity score is a field value similarity score between the data field and each base field; according to each data field and the corresponding associated base field, the multi-source heterogeneous data group is subjected to data fusion to obtain a data fusion result.

[0129] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source heterogeneous data fusion method, characterized in that: include: Acquire a multi-source heterogeneous data set of a business, and determine all data fields in the multi-source heterogeneous data set; For any data field, determine a first similarity score and a second similarity score between the data field and each base field of the business, and determine an associated base field associated with the data field according to each of the first similarity scores and each of the second similarity scores; wherein the first similarity score is a semantic similarity score between the data field and each of the base fields, and the second similarity score is a field value similarity score between the data field and each of the base fields; According to each of the data fields and the corresponding associated base fields, the multi-source heterogeneous data sets are subjected to data fusion to obtain a data fusion result, wherein the data fusion result is a business data relationship table, and the business data relationship table includes each base field of the business and each corresponding data field; Wherein, the acquiring of a multi-source heterogeneous data set of a business includes: setting a data acquisition scheme and a data standard specification of the business; extracting the multi-source heterogeneous data set from each system according to the data acquisition scheme and the data standard specification; The multi-source heterogeneous data set of the business includes at least two groups. Before determining all data fields in the multi-source heterogeneous data set, the method further includes: A standard field set for the business is set, wherein the standard field set includes at least one standard field information required by the business; a target multi-source heterogeneous data set is determined from all the multi-source heterogeneous data sets according to the standard field set; all target fields in the target multi-source heterogeneous data set are determined; for any target field, if the target field is semantically related to the corresponding standard field information, the target field is used as a base field corresponding to the standard field information; For any data field, determining a second similarity score between the data field and each base field of the service includes: Determine all field values ​​of the data field; randomly extract a preset amount of field values ​​from all the field values, and input the extracted field values ​​into a pre-trained encoder model to obtain a first feature vector corresponding to each field value; for any field value, determine the intermediate similarity score between the data field value and each base field value according to the first feature vector of the data field value and the second feature vector of each base field value, wherein the second feature vector is obtained by extracting features from the base field value using the pre-trained encoder model; for any base field, average the intermediate similarity scores between each field value and the base field value to obtain the second similarity score between the data field and the base field; the feature parameter factors of the encoder model include field value type, field value length, field value content, field value range, field value format and field value uniqueness.

2. The multi-source heterogeneous data fusion method according to claim 1, characterized in that: The step of determining, for any data field, a first similarity score between the data field and each base field of the service comprises: The data field is input into a pre-trained semantic similarity model to obtain the first similarity score between the data field and each of the base fields. The semantic similarity model is used to compare the semantic similarity of the data field with each of the base fields to obtain the corresponding first similarity score.

3. The multi-source heterogeneous data fusion method according to claim 1, characterized in that: The determining, according to each of the first similarity scores and each of the second similarity scores, an associated base field associated with the data field comprises: For any base field, performing a weighted operation on the first similarity score and the second similarity score between the data field and the base field to obtain a total score of the data field and the base field; The associated base field is determined from all the base fields according to the total score of the data field and each of the base fields.

4. The multi-source heterogeneous data fusion method according to claim 1, characterized in that: After fusing the multi-source heterogeneous data sets to obtain a data fusion result, the method further includes: For any data field, the standard field information corresponding to the associated base field is determined, and the data field is used as the base field corresponding to the standard field information.

5. The multi-source heterogeneous data fusion method according to claim 1, characterized in that: The step of extracting the multi-source heterogeneous data set from each system according to the data collection scheme and the data standard specification includes: Performing data cleaning and preprocessing on the data in the multi-source heterogeneous data set according to the data standard specification; The preprocessed multi-source heterogeneous data sets are converted into a common file format and stored.

6. A multi-source heterogeneous data fusion device, characterized in that: include: A data acquisition module, used to acquire a multi-source heterogeneous data set of a business and determine all data fields in the multi-source heterogeneous data set; A similarity comparison module, for determining, for any data field, a first similarity score and a second similarity score between the data field and each base field of the business, and determining an associated base field associated with the data field according to each of the first similarity scores and each of the second similarity scores; wherein the first similarity score is a semantic similarity score between the data field and each of the base fields, and the second similarity score is a field value similarity score between the data field and each of the base fields; A data fusion module, used for fusing the multi-source heterogeneous data sets according to the data fields and the corresponding associated base fields to obtain a data fusion result, wherein the data fusion result is a business data relationship table, and the business data relationship table includes the base fields of the business and the corresponding data fields; The data acquisition module is specifically used to: set the data acquisition plan and data standard specification of the business; extract the multi-source heterogeneous data set from each system according to the data acquisition plan and the data standard specification; The multi-source heterogeneous data set of the service includes at least two groups, and the device also includes a base configuration module, and the base configuration module is used to: A standard field set for the business is set, wherein the standard field set includes at least one standard field information required by the business; a target multi-source heterogeneous data set is determined from all the multi-source heterogeneous data sets according to the standard field set; all target fields in the target multi-source heterogeneous data set are determined; for any target field, if the target field is semantically related to the corresponding standard field information, the target field is used as a base field corresponding to the standard field information; The similarity comparison module is specifically used for: Determine all field values ​​of the data field; randomly extract a preset amount of field values ​​from all the field values, and input the extracted field values ​​into a pre-trained encoder model to obtain a first feature vector corresponding to each field value; for any field value, determine the intermediate similarity score between the data field value and each base field value according to the first feature vector of the data field value and the second feature vector of each base field value, wherein the second feature vector is obtained by extracting features from the base field value using the pre-trained encoder model; for any base field, average the intermediate similarity scores between each field value and the base field value to obtain the second similarity score between the data field and the base field; the feature parameter factors of the encoder model include field value type, field value length, field value content, field value range, field value format and field value uniqueness.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-source heterogeneous data fusion method according to any one of claims 1 to 5 is implemented.

Images