A multi-modal, multi-source, heterogeneous data fusion method

Abstract

A multimodal multi-source heterogeneous data fusion method belongs to the field of data processing technology, and includes the following steps: S1, configuring a data pipeline from an original database to a target database, and extracting the source data from the original database to the target database in full through the data pipeline to obtain a source data set; S2, structuring and cleaning the source data set to obtain a first data set; S3, dumping the first data set to obtain a second data set; S4, performing data fusion processing on the second data set, including data mapping and conversion, data standardization, to obtain a third data set; S5, for different types of data, performing feature extraction and processing on the third data set to obtain a final data set; S6, persistently storing the final data set. The multimodal multi-source heterogeneous data fusion method provided by the present invention can effectively solve the data quality problem in multimodal multi-source heterogeneous data fusion and improve the consistency and reliability of the fusion results.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a multi-modal multi-source heterogeneous data fusion method. Background Art

[0002] With the rapid development of technologies such as the Internet of Things and social media, the way data is generated has become more diverse, including multiple modalities such as text, images, videos, and audio. These data often come from different devices or platforms and have different formats and structures. There are usually rich connections and information between these data, but due to different modalities and different representation methods, unified processing and fusion are required.

[0003] Knowledge fusion of multi-source heterogeneous multimodal data is a complex and important research topic, which involves extracting and integrating information from data of different sources, different formats and different modalities to build a more comprehensive and accurate knowledge graph or intelligent system. Data in the real world often comes from different sources and modalities, including text, images, audio and other forms. Most existing fusion methods can only process limited modal combinations and are highly dependent on manual processing. There is a lack of effective fusion means for the increasingly complex multimodal data fusion. In addition, due to the diverse data sources, uneven data quality, and difficulty in ensuring consistency, the reliability of the fusion results is affected. Therefore, a more effective data fusion method is needed to process multi-source heterogeneous data, so as to facilitate more accurate decision support or predictive analysis based on the fused data. The present invention solves this technical problem. Summary of the invention

[0004] The present invention provides a multimodal multi-source heterogeneous data fusion method, which can effectively solve the data quality problem in multimodal multi-source heterogeneous data fusion, improve the consistency and reliability of fusion results, can be applied to a variety of scenarios, and improves computing efficiency, thereby improving versatility and practicality.

[0005] A multi-modal multi-source heterogeneous data fusion method comprises the following steps:

[0006] S1. Configure a data pipeline from the original database to the target database, extract all source data from the original database to the target database through the data pipeline, and obtain a source data set;

[0007] S2, performing a cleaning operation on the source data set in the target database to obtain a first data set;

[0008] S3, dumping the first data set, and determining whether the file link contained in the data set is valid, and obtaining the second data set after dumping;

[0009] S4, performing data fusion processing on the second data set, including data mapping and conversion, and data standardization, so as to obtain a third data set;

[0010] S5. For the third data set, extract and process features of different types of data to obtain a final data set;

[0011] S6. Persistently store the final data set.

[0012] Furthermore, the step S2 specifically includes the following steps:

[0013] S21. Perform data preprocessing, perform summary statistics on the data, and understand the basic situation of the source data;

[0014] S22. Perform data deduplication. According to the characteristics of the data set, unique identifiers are assigned to data from different sources, and duplicate data in the data set is deduplicated based on the unique identifier. The specific formula is:

[0015] ,

[0016] in, represents the data set after deduplication, and D represents the original data set;

[0017] S23. Predict missing values ​​of the deduplicated data set and detect outliers.

[0018] Furthermore, the step S23 specifically includes the following steps:

[0019] S231. Use mean filling method to predict missing values:

[0020] ,

[0021] in, represents the average value of the data in the jth column;

[0022] S232. Use statistical methods to detect outliers:

[0023] ,

[0024] in, Represents all the data in the jth column.

[0025] Furthermore, the step S3 specifically includes the following steps:

[0026] S31, for data containing file links, judging whether the file links contained in the data set are valid according to the interface, if valid, dumping them to the local server, if invalid, marking them as invalid;

[0027] S32, after the file link is dumped, a local OSS storage path link is generated to replace the original link;

[0028] S33. Generate a corresponding file hash value as its primary key according to the resource file dumped to the local server.

[0029] Furthermore, the data mapping and conversion in step S4 specifically includes the following steps:

[0030] S41. Define a unified data model: Define a set of standard data models, including field names and data types, based on actual needs and data source details;

[0031] S42. Vectorize the fields: Use the pre-trained Word2Vec model to obtain the field name vectors of the source dataset and the standardized data model;

[0032] S43, Field mapping: Match and map the fields in different data sources with the fields in the standard model;

[0033] S44, dynamically selecting the best mapping relationship according to the result of similarity calculation;

[0034] S45, Field conversion: For fields that do not fully match, perform necessary conversion operations to make the fields meet the requirements of the standard model.

[0035] Furthermore, the step S43 specifically includes the following contents:

[0036] Use the text similarity calculation method to compare the similarity between fields in different data sources and fields in the standard model. The specific formula is as follows:

[0037] ,

[0038] in, is the field in the ith data source, is a field in the standard model, Indicates the similarity between the two.

[0039] Furthermore, the specific formula in step S44 is as follows:

[0040] ,

[0041] ,

[0042] in, Represents the field in the i-th data source Maps to fields in the standard model.

[0043] Furthermore, the specific formula in step S45 is as follows:

[0044] ,

[0045] in, Represents the converted field. Indicates the mapping result.

[0046] Furthermore, the data standardization in step S4 specifically includes the following steps:

[0047] S46, data type conversion: convert the data into a unified format to ensure that the data types of all fields are consistent after fusion;

[0048] S47, Numerical standardization: Standardize numerical data. The values ​​in the same field of data from different sources are mapped to the value range in the standard model to ensure that the data are in the same scale. The specific formula is as follows:

[0049] ,

[0050] Where D is the source data field, represents the i-th value in the field, is the normalized mapping value.

[0051] The technical effects of the present invention are as follows:

[0052] (1) This solution preprocesses the data to understand the basic situation of the data and cleans the repeated data to ensure data quality. It then predicts and detects missing values ​​and outliers in the data to find problems in the data, thereby further ensuring data quality and improving the consistency and reliability of subsequent data fusion results.

[0053] (2) This solution uses a pre-trained Word2Vec model to obtain the field name vectors of the source data set and the standardized data model, and matches and maps the fields in different data sources with the fields in the standard model to achieve data fusion, thereby improving the computational efficiency of the fusion algorithm in the above manner. In addition, this solution can also compare the similarity between the fields in the data source and the fields in the standard model, and select the best mapping relationship based on the similarity. In addition, for completely unmatched fields, conversion operations can also be performed to make them meet the requirements of the standard model, thereby improving the versatility and practicality of this method;

[0054] (3) This solution converts the data into a unified format by performing data type conversion, thereby ensuring that the data types of all fields after fusion are consistent, improving consistency, and through numerical standardization processing, it can ensure that the data are on the same scale, thereby improving the reliability of data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0056] See also Figure 1 , a multi-modal multi-source heterogeneous data fusion method, comprising the following steps:

[0057] S1. Configure a data pipeline from the original database to the target database, extract all source data from the original database to the target database through the data pipeline, and obtain a source data set;

[0058] S2, performing a cleaning operation on the source data set in the target database to obtain a first data set;

[0059] S3, dumping the first data set, and determining whether the file link contained in the data set is valid, and obtaining the second data set after dumping;

[0060] S4, performing data fusion processing on the second data set, including data mapping and conversion, and data standardization, so as to obtain a third data set;

[0061] S5. For the third data set, extract and process features of different types of data to obtain a final data set;

[0062] S6. Persistently store the final data set.

[0063] Preferably, in this embodiment, an ETL tool is used to configure a data pipeline from the original database to the target database; specifically, for step S6, this solution can select a suitable data storage solution, such as a relational database or a distributed file storage system, according to the data volume and access frequency.

[0064] Furthermore, step S2 specifically includes the following steps:

[0065] S21. Perform data preprocessing and perform summary statistics on the data to understand the basic situation of the source data, such as the distribution of missing values ​​and abnormal data formats;

[0066] S22. Perform data deduplication. According to the characteristics of the data set, unique identifiers are assigned to data from different sources, and duplicate data in the data set is deduplicated based on the unique identifier. The specific formula is:

[0067] ,

[0068] in, represents the data set after deduplication, and D represents the original data set;

[0069] S23. Predict missing values ​​of the deduplicated data set and detect outliers.

[0070] Furthermore, step S23 specifically includes the following steps:

[0071] S231. Use mean / median filling method to predict missing values:

[0072] ,

[0073] in, Represents the mean or median of the jth column of data;

[0074] For this part, you can also predict missing values ​​through interpolation methods or use machine learning models;

[0075] S232. Use statistical methods (such as standard deviation, IQR) to detect outliers:

[0076] ,

[0077] in, Represents all the data in the jth column.

[0078] For outliers, after communicating with the data provider, actions such as replacement, deletion or retention can be taken.

[0079] Furthermore, step S3 specifically includes the following steps:

[0080] S31, for data containing file links, judging whether the file links contained in the data set are valid according to the interface, if valid, dumping them to the local server, if invalid, marking them as invalid;

[0081] S32, after the file link is dumped, a local OSS storage path link is generated to replace the original link;

[0082] S33. Generate a corresponding file hash value as its primary key according to the resource file dumped to the local server.

[0083] Preferably, the interface in this embodiment is implemented by defining a function for determining link validity:

[0084] def is_url_valid(url):

[0085] try:

[0086] response = requests.head(url)

[0087] if response.status_code == 200:

[0088] return True

[0089] except requests.exceptions.RequestException as e:

[0090] print(f"Error: {e}")

[0091] return False

[0092] This interface defines a function is_url_valid(url) and accepts a URL parameter. The link whose validity needs to be judged is passed into the interface as a parameter. The interface uses the requests.head() method to send an HTTP HEAD request to the URL. The request obtains the header information of the response. If the status code of the response is 200, the request is successful, and the link is judged to be a valid link, which can be accessed and dumped.

[0093] If any exceptions are encountered while sending the request (such as network errors, invalid URLs, etc.), these exceptions are caught and ignored through the pass statement;

[0094] If all the above conditions are not met, the function returns False by default, indicating that the URL is invalid.

[0095] Furthermore, the data mapping and conversion in step S4 specifically includes the following steps:

[0096] S41. Define a unified data model: Define a set of standard data models, including field names and data types, based on actual needs and data source details;

[0097] S42. Vectorize the fields: Use the pre-trained Word2Vec model to obtain the field name vectors of the source dataset and the standardized data model;

[0098] S43, Field mapping: Match and map the fields in different data sources with the fields in the standard model;

[0099] S44, dynamically selecting the best mapping relationship according to the result of similarity calculation;

[0100] S45, Field conversion: For fields that do not fully match, perform necessary conversion operations to make the fields meet the requirements of the standard model.

[0101] Furthermore, step S43 specifically includes the following contents:

[0102] Use the text similarity calculation method to compare the similarity between fields in different data sources and fields in the standard model. The specific formula is as follows:

[0103] ,

[0104] in, is the field in the ith data source, is a field in the standard model, Indicates the similarity between the two.

[0105] Furthermore, the specific formula in step S44 is as follows:

[0106] ,

[0107] ,

[0108] in, Represents the field in the i-th data source Maps to fields in the standard model.

[0109] Furthermore, the specific formula in step S45 is as follows:

[0110] ,

[0111] in, Represents the converted field. Indicates the mapping result.

[0112] Furthermore, the data standardization in step S4 specifically includes the following steps:

[0113] S46, data type conversion: convert the data into a unified format to ensure that the data types of all fields are consistent after fusion;

[0114] S47, Numerical standardization: Standardize numerical data. The values ​​in the same field of data from different sources are mapped to the value range in the standard model to ensure that the data are in the same scale. The specific formula is as follows:

[0115] ,

[0116] Where D is the source data field, represents the i-th value in the field, is the normalized mapping value.

[0117] Regarding the content mentioned in step S5, this embodiment is mainly implemented in the following ways:

[0118] (1) For image data, the image files are processed through machine learning models to obtain image-related data information, mainly including image quality, resolution, OCR text, person recognition, and generation of image corresponding vector values;

[0119] (2) For video data, the video files are processed through machine learning models to obtain video-related data information, mainly including video quality, video duration, resolution, OCR text, video audio track extraction, video subtitle text generation, character recognition, video frame extraction, and generation of corresponding vector values ​​of video frames;

[0120] (3) For audio data, the audio files are processed through machine learning models to obtain audio-related data information, mainly including: audio subtitle text generation, audio duration, sensitive word extraction, generation of audio corresponding vector values, etc.;

[0121] (4) For documents and PDF data, retain their OSS storage path link;

[0122] (5) Wherein, for the video resource file, a frame is automatically extracted whenever the video screen switches, and the video frame file can be regarded as image data and processed through the image processing interface, so as to obtain image-related data information of the video frame as relevant information of the video resource file;

[0123] For a video resource file, its audio track is extracted as an audio file, which can be regarded as audio data and processed through an audio processing interface to obtain relevant data information of the audio file as relevant information of the video resource file.

[0124] The working principle of the present invention is as follows:

[0125] First, configure the data pipeline, extract the data to the target database through the data pipeline, and obtain the source data set. Then clean the source data set, including preprocessing and deduplication, and detect missing values ​​and outliers in the data to obtain the first data set. Then dump the first data set to obtain the second data set, and then perform data fusion processing on the second data set. Specifically, the pre-trained Word2Vec model is used to obtain the field name vectors of the source data set and the standardized data model, and the fields in different data sources are matched and mapped with the fields in the standard model to achieve data fusion, and the similarity between the fields in the data source and the fields in the standard model is compared. The best mapping relationship is selected according to the similarity. For completely unmatched fields, conversion operations can also be performed to make them meet the requirements of the standard model to achieve data standardization and obtain the third data set. Then, feature extraction and processing are performed on the third data set after fusion. Different methods can be used to process different types of data to obtain the final data set. Finally, a suitable data storage solution is selected according to the data volume and access frequency to store the final data set persistently.

[0126] The above embodiments are only preferred embodiments of the present invention. Those skilled in the art can derive other embodiments from the above embodiments without creative work. Therefore, this application protects not only the above embodiments, but also the scope consistent with the principles and features of this application.

Claims

1. A multi-modal multi-source heterogeneous data fusion method, characterized in that: The following steps are involved: S1. Configure a data pipeline from the original database to the target database, extract all source data from the original database to the target database through the data pipeline, and obtain a source data set; S2, performing a cleaning operation on the source data set in the target database to obtain a first data set; S3, dumping the first data set, and determining whether the file link contained in the data set is valid, and obtaining the second data set after dumping; S4, performing data fusion processing on the second data set, including data mapping and conversion, and data standardization, so as to obtain a third data set; S5. For the third data set, extract and process features of different types of data to obtain a final data set; S6. Persistently store the final data set; The step S3 specifically comprises the following steps: S31, for data containing file links, judging whether the file links contained in the data set are valid according to the interface, if valid, dumping them to the local server, if invalid, marking them as invalid; S32, after the file link is dumped, a local OSS storage path link is generated to replace the original link; S33, generating a corresponding file hash value as its primary key according to the resource file dumped to the local server; The step S4 specifically comprises the following steps: S41. Define a unified data model: Define a set of standard data models, including field names and data types, based on actual needs and data source details; S42. Vectorize the fields: Use the pre-trained Word2Vec model to obtain the field name vectors of the source dataset and the standardized data model; S43, Field mapping: Match and map the fields in different data sources with the fields in the standard model; S44, dynamically selecting the best mapping relationship according to the result of similarity calculation; S45, Field conversion: For fields that are not fully matched, necessary conversion operations are performed to make the fields meet the requirements of the standard model; S46, Data type conversion: Convert the data into a unified format to ensure that the data types of all fields are consistent after fusion; S47, Numerical standardization: Standardize numerical data. The values ​​in the same field of data from different sources are mapped to the value range in the standard model to ensure that the data are in the same scale. The specific formula is as follows: ， Where D is the source data field, represents the i-th value in the field, is the normalized mapping value.

2. The multi-modal multi-source heterogeneous data fusion method according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21. Perform data preprocessing, perform summary statistics on the data, and understand the basic situation of the source data; S22. Perform data deduplication. According to the characteristics of the data set, unique identifiers are assigned to data from different sources, and duplicate data in the data set is deduplicated based on the unique identifier. The specific formula is: ， in, represents the data set after deduplication, and D represents the original data set; S23. Predict missing values ​​of the deduplicated data set and detect outliers.

3. The multi-modal multi-source heterogeneous data fusion method according to claim 2, characterized in that: The step S23 specifically includes the following steps: S231. Use mean filling method to predict missing values: ， in, represents the average value of the data in the jth column; S232. Use statistical methods to detect outliers: ， in, Represents all the data in the jth column.

4. The multi-modal multi-source heterogeneous data fusion method according to claim 1, characterized in that: The step S43 specifically includes the following contents: Use the text similarity calculation method to compare the similarity between fields in different data sources and fields in the standard model. The specific formula is as follows: ， in, is the field in the ith data source, is a field in the standard model, Indicates the similarity between the two.

5. The multi-modal multi-source heterogeneous data fusion method according to claim 1, characterized in that: The specific formula in step S44 is as follows: ， ， in, Represents the field in the i-th data source Maps to fields in the standard model.

6. The multi-modal multi-source heterogeneous data fusion method according to claim 1, characterized in that: The specific formula in step S45 is as follows: ， in, Represents the converted field. Indicates the mapping result.

Images