A multi-source data fusion method, device and medium
By screening and determining the target model, multi-source heterogeneous data are processed, which solves the problem that traditional technology is difficult to process multi-source heterogeneous data, and improves the accuracy and efficiency of data processing.
Patent Information
- Application Number
- CN202411498122.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-03-14
- Estimated Expiration
- 2044-10-25
AI Technical Summary
The prior art is difficult to effectively process multi-source heterogeneous data, especially when the data sources are diverse, the formats are complex and the quality is uneven, and traditional methods are difficult to adapt to the variability of data.
By obtaining the target user request, filtering the preset initial model, determining the target model based on the metadata of the target object data, and processing heterogeneous data, realizing intelligent identification, classification, association and integration of data.
It improves the accuracy of heterogeneous data processing, reduces manual intervention and errors, and can more accurately obtain the target result data requested by the target user.
Smart Images
Figure CN119026087B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a multi-source data fusion method, device and medium. Background Art
[0002] With the rapid development of the information age, a large amount of data of different types and formats has been generated. These multi-source heterogeneous data have become an indispensable resource in various fields. The characteristics of these data are diversity and complexity of format, and the data quality is also uneven. Traditional data processing methods face many challenges when processing such multi-source heterogeneous data, mainly including the following two aspects. On the one hand, the source of data may include structured data (such as database records), semi-structured data (such as XML documents) and unstructured data (such as text, images, etc.). The formats and semantics of these data vary greatly and cannot be directly processed and analyzed uniformly. On the other hand, multi-source heterogeneous data often have problems such as missing values and inconsistencies, which increases the difficulty of data processing.
[0003] In order to solve these problems, many technologies and methods have been developed in the market. For example, rule-based methods process these multi-source heterogeneous data by defining clear conversion rules and mapping relationships, and use data templates to match and convert data in different formats. However, these methods are difficult to adapt to the variability of data.
[0004] With the rapid development of artificial intelligence technology, machine learning-based methods have shown great potential in multi-source heterogeneous data processing. Machine learning technology automatically learns the characteristics of data through training models, thereby realizing the intelligent identification, classification, association and integration of multi-source heterogeneous data. This method can not only automatically adapt to data changes and reduce manual operations, but also improve the efficiency and accuracy of data processing.
[0005] However, there are many challenges in the current multi-source heterogeneous data processing methods based on machine learning. How to select appropriate algorithms and optimize them according to the characteristics of the data and task requirements still requires in-depth research. Summary of the invention
[0006] In view of the above technical problems, the technical solution adopted by the present invention is: a multi-source data fusion method, the method comprising the following steps:
[0007] Obtaining a target user request, and obtaining a target object data list from an initial data source according to the target user request, wherein the target object data list includes a plurality of target object data, and the preset initial data source includes data of a plurality of different types of structures and formats;
[0008] Based on the target user's request, obtain the implementation path of the target user's request;
[0009] Screening the preset initial model based on the implementation path requested by the target user to obtain an intermediate model list, wherein the intermediate model list includes a plurality of intermediate models;
[0010] Based on the metadata corresponding to the target object data, the intermediate model list is screened to obtain a target model list, wherein the target model list includes a plurality of target models, and the target model is an intermediate model in the intermediate model list;
[0011] Input the target object data into the corresponding target model to obtain the intermediate result data;
[0012] If the preset indicator data of all intermediate result data meet the preset user requirements, the target result data is obtained by fusion based on the intermediate result data;
[0013] If there is a preset indicator data of the intermediate result data that does not meet the preset user requirements, the target object data corresponding to the intermediate result data is marked as problem data;
[0014] If there is an intermediate model corresponding to the problem data that meets the preset scoring requirements, the intermediate model is used to process the problem data to obtain new result data, and the new result data is fused to obtain the target result data.
[0015] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the aforementioned method.
[0016] According to yet another aspect of the present invention, an electronic device is provided, comprising a processor and the aforementioned non-transitory computer-readable storage medium.
[0017] The present invention has at least the following beneficial effects: In summary, a target user request is obtained, and according to the target user request, a target object data list is obtained from an initial data source, based on the target user request, an implementation path of the target user request is obtained, based on the implementation path of the target user request, a preset initial model is screened, an intermediate model list is obtained, based on the metadata corresponding to the target object data, the intermediate model list is screened, the target model list is obtained, the target object data is input into the corresponding target model, and intermediate result data is obtained; if the preset indicator data of all intermediate result data meet the preset user requirements, fusion is performed based on the intermediate result data to obtain the target result data; if there is an intermediate result data whose preset indicator data does not meet the preset user requirements, the target object data corresponding to the intermediate result data is marked as problem data; if there is an intermediate model corresponding to the problem data that meets the preset scoring requirements, the intermediate model is used to process the problem data to obtain new result data, and fusion is performed based on the new result data to obtain the target result data; the present invention determines the target model based on the metadata of the target object data, and uses the target model to process heterogeneous data, thereby improving the accuracy of processing heterogeneous data, reducing manual intervention and errors, and thus more accurately obtaining the target result data requested by the target user. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0019] Figure 1 The present invention provides a flowchart of a multi-source data fusion method. DETAILED DESCRIPTION
[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0021] The embodiment of the present invention provides a multi-source data fusion method, such as Figure 1 As shown, the method comprises the following steps:
[0022] S100, obtaining a target user request, and obtaining a target object data list from an initial data source according to the target user request, wherein the target object data list includes a plurality of target object data, and the preset initial data source includes data of a plurality of different types of structures and formats.
[0023] S200, based on the target user request, obtain an implementation path of the target user request. The implementation path is a path for executing the target user request. In one embodiment of the present invention, the target user request is matched with a preset request to obtain the preset request with the highest matching degree with the target user request, and the implementation path corresponding to the preset request is used as the implementation path of the target user request. In an exemplary description of the present invention, the implementation path of a target user request is to first use a classification model and then use a neural network model.
[0024] S300, based on the implementation path requested by the target user, the preset initial model is screened to obtain an intermediate model list, wherein the intermediate model list includes several intermediate models. The preset initial model includes classification problem models such as logistic regression, decision tree classifier, random forest classifier, support vector machine (SVM), naive Bayes classifier or neural network, regression problem models such as linear regression, ridge regression, lasso regression, decision tree regression or neural network model, clustering problem models such as k-means clustering, hierarchical clustering or DBSCAN, etc.
[0025] In an exemplary description of the present invention, the preset initial model is screened, and SVM and random forest are obtained as intermediate models corresponding to the classification model, and RNN is obtained as the intermediate model corresponding to the neural network model.
[0026] S400, based on metadata corresponding to target object data, filter the intermediate model list to obtain a target model list, wherein the target model list includes a plurality of target models, and the target model is an intermediate model in the intermediate model list.
[0027] In an exemplary description of the present invention, based on the metadata corresponding to the target object data, SVM is used as the target model corresponding to the classification model, and RNN is used as the target model corresponding to the neural network model.
[0028] Wherein, based on the metadata corresponding to the target object data, the intermediate model list is screened, and obtaining the target model list includes: scoring the intermediate model based on the metadata corresponding to the initial object data and the metadata of the intermediate model, obtaining the score of the intermediate model, and screening the intermediate model list based on the score of the intermediate model to obtain the target model list. Those skilled in the art know that any method of scoring the intermediate model based on matching metadata in the prior art belongs to the protection scope of the present invention. For example, in one embodiment of the present invention, matching is performed based on the data type included in the metadata corresponding to the target object data and the processable data type included in the metadata of the intermediate model, so as to obtain the matching degree as the score of the intermediate model.
[0029] S500, inputting the target object data into the corresponding target model to obtain intermediate result data.
[0030] S600: If the preset indicator data of all the intermediate result data meet the preset user requirements, the target result data is obtained by fusion based on the intermediate result data. Specifically, the preset indicator data can be set according to actual requirements.
[0031] S700: If there is a preset indicator data of the intermediate result data that does not meet the preset user requirements, mark the target object data corresponding to the intermediate result data as problem data.
[0032] S800: If there is an intermediate model corresponding to the problem data that meets the preset scoring requirements, the intermediate model is used to process the problem data to obtain new result data, and the new result data is fused to obtain target result data.
[0033] Specifically, the preset scoring requirement is: the score of the intermediate model corresponding to the problem data is greater than the score of the target model.
[0034] In summary, a target user request is obtained, and according to the target user request, a target object data list is obtained from an initial data source, based on the target user request, an implementation path of the target user request is obtained, based on the implementation path of the target user request, a preset initial model is screened, and an intermediate model list is obtained, based on the metadata corresponding to the target object data, the intermediate model list is screened, and a target model list is obtained, the target object data is input into a corresponding target model, and intermediate result data is obtained, if the preset indicator data of all intermediate result data meet the preset user requirements, fusion is performed based on the intermediate result data to obtain the target result data, if there is an intermediate result data whose preset indicator data does not meet the preset user requirements, the target object data corresponding to the intermediate result data is marked as problem data, if there is an intermediate model corresponding to the problem data that meets the preset scoring requirements, the intermediate model is used to process the problem data to obtain new result data, and fusion is performed based on the new result data to obtain the target result data, the present invention determines the target model based on the metadata of the target object data, and uses the target model to process heterogeneous data, thereby improving the accuracy of processing heterogeneous data, reducing manual intervention and errors, and thus more accurately obtaining the target result data requested by the target user.
[0035] Specifically, in S100, according to the target user's request, obtaining a target object data list from an initial data source includes:
[0036] The target user request is processed to obtain a keyword list, where the keyword list includes several keywords included in the target user request.
[0037] Specifically, those skilled in the art know that any method of processing a sentence to obtain keywords in the prior art falls within the protection scope of the present invention and will not be described in detail herein.
[0038] Based on the keyword list, an initial object data list and metadata corresponding to the initial object data are acquired from a preset initial data source, wherein the initial object data list includes a plurality of initial object data.
[0039] Specifically, the metadata corresponding to the initial object data is data describing the initial object data. Specifically, the database includes traditional numerical features, mined text, images, audio and other non-numerical features.
[0040] Based on the metadata corresponding to the initial object data, the initial object data is preprocessed to obtain the intermediate object data, thereby obtaining a list of the intermediate object data.
[0041] Specifically, the preprocessing includes unit conversion, field mapping, and data value mapping. The unit conversion is as follows: if the data in different data sources use different measurement units, unit conversion is required to unify the measurement standard; the field mapping is to establish a mapping relationship between fields in different data sources to ensure that the data can correspond correctly during the conversion process; for certain specific data values, specific mapping rules may need to be formulated, such as mapping "yes / no" in different data sources to a unified "1 / 0".
[0042] Feature extraction is performed on the intermediate object data to obtain the target object data, thereby obtaining a target object data list.
[0043] In summary, the target user request is processed to obtain a keyword list. Based on the keyword list, the initial object data list and the metadata corresponding to the initial object data are obtained from the preset initial data source. Based on the metadata corresponding to the initial object data, the initial object data is preprocessed to obtain the intermediate object data. Feature extraction is performed on the intermediate object data to obtain the target object data, thereby obtaining the target object data list to eliminate dimensional differences between different data sources and ensure data accuracy.
[0044] Specifically, extracting features from the intermediate object data to obtain the target object data includes:
[0045] When the intermediate object data is text data, a natural language processing model is used to extract features and obtain a target object data list. Specifically, in one embodiment of the present invention, the natural language processing model is a BERT model. The BERT model is used to segment text data into words or phrases, and these words or phrases are converted into vector representations. These vectors can capture the semantic information in the text and provide effective feature input for subsequent machine learning models.
[0046] When the intermediate object data is image data, a deep learning model is used to extract features and obtain a target object data list. The deep learning model is a CNN model. Specifically, in one embodiment of the present invention, the deep learning model is a CNN model. The CNN model is used to extract features such as edges, textures, and colors from the image. These features reflect the essential attributes of the image, which helps to improve the performance of the machine learning model.
[0047] When the intermediate object data is numerical data, a preset statistical feature is obtained and a feature value is obtained based on the preset statistical feature as a target object data list. Specifically, the preset statistical feature includes mean, variance, etc., which can characterize the distribution characteristics and change rules of the data.
[0048] Specifically, based on the metadata corresponding to the initial object data, preprocessing the initial object data includes:
[0049] Based on the metadata corresponding to the initial object data, a data structure of the initial object data is obtained.
[0050] Based on the data structure of the initial object data, the initial object data is preprocessed. In one embodiment of the present invention, for structured data, we use AQL query or ETL tools to clean the data and remove duplicate, erroneous or irrelevant data to ensure the accuracy and consistency of the data.
[0051] In one embodiment of the present invention, for unstructured data, we use natural language processing and image processing technology to perform deep cleaning and preprocessing on the data to extract useful information.
[0052] Specifically, the target result data is obtained by fusing the intermediate result data, including:
[0053] The target object data is fused and input into the fused target model to obtain key result data. Specifically, the unique advantages of various target models are utilized and they are closely combined through efficient fusion methods such as weighted averaging, voting or stacking. This fusion strategy is used to improve the accuracy and stability of the prediction results.
[0054] Use the intermediate result data to compare with the key result data to obtain the target result data.
[0055] In summary, the target object data is fused, the fused target object data is input into the fused target model, the key result data is obtained, the intermediate result data is compared with the key result data, and the target result data is obtained.
[0056] Furthermore, based on the target user request, the implementation path of obtaining the target user request includes:
[0057] A historical user request combination list is obtained, wherein the historical user request combination list includes a plurality of historical user request combinations, and the historical user request combinations include: historical user requests and execution paths corresponding to the historical user requests.
[0058] The preset neural network model is trained using the historical user request combination list to obtain the target neural network model.
[0059] The target user request is input into the target neural network model to obtain the implementation path of the target user request.
[0060] In summary, obtain a combination list of historical user requests, use the combination list of historical user requests to train the preset neural network model, obtain the target neural network model, input the target user requests into the target neural network model, obtain the implementation path of the target user requests, and obtain the implementation path more accurately based on the neural network model.
[0061] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.
[0062] An embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned non-transitory computer-readable storage medium.
[0063] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.
Claims
1. A multi-source data fusion method, characterized in that: The method comprises the following steps: Obtaining a target user request, and obtaining a target object data list from an initial data source according to the target user request, wherein the target object data list includes a plurality of target object data, and the initial data source includes data of a plurality of different types of structures and formats; Based on the target user's request, obtain the implementation path of the target user's request; Screening the preset initial model based on the implementation path requested by the target user to obtain an intermediate model list, wherein the intermediate model list includes a plurality of intermediate models; Based on the metadata corresponding to the target object data, the intermediate model list is screened to obtain a target model list, wherein the target model list includes a plurality of target models, and the target model is an intermediate model in the intermediate model list; Input the target object data into the corresponding target model to obtain the intermediate result data; If the preset indicator data of all intermediate result data meet the preset user requirements, the target result data is obtained by fusion based on the intermediate result data; If there is a preset indicator data of the intermediate result data that does not meet the preset user requirements, the target object data corresponding to the intermediate result data is marked as problem data; If there is an intermediate model corresponding to the problem data that meets the preset scoring requirements, the intermediate model is used to process the problem data to obtain new result data, and the new result data is fused to obtain the target result data; Wherein, according to the target user's request, obtaining the target object data list from the initial data source includes: Processing the target user request to obtain a keyword list, wherein the keyword list includes a plurality of keywords included in the target user request; Based on the keyword list, an initial object data list and metadata corresponding to the initial object data are acquired from an initial data source, wherein the initial object data list includes a plurality of initial object data; Based on the metadata corresponding to the initial object data, preprocess the initial object data to obtain the intermediate object data, thereby obtaining a list of the intermediate object data; Feature extraction is performed on the intermediate object data to obtain the target object data, thereby obtaining a target object data list.
2. The multi-source data fusion method according to claim 1, characterized in that: Extract features from the intermediate object data to obtain the target object data, including: When the intermediate object data is text data, a natural language processing model is used to extract features and obtain a list of target object data; When the intermediate object data is image data, a deep learning model is used to extract features and obtain a list of target object data; When the intermediate object data is numerical data, a preset statistical feature is obtained and a feature value is obtained based on the preset statistical feature as the target object data list.
3. The multi-source data fusion method according to claim 2, characterized in that: The deep learning model is a CNN model.
4. The multi-source data fusion method according to claim 1, characterized in that: Based on the metadata corresponding to the initial object data, the initial object data is preprocessed including: Based on the metadata corresponding to the initial object data, obtaining a data structure of the initial object data; The initial object data is preprocessed based on the data structure of the initial object data.
5. The multi-source data fusion method according to claim 4, characterized in that: Based on the intermediate result data, the target result data is obtained by fusion, including: Fusing the target object data, and inputting the fused target object data into the fused target model to obtain key result data; Use the intermediate result data to compare with the key result data to obtain the target result data.
6. The multi-source data fusion method according to claim 1, characterized in that: Based on the metadata corresponding to the target object data, the intermediate model list is filtered to obtain the target model list including: Based on the metadata corresponding to the initial object data and the metadata of the intermediate model, the intermediate model is scored to obtain the score of the intermediate model, and the intermediate model list is filtered based on the score of the intermediate model to obtain the target model list.
7. The multi-source data fusion method according to claim 6, characterized in that: The preset scoring requirement is that the score of the intermediate model corresponding to the problem data is greater than the score of the target model.
8. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the multi-source data fusion method as described in any one of claims 1 to 7.
9. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 8.
Citation Information
Patent Citations
Data normalization method for multiple data sources, electronic equipment and storage medium
CN118069660A
Product data fusion method, apparatus and device, and storage medium
WO2022257457A1