Heterogeneous data processing system
By designing a heterogeneous data processing system, using keyword extraction, preprocessing, feature extraction and model screening methods, the problems of multi-source heterogeneous data processing in traditional technologies are solved, and the accuracy and efficiency of data processing are improved.
Patent Information
- Application Number
- CN202411498218.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-03-21
AI Technical Summary
Traditional technologies are difficult to effectively process multi-source heterogeneous data, especially due to the processing difficulty caused by data format and semantic differences, missing values and inconsistencies.
A heterogeneous data processing system is designed to obtain user requests, perform keyword extraction and initial data acquisition, perform preprocessing and feature extraction, and filter the model to obtain target result data.
It improves the accuracy and efficiency of heterogeneous data processing, and can more accurately obtain the result data requested by the target user, adapting to the diversity and variation of the data.
Smart Images
Figure CN119669320A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a heterogeneous data processing system. Background Art
[0002] With the rapid development of the information age, a large amount of data of different types and formats has been generated. These multi-source heterogeneous data have become an indispensable resource in various fields. The characteristics of these data are diversity and complexity of format, and the data quality is also uneven. Traditional data processing methods face many challenges when processing such multi-source heterogeneous data, mainly including the following two aspects. On the one hand, the source of data may include structured data (such as database records), semi-structured data (such as XML documents) and unstructured data (such as text, images, etc.). The formats and semantics of these data vary greatly and cannot be directly processed and analyzed uniformly. On the other hand, multi-source heterogeneous data often have problems such as missing values and inconsistencies, which increases the difficulty of data processing.
[0003] In order to solve these problems, many technologies and methods have been developed in the market, and the following two are common. The first is a rule-based method that processes these multi-source heterogeneous data by defining clear conversion rules and mapping relationships. This method relies on a lot of manual operations and professional knowledge, and it is difficult to cope with data changes, resulting in low data processing efficiency, and it is difficult to ensure accuracy and consistency; the second is to use data templates to match and convert different suitable data. This method has high template design and maintenance costs and is difficult to adapt to the variability of data.
[0004] Different types of data have different feature representations. How to effectively extract features from multi-source heterogeneous data and apply them remains a big challenge. Summary of the invention
[0005] In view of the above technical problems, the technical solution adopted by the present invention is: a heterogeneous data processing system, the system includes a database, a processor and a memory storing a computer program, wherein the memory stores data of several different types of structures and formats, and when the processor executes the computer program, the following steps are implemented: S100, obtaining a target user request, and processing the target user request to obtain a keyword list, wherein the keyword list includes a plurality of keywords included in the target user request; S200, based on the keyword list, acquiring an initial object data list and metadata corresponding to the initial object data from a database, wherein the initial object data list includes a plurality of initial object data; S300, preprocessing the initial object data based on metadata corresponding to the initial object data, obtaining intermediate object data, and thereby obtaining an intermediate object data list; S400, extracting features from the intermediate object data to obtain target object data, thereby obtaining a target object data list; S500, based on the target user request, obtaining an implementation path of the target user request; S600, screening the preset initial model based on the implementation path requested by the target user, and obtaining an intermediate model list, wherein the intermediate model list includes a plurality of intermediate models; S700, based on metadata corresponding to target object data, screening the intermediate model list to obtain a target model list, the target model list including a plurality of target models, the target model being an intermediate model in the intermediate model list; S800, acquiring target result data based on the target object data list and the target model list.
[0006] The present invention has at least the following beneficial effects: In summary, a target user request is obtained, and the target user request is processed to obtain a keyword list. Based on the keyword list, an initial object data list and metadata corresponding to the initial object data are obtained from a database. Based on the metadata corresponding to the initial object data, the initial object data is preprocessed to obtain intermediate object data, thereby obtaining an intermediate object data list. Feature extraction is performed on the intermediate object data to obtain target object data, thereby obtaining a target object data list. Based on the target user request, an implementation path of the target user request is obtained. Based on the implementation path of the target user request, a preset initial model is screened to obtain an intermediate model list. Based on the metadata corresponding to the target object data, the intermediate model list is screened to obtain a target model list. Based on the target object data list and the target model list, target result data is obtained. The present invention preprocesses data of various different types of structures and formats by preprocessing, feature extraction, etc., and screens through the metadata of the target object data to obtain the target model for processing, thereby improving the accuracy of heterogeneous data processing and more accurately obtaining the target result data requested by the target user. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0008] Figure 1A flowchart of a heterogeneous data processing system executing a computer program is provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0009] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0010] An embodiment of the present invention provides a heterogeneous data processing system, the system comprising a database, a processor and a memory storing a computer program, wherein the database stores data of several different types of structures and formats, such as Figure 1 As shown, when the processor executes the computer program, the following steps are implemented: S100, obtaining a target user request, and processing the target user request to obtain a keyword list, wherein the keyword list includes a plurality of keywords included in the target user request.
[0011] Specifically, those skilled in the art know that any method of processing a sentence to obtain keywords in the prior art falls within the protection scope of the present invention and will not be described in detail herein.
[0012] S200, based on the keyword list, obtaining an initial object data list and metadata corresponding to the initial object data from a database, wherein the initial object data list includes a plurality of initial object data.
[0013] Specifically, the metadata corresponding to the initial object data is data describing the initial object data. Specifically, the database includes traditional numerical features, mined text, images, audio and other non-numerical features.
[0014] S300, preprocessing the initial object data based on metadata corresponding to the initial object data, acquiring intermediate object data, and thereby acquiring an intermediate object data list.
[0015] Specifically, the preprocessing includes unit conversion, field mapping, and data value mapping. The unit conversion is to establish a mapping relationship between fields in different data sources to ensure that the data can correspond correctly during the conversion process. For some specific data values, it may be necessary to formulate specific mapping rules, such as mapping "yes / no" in different data sources to a unified "1 / 0".
[0016] S400, extracting features from the intermediate object data to obtain target object data, thereby obtaining a target object data list.
[0017] S500, based on the target user request, obtain an implementation path of the target user request. The implementation path is a path for executing the target user request. In one embodiment of the present invention, the target user request is matched with a preset request to obtain the preset request with the highest matching degree with the target user request, and the implementation path corresponding to the preset request is used as the implementation path of the target user request. In an exemplary description of the present invention, the implementation path of a target user request is to use a classification model first and then a neural network model.
[0018] S600, based on the implementation path requested by the target user, the preset initial model is screened to obtain an intermediate model list, wherein the intermediate model list includes a plurality of intermediate models. The preset initial model includes classification problem models such as logistic regression, decision tree classifier, random forest classifier, support vector machine (SVM), naive Bayes classifier or neural network, regression problem models such as linear regression, ridge regression, lasso regression, decision tree regression or neural network model, clustering problem models such as k-means clustering, hierarchical clustering or DBSCAN, etc.
[0019] In an exemplary description of the present invention, the preset initial model is screened, and SVM and random forest are obtained as intermediate models corresponding to the classification model, and RNN is obtained as the intermediate model corresponding to the neural network model.
[0020] S700, based on metadata corresponding to target object data, filter the intermediate model list to obtain a target model list, wherein the target model list includes a plurality of target models, and the target model is an intermediate model in the intermediate model list.
[0021] In an exemplary description of the present invention, based on the metadata corresponding to the target object data, SVM is used as the target model corresponding to the classification model and RNN is used as the target model corresponding to the neural network model.
[0022] S800, acquiring target result data based on the target object data list and the target model list.
[0023] In summary, the target user request is obtained, and the target user request is processed to obtain a keyword list. Based on the keyword list, an initial object data list and metadata corresponding to the initial object data are obtained from a database. Based on the metadata corresponding to the initial object data, the initial object data is preprocessed to obtain intermediate object data, thereby obtaining an intermediate object data list. Feature extraction is performed on the intermediate object data to obtain target object data, thereby obtaining a target object data list. Based on the target user request, an implementation path of the target user request is obtained. Based on the implementation path of the target user request, a preset initial model is screened to obtain an intermediate model list. Based on the metadata corresponding to the target object data, the intermediate model list is screened to obtain a target model list. Based on the target object data list and the target model list, target result data is obtained. The present invention preprocesses data of various different types of structures and formats by preprocessing, feature extraction, etc., and screens through the metadata of the target object data to obtain the target model for processing, thereby improving the accuracy of heterogeneous data processing and more accurately obtaining the target result data requested by the target user.
[0024] Specifically, in S400, extracting features from the intermediate object data to obtain the target object data includes: S410: When the intermediate object data is text data, a natural language processing model is used to perform feature extraction to obtain a target object data list.
[0025] Specifically, in one embodiment of the present invention, the natural language processing model is a BERT model. The BERT model is used to segment text data into words or phrases, and these words or phrases are converted into vector representations. These vectors can capture the semantic information in the text and provide effective feature input for subsequent machine learning models.
[0026] S420: When the intermediate object data is image data, a deep learning model is used to perform feature extraction to obtain a target object data list.
[0027] Specifically, in one embodiment of the present invention, the deep learning model is a CNN model. The CNN model is used to extract features such as edges, textures, and colors from the image. These features reflect the essential attributes of the image, which helps to improve the performance of the machine learning model.
[0028] S430, when the intermediate object data is numerical data, obtain preset statistical features and obtain feature values based on the preset statistical features as a target object data list. Specifically, the preset statistical features include mean, variance, etc., which can describe the distribution characteristics and change rules of the data.
[0029] Specifically, in S300, preprocessing the initial object data based on metadata corresponding to the initial object data includes: S310: Acquire a data structure of the initial object data based on metadata corresponding to the initial object data.
[0030] S320, preprocessing the initial object data based on the data structure of the initial object data.
[0031] In one embodiment of the present invention, for structured data, we use AQL query or ETL tools to clean the data and remove duplicate, erroneous or irrelevant data to ensure the accuracy and consistency of the data.
[0032] In one embodiment of the present invention, for unstructured data, we use natural language processing and image processing technology to perform deep cleaning and preprocessing on the data to extract useful information.
[0033] Specifically, in S500, based on the target user request, obtaining the implementation path of the target user request includes: S510, obtaining a historical user request combination list, wherein the historical user request combination list includes a plurality of historical user request combinations, and the historical user request combinations include: historical user requests and execution paths corresponding to the historical user requests.
[0034] S520, training a preset neural network model using the historical user request combination list to obtain a target neural network model.
[0035] S530, input the target user request into the target neural network model to obtain the implementation path of the target user request.
[0036] In summary, obtain a combination list of historical user requests, use the combination list of historical user requests to train the preset neural network model, obtain the target neural network model, input the target user requests into the target neural network model, obtain the implementation path of the target user requests, and obtain the implementation path more accurately based on the neural network model.
[0037] Specifically, S700 also includes: scoring the intermediate model based on the metadata corresponding to the target object data and the metadata of the intermediate model, obtaining the score of the intermediate model corresponding to the target object data, and filtering the intermediate model list based on the score of the intermediate model to obtain the target model list. Those skilled in the art know that any method of scoring the intermediate model based on matching metadata in the prior art falls within the protection scope of the present invention. For example, in one embodiment of the present invention, matching is performed based on the data type included in the metadata corresponding to the target object data and the processable data type included in the metadata of the intermediate model, thereby obtaining the matching degree as the score of the intermediate model.
[0038] Furthermore, S800 also includes: S810, input the target object data into the corresponding target model according to the implementation path, and obtain intermediate result data.
[0039] S820, obtaining preset indicator data of the intermediate result data. Specifically, the preset indicator data can be set according to actual needs.
[0040] S830: If all preset indicator data meet preset user requirements, obtain target result data based on the intermediate result data.
[0041] S840: If a preset indicator data does not meet the preset user requirements, the target object data corresponding to the intermediate result data is marked as problem data, and S850 is executed.
[0042] S850: If there is an intermediate model corresponding to the problem data whose score is greater than the score of the target model corresponding to the problem data, use the intermediate model corresponding to the problem data to process the problem data to obtain new result data. The intermediate model corresponding to the problem data is the intermediate model corresponding to the target object data corresponding to the problem data.
[0043] S860, obtaining target result data based on the new result data.
[0044] In summary, the target object data is input into the corresponding target model according to the implementation path, and the intermediate result data is obtained, and the preset indicator data of the intermediate result data is obtained. If all the preset indicator data meet the preset user requirements, the target result data is obtained based on the intermediate result data. If a preset indicator data does not meet the preset user requirements, the target object data corresponding to the intermediate result data is marked as problem data. If there is an intermediate model corresponding to the problem data whose score is greater than the score of the target model corresponding to the problem data, the intermediate model corresponding to the problem data is used to process the problem data to obtain new result data, thereby ensuring that the target object data is processed using a more accurate intermediate model.
[0045] Specifically, S830 also includes: S831, fuse the target object data, input the fused target object data into the fused target model, and obtain key result data.
[0046] Specifically, the unique advantages of various target models are utilized and closely combined through efficient fusion methods such as weighted averaging, voting or stacking. This fusion strategy is used to improve the accuracy and stability of the prediction results.
[0047] S832, compare the intermediate result data with the key result data to obtain the target result data.
[0048] In summary, the target object data is fused, the fused target object data is input into the fused target model, the key result data is obtained, the intermediate result data is compared with the key result data, and the target result data is obtained.
[0049] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.
Claims
1. A heterogeneous data processing system, characterized in that: The system includes a database, a processor, and a memory storing a computer program, wherein the memory stores data of several different types of structures and formats. When the processor executes the computer program, the following steps are implemented: S100, obtaining a target user request, and processing the target user request to obtain a keyword list, wherein the keyword list includes a plurality of keywords included in the target user request; S200, based on the keyword list, acquiring an initial object data list and metadata corresponding to the initial object data from a database, wherein the initial object data list includes a plurality of initial object data; S300, preprocessing the initial object data based on metadata corresponding to the initial object data, obtaining intermediate object data, and thereby obtaining an intermediate object data list; S400, extracting features from the intermediate object data to obtain target object data, thereby obtaining a target object data list; S500, based on the target user request, obtaining an implementation path of the target user request; S600, screening the preset initial model based on the implementation path requested by the target user, and obtaining an intermediate model list, wherein the intermediate model list includes a plurality of intermediate models; S700, based on metadata corresponding to target object data, screening the intermediate model list to obtain a target model list, the target model list including a plurality of target models, the target model being an intermediate model in the intermediate model list; S800, acquiring target result data based on the target object data list and the target model list.
2. The heterogeneous data processing system according to claim 1, characterized in that: In S400, feature extraction is performed on the intermediate object data to obtain target object data, including: S410, when the intermediate object data is text data, use a natural language processing model to perform feature extraction to obtain a target object data list; S420, when the intermediate object data is image data, use a deep learning model to perform feature extraction to obtain a target object data list; S430: When the intermediate object data is numerical data, obtain preset statistical features and obtain feature values based on the preset statistical features as a target object data list.
3. The heterogeneous data processing system according to claim 1, characterized in that: In S300, preprocessing the initial object data based on metadata corresponding to the initial object data includes: S310, acquiring a data structure of the initial object data based on metadata corresponding to the initial object data; S320, preprocessing the initial object data based on the data structure of the initial object data.
4. The heterogeneous data processing system according to claim 3, characterized in that: The preprocessing includes unit conversion, field mapping, and data value mapping.
5. The heterogeneous data processing system according to claim 1, characterized in that: In S500, based on the target user request, the implementation path of obtaining the target user request includes: S510, obtaining a historical user request combination list, wherein the historical user request combination list includes a plurality of historical user request combinations, wherein the historical user request combinations include: historical user requests and execution paths corresponding to the historical user requests; S520, training a preset neural network model using the historical user request combination list to obtain a target neural network model; S530, input the target user request into the target neural network model to obtain the implementation path of the target user request.
6. The heterogeneous data processing system according to claim 1, characterized in that: S700 also includes: scoring the intermediate model based on the metadata included in the target object data and the metadata of the intermediate model, obtaining the score of the intermediate model corresponding to the target object data, filtering the intermediate model list based on the score of the intermediate model, and obtaining the target model list.
7. The heterogeneous data processing system according to claim 6, characterized in that: Also includes: The S800 also includes: S810, inputting the target object data into the corresponding target model according to the implementation path to obtain intermediate result data; S820, obtaining preset indicator data of the intermediate result data; S830, if all preset indicator data meet the preset user requirements, obtain target result data based on the intermediate result data; S840, if a preset indicator data does not meet the preset user requirements, mark the target object data corresponding to the intermediate result data as problem data, and execute S850; S850, if there is an intermediate model corresponding to the problem data whose score is greater than the score of the target model corresponding to the problem data, use the intermediate model corresponding to the problem data to process the problem data to obtain new result data; S860, obtaining target result data based on the new result data.
8. The heterogeneous data processing system according to claim 7, characterized in that: The S830 also includes: S831, fusing the target object data, and inputting the fused target object data into the fused target model to obtain key result data; S832, compare the intermediate result data with the key result data to obtain the target result data.
9. The heterogeneous data processing system according to claim 2, characterized in that: The natural language processing model is a BERT model.
10. The heterogeneous data processing system according to claim 2, characterized in that: The deep learning model is a CNN model.