Methods, apparatus, and equipment for heterogeneous data mapping based on machine learning algorithms

Abstract

The present application discloses a heterogeneous data mapping method, device and equipment based on machine learning algorithm, which relates to the field of computer technology and can solve the technical problems of low efficiency and low accuracy of heterogeneous data mapping. It includes: receiving a heterogeneous data mapping request, which carries the original data table structure, random sampling data of the original data table structure, and the target data table structure to be mapped; performing feature processing on the random sampling data according to the original data table structure to obtain the original data features, and the feature processing includes at least one of data cleaning, data screening, data residual filling and feature engineering; inputting the original data table structure, the original data features and the target data table structure into the pre-trained heterogeneous data mapping model to obtain the predicted mapping result; determining the target data content after the random sampling data is mapped from the original data table structure to the target data table structure according to the predicted mapping result. The present application is applicable to the automated mapping of heterogeneous data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a heterogeneous data mapping method, device and equipment based on a machine learning algorithm. Background Art

[0002] With the development of network technology and database system technology, there is a technical demand for heterogeneous data exchange and integration between enterprises and between departments within an enterprise. Currently, consistent mapping of heterogeneous data is achieved through data mapping technology.

[0003] Currently, when performing heterogeneous data mapping, a large number of professionals are required to participate in logical steps such as data extraction, logical mapping, back-extraction comparison and verification between heterogeneous data. This requires high business skills of professionals and results in a large workload, low efficiency and low accuracy in heterogeneous data mapping. Summary of the invention

[0004] In view of this, the present application provides a heterogeneous data mapping method, device and equipment based on machine learning algorithm, which can be used to solve the technical problems that the current mapping method of heterogeneous data requires high business skills of professionals and leads to large workload, low efficiency and low accuracy of heterogeneous data mapping.

[0005] According to one aspect of the present application, a heterogeneous data mapping method based on a machine learning algorithm is provided, the method comprising:

[0006] receiving a heterogeneous data mapping request, wherein the heterogeneous data mapping request carries an original data table structure, randomly sampled data of the original data table structure, and a target data table structure to be mapped;

[0007] Performing feature processing on the randomly sampled data according to the original data table structure to obtain original data features, wherein the feature processing includes at least one of data cleaning, data screening, data residual filling and feature engineering;

[0008] Inputting the original data table structure, the original data features and the target data table structure into a pre-trained heterogeneous data mapping model to obtain a predicted mapping result;

[0009] The target data content of the randomly sampled data mapped from the original data table structure to the target data table structure is determined according to the predicted mapping result.

[0010] According to another aspect of the present application, a heterogeneous data mapping device based on a machine learning algorithm is provided, the device comprising:

[0011] A receiving module, configured to receive a heterogeneous data mapping request, wherein the heterogeneous data mapping request carries an original data table structure, randomly sampled data of the original data table structure, and a target data table structure to be mapped;

[0012] A processing module, used for performing feature processing on the randomly sampled data according to the original data table structure to obtain original data features, wherein the feature processing includes at least one of data cleaning, data screening, data residual filling and feature engineering;

[0013] An input module, used for inputting the original data table structure, the original data features and the target data table structure into the pre-trained heterogeneous data mapping model to obtain a predicted mapping result;

[0014] A determination module is used to determine the target data content of the randomly sampled data after the random sampling data is mapped from the original data table structure to the target data table structure according to the predicted mapping result.

[0015] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned heterogeneous data mapping method based on machine learning algorithm is implemented.

[0016] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned heterogeneous data mapping method based on machine learning algorithm when executing the program.

[0017] By means of the above technical scheme, the present application provides a heterogeneous data mapping method, device and equipment based on a machine learning algorithm. Compared with the current heterogeneous data mapping method, the present application can perform feature preprocessing on the random sampling data according to the original data table structure after receiving a heterogeneous data mapping request carrying the original data table structure, the random sampling data of the original data table structure and the target data table structure to be mapped, and then input the original data table structure, the preprocessed random sampling data and the target data table structure into the pre-trained heterogeneous data mapping model to obtain the predicted mapping result; finally, the target data content after the random sampling data is mapped from the original data table structure to the target data table structure is determined according to the predicted mapping result. Through the technical scheme in the present application, the machine learning algorithm can be used to identify and reduce the misunderstandings or human errors that occur during manual mapping, improve the accuracy of heterogeneous data mapping, and use automated data mapping, which can greatly improve work efficiency, reduce the input of technical personnel and shorten the overall task cycle. In addition, the entire mapping process does not require the intervention of a large number of professionals, and the business skills requirements for professionals are relatively low.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the local application. In the drawings:

[0020] Figure 1 A schematic diagram of a process of a heterogeneous data mapping method based on a machine learning algorithm provided in an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of a process of another heterogeneous data mapping method based on a machine learning algorithm provided in an embodiment of the present application is shown;

[0022] Figure 3 A schematic diagram of the structure of a heterogeneous data mapping device based on a machine learning algorithm provided in an embodiment of the present application is shown;

[0023] Figure 4 A schematic diagram of the structure of another heterogeneous data mapping device based on a machine learning algorithm provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The embodiments of the present application can realize automatic mapping of heterogeneous data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0025] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0026] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0027] In view of the current heterogeneous data mapping method, which requires high professional skills of professionals and leads to technical problems such as large workload, low efficiency and low accuracy of heterogeneous data mapping, this application provides a heterogeneous data mapping method based on machine learning algorithm, such as Figure 1 As shown, the method includes:

[0028] 101. Receive a heterogeneous data mapping request, where the heterogeneous data mapping request carries an original data table structure, randomly sampled data of the original data table structure, and a target data table structure to be mapped.

[0029] In a specific application scenario, when an integrated table comes from multiple original data tables, when performing data mapping, an intermediate table is often designed manually. However, when designing the intermediate table, the technician needs to have business knowledge of multiple industries at the same time, so that multiple external data can be accurately mapped to one table through the intermediate table. This method requires manual judgment of table creation statements, data structure, primary and foreign key relationships, field types, sampled data distribution range, business data authenticity, self-consistency between business fields, etc., resulting in high requirements on the business skills of professionals, and the mapping efficiency and accuracy of heterogeneous data are low. In this regard, in this application, the data relationship between tables is intelligently determined through machine learning to achieve intelligent mapping and determination of heterogeneous data.

[0030] For this embodiment, in order to realize the automatic mapping of heterogeneous data, it can respond to the heterogeneous data mapping request, realize the automatic mapping processing of the heterogeneous data according to the original data table structure carried in the heterogeneous data mapping request, the randomly sampled data of the original data table structure and the target data table structure to be mapped, and obtain the target data content.

[0031] The execution subject of the present application may be a device or apparatus for automatically mapping heterogeneous data, which may be configured on the client side or the server side, and may obtain the original data table structure, the randomly sampled data of the original data table structure, and the target data table structure to be mapped according to the uploaded heterogeneous data mapping request; and then perform feature preprocessing on the randomly sampled data according to the original data table structure; input the original data table structure, the preprocessed randomly sampled data, and the target data table structure into a pre-trained heterogeneous data mapping model to obtain a predicted mapping result; and determine the target data content after the randomly sampled data is mapped from the original data table structure to the target data table structure according to the predicted mapping result.

[0032] 102. Perform feature processing on the randomly sampled data according to the original data table structure to obtain the original data features, wherein the feature processing includes at least one of data cleaning, data screening, data residual filling and feature engineering.

[0033] Among them, by performing data cleaning on randomly sampled data, high saturation and accurate data can be screened out. Data cleaning includes default value filling and outlier deletion. Default value filling refers to filling in missing data items. The method of filling data includes filling in the average value, specifically: determining the first feature of the missing data item, calculating the average value of the data under the first feature, and using the average value as the filling value. Outliers refer to data that is inconsistent with the actual situation in historical inquiry transaction data, or data collected under system failures. Outlier data is deleted. In addition, preprocessing can also include vector matrix conversion processing of structured or unstructured long text data to achieve standard structured processing of data, etc.

[0034] 103. Input the original data table structure, original data features, and target data table structure into the pre-trained heterogeneous data mapping model to obtain a predicted mapping result.

[0035] Among them, the heterogeneous data mapping model is constructed in advance according to the historical heterogeneous data mapping results. It should be noted that before executing the steps of this embodiment, the heterogeneous data mapping model needs to be trained in advance. Among them, the heterogeneous data mapping model can select a linear regression model, a logistic regression model, a naive Bayes model, a decision tree model, a support vector machine (SVM) model, a neural network model, a KNN model, a conditional random field (CRF) model, etc. In the steps of this embodiment, the heterogeneous data mapping model is an SVM model as an example for explanation. Support vector machine (SVM) is a binary classification model, and its basic model is a linear classifier with the largest interval defined in the feature space. The largest interval distinguishes it from the perceptron and can be used for classification, regression and outlier detection (clustering) tasks. In this application, it can be applied to the SVM model to realize the classification prediction of the heterogeneous data mapping results, and output the prediction probability and prediction confidence for each preset table and preset field respectively, so as to realize the judgment of the final data mapping result based on the prediction probability and prediction confidence.

[0036] 104. Determine the target data content after the random sampling data is mapped from the original data table structure to the target data table structure according to the prediction mapping result.

[0037] For this embodiment, after determining that the prediction mapping result is obtained, when the corresponding confidence levels in the prediction mapping result are greatly differentiated, the target data content can be automatically determined directly based on the prediction data with higher confidence levels; when the confidence levels corresponding to the various prediction data are relatively similar, manual participation can be provided to verify or correct the prediction mapping result, and specifically, a final data mapping structure file can be formed, and then the final data mapping structure file can be imported into a commonly used ETL tool to extract the target data content.

[0038] Through the heterogeneous data mapping method based on the machine learning algorithm in this embodiment, after receiving the heterogeneous data mapping request carrying the original data table structure, the random sampling data of the original data table structure and the target data table structure to be mapped, the random sampling data can be feature preprocessed according to the original data table structure, and then the original data table structure, the preprocessed random sampling data and the target data table structure are input into the pre-trained heterogeneous data mapping model to obtain the predicted mapping result; finally, the target data content after the random sampling data is mapped from the original data table structure to the target data table structure is determined according to the predicted mapping result. Through the technical solution in this application, the machine learning algorithm can be used to identify and reduce the misunderstandings or human errors that occur during manual mapping, improve the accuracy of heterogeneous data mapping, and use automated data mapping, which can greatly improve work efficiency, reduce the input of technical personnel and shorten the overall task cycle. In addition, the entire mapping process does not require the intervention of a large number of professionals, and the business skills requirements for professionals are relatively low.

[0039] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another heterogeneous data mapping method based on a machine learning algorithm is provided, such as Figure 2 As shown, the method includes:

[0040] 201. Receive a heterogeneous data mapping request, where the heterogeneous data mapping request carries an original data table structure, randomly sampled data of the original data table structure, and a target data table structure to be mapped.

[0041] This embodiment can be applied to various heterogeneous data mapping scenarios, and accordingly, the randomly sampled data can correspond to financial data, statistical data, insurance data, etc. In the embodiment of the present application, the heterogeneous data mapping of insurance data is taken as an example for explanation. Specifically, the insurance industry data can be first collected and integrated. The insurance industry data can specifically include policy registration data, auto insurance data, insurance intermediary data, agricultural insurance data, tax-preferred health insurance data, tax-deferred pension insurance data, etc., and then the insurance industry data under each data structure is uploaded, and a heterogeneous data mapping request is generated. Then, in response to the heterogeneous data mapping request, the intelligent mapping of heterogeneous data can be achieved by executing steps 202 to 205 of the embodiment.

[0042] 202. Perform feature processing on the randomly sampled data according to the original data table structure to obtain the original data features, where the feature processing includes at least one of data cleaning, data screening, data residual filling and feature engineering.

[0043] In specific application scenarios, feature processing includes: table creation statements, data structure, primary and foreign key relationships, field types, sampled data distribution range, business data authenticity, self-consistency between business fields, etc. For example, random sampled data can correspond to non-structured data such as long text and pictures. After obtaining the random sampled data, the random sampled data can be cleaned first to ensure the quality of the data input into the model and reduce the errors caused by quality problems; secondly, data with low saturation and low accuracy can be eliminated. Furthermore, the random sampled data can be converted according to the original data table structure.

[0044] Correspondingly, for this embodiment, when performing feature conversion processing on the randomly sampled data according to the original data table structure, step 202 of the embodiment may specifically include: if the randomly sampled data is determined to be structured or unstructured long text data according to the original data table structure, then obtaining a data conversion format matching the original data table structure and a corresponding feature vector space dimension; converting the randomly sampled data into a vector feature matrix matching the feature vector space dimension according to the data conversion format.

[0045] It should be noted that before executing steps 203 and 204 of the embodiment, the heterogeneous data mapping model needs to be pre-trained in advance. After the training is completed, the trained heterogeneous data mapping model is further used to predict the mapping result. In a specific application scenario, the heterogeneous data mapping model may include a first heterogeneous data mapping model for table matching and a second heterogeneous data mapping model for field matching. Accordingly, during pre-training, the embodiment steps may specifically include: constructing a training set based on historical heterogeneous data mapping results, and using the training set to train the first heterogeneous data mapping model and the second heterogeneous data mapping model respectively, so that the first heterogeneous data mapping model and the second heterogeneous data mapping model meet the preset training standards, and determine that the pre-training is completed. Among them, the historical heterogeneous data mapping result is the heterogeneous data mapping result that completed the mapping process at a historical moment, and may specifically include the original data table structure, the randomly sampled data of the original data table structure, and the mapped data structure and data content.

[0046] In a specific application scenario, as an optional method, when building a training set based on historical heterogeneous data mapping results to train the first heterogeneous data mapping model and the second heterogeneous data mapping model, the historical heterogeneous data mapping results can be randomly split into 5 parts, 80% training set and 20% test set, and cross-validation can be implemented. GridSearch is used to tune the model parameters, and confusion matrix, learning curve and other methods are used to reduce model overfitting and optimize model parameters. The model verification results of the test set are further judged by AUC and KS values. When it is determined that the AUC and KS values ​​corresponding to the heterogeneous data mapping model are greater than the preset threshold, it can be determined that the heterogeneous data mapping model meets the preset training standards and the pre-training is completed.

[0047] 203. Input the original data table structure, original data features, and target data table structure into a pre-trained first heterogeneous data mapping model to obtain a first predicted mapping result about the target table, wherein the first heterogeneous data mapping model is used for table matching.

[0048] For this embodiment, the original data table structure can be saved as a CSV file in a format as input item 1, the original data features after feature processing can be used as input item 2, and the target data table structure can be saved as a CSV file in the same format as input item 3. Then, according to the model structure output item requirements, input items 1, 2, and 3 are respectively input into the first heterogeneous data mapping model to obtain the first prediction mapping result about the target table, and the prediction confidence corresponding to each preset table is included in the first prediction mapping result. In a specific application scenario, the preset table whose corresponding prediction confidence is greater than the preset confidence threshold can be determined as the target table.

[0049] 204. Input the original data table structure, original data features, and target data table structure into the pre-trained second heterogeneous data mapping model to obtain a second predicted mapping result of the target field in the target table, wherein the second heterogeneous data mapping model is used for field matching.

[0050] Corresponding to step 203 of the embodiment, for this embodiment, the original data table structure can also be saved in a format as a CSV file as input item 1, the original data features after feature processing can be used as input item 2, and the target data table structure can also be saved in a format as a CSV file as input item 3. Then, according to the requirements of the model structure output items, the input items 1, 2, and 3 are respectively input into the second heterogeneous data mapping model to obtain a second prediction mapping result about the field, which includes the prediction confidence corresponding to each preset field under the target table.

[0051] 205. Determine the target data content after the random sampling data is mapped from the original data table structure to the target data table structure according to the predicted mapping result.

[0052] In a specific application scenario, for this embodiment, as an optional method, when the corresponding confidences in the prediction mapping results vary greatly, the target data content generated by the automated mapping process can be directly determined in order of confidence from high to low. Specifically, according to the prediction confidence corresponding to each preset field under the target table included in the second prediction mapping result corresponding to step 204 of the embodiment, the preset field with the highest corresponding prediction confidence can be screened out as the target field, and then the target field under the target table is determined as the target data content after the randomly sampled data is mapped from the original data table structure to the target data table structure.

[0053] As another preferred method, for results that cannot be classified or have a relatively average confidence, manual assistance is required to determine the final mapping result based on the model output results. Accordingly, step 205 of the embodiment may specifically include: screening out a preset number of prediction mapping tables that match the target data table structure in the first prediction mapping result in order of confidence from high to low; screening out a preset number of prediction mapping fields in the second prediction mapping result in order of confidence from high to low; outputting the prediction mapping fields under the prediction mapping table, so that the target data content under the target data table structure can be manually determined based on the prediction mapping fields under the prediction mapping table.

[0054] In a specific application scenario, in order to realize the reuse of the generated target data content, as a preferred method, a heterogeneous data mapping table can also be generated according to the historical heterogeneous data mapping results, so that when the heterogeneous data is mapped, the corresponding target data content can be directly queried in the heterogeneous data mapping table. Accordingly, the embodiment steps can specifically include: generating a heterogeneous data mapping table according to the random sampling data under the original data table structure and the target data content under the target data table structure, so as to screen the target data content matching the original data table structure, the random sampling data and the target data table structure based on the heterogeneous data mapping table.

[0055] Accordingly, as another optional method, in order to improve the model accuracy of the heterogeneous data mapping model, the heterogeneous data mapping model can also be iterated by adding the input data method, and the model result is optimized by adding the data of the model training set. Each time a new training set is added, the data preprocessing and target algorithm parameter tuning process are repeated with reference to the algorithm modeling process. Accordingly, the embodiment steps may specifically include: updating the training set according to the randomly sampled data under the original data table structure and the target data content under the target data table structure, so as to iteratively train the first heterogeneous data mapping model and the second heterogeneous data mapping model according to the updated training set.

[0056] By means of the above-mentioned heterogeneous data mapping method based on machine learning algorithm, after receiving a heterogeneous data mapping request carrying the original data table structure, random sampling data of the original data table structure and the target data table structure to be mapped, the random sampling data can be feature preprocessed according to the original data table structure, and then the original data table structure, the preprocessed random sampling data and the target data table structure are input into the pre-trained heterogeneous data mapping model to obtain the predicted mapping result; finally, the target data content after the random sampling data is mapped from the original data table structure to the target data table structure is determined according to the predicted mapping result. Through the technical solution in this application, the machine learning algorithm can be used to identify and reduce misunderstandings or human errors that occur during manual mapping, improve the accuracy of heterogeneous data mapping, and use automated data mapping, which can greatly improve work efficiency, reduce technical personnel input and shorten the overall task cycle. In addition, the entire mapping process does not require the intervention of a large number of professionals, and the business skills requirements for professionals are relatively low.

[0057] Further, as Figure 1 and Figure 2 The specific implementation of the method shown in the embodiment of the present application provides a heterogeneous data mapping device based on a machine learning algorithm, such as Figure 3 As shown, the device includes: a receiving module 31, a processing module 32, an input module 33, and a determination module 34;

[0058] The receiving module 31 may be used to receive a heterogeneous data mapping request, wherein the heterogeneous data mapping request carries an original data table structure, randomly sampled data of the original data table structure, and a target data table structure to be mapped;

[0059] A processing module 32, which can be used to perform feature processing on the randomly sampled data according to the original data table structure to obtain the original data features, wherein the feature processing includes at least one of data cleaning, data screening, data residual filling and feature engineering;

[0060] An input module 33 may be used to input the original data table structure, original data features, and target data table structure into the pre-trained heterogeneous data mapping model to obtain a predicted mapping result;

[0061] The determination module 34 may be used to determine the target data content after the random sampling data is mapped from the original data table structure to the target data table structure according to the predicted mapping result.

[0062] In a specific application scenario, when performing feature processing on randomly sampled data, the processing module 32 can be specifically used to obtain a data conversion format matching the original data table structure and the corresponding feature vector space dimension if the randomly sampled data is determined to be structured or unstructured long text data based on the original data table structure; and convert the randomly sampled data into a vector feature matrix matching the feature vector space dimension based on the data conversion format.

[0063] In a specific application scenario, the heterogeneous data mapping model includes a first heterogeneous data mapping model for table matching and a second heterogeneous data mapping model for field matching; accordingly, in order to pre-train the heterogeneous data mapping model, such as Figure 4 As shown, the device further includes: a training module 35;

[0064] The training module 35 can be used to construct a training set based on historical heterogeneous data mapping results, and use the training set to train the first heterogeneous data mapping model and the second heterogeneous data mapping model respectively, so that the first heterogeneous data mapping model and the second heterogeneous data mapping model meet the preset training standard, and determine that the pre-training is completed.

[0065] In a specific application scenario, when the original data table structure, original data features and target data table structure are input into a pre-trained heterogeneous data mapping model to obtain a predicted mapping result, the input module 33 can be specifically used to input the original data table structure, original data features and target data table structure into a pre-trained first heterogeneous data mapping model to obtain a first predicted mapping result about the target table; and input the original data table structure, original data features and target data table structure into a pre-trained second heterogeneous data mapping model to obtain a second predicted mapping result for the target field in the target table.

[0066] In a specific application scenario, when determining the target data content after the randomly sampled data is mapped from the original data table structure to the target data table structure based on the prediction mapping results, the determination module 34 can be specifically used to filter out a preset number of prediction mapping tables that match the target data table structure in the first prediction mapping result in order from high to low confidence; filter out a preset number of prediction mapping fields in the second prediction mapping result in order from high to low confidence; and output the prediction mapping fields under the prediction mapping table so that the target data content under the target data table structure can be manually determined based on the prediction mapping fields under the prediction mapping table.

[0067] In specific application scenarios, in order to achieve fast mapping processing of heterogeneous data, such as Figure 4 As shown, the device further includes: a generating module 36;

[0068] The generation module 36 can be used to generate a heterogeneous data mapping table according to the random sampling data under the original data table structure and the target data content under the target data table structure, so as to screen the target data content matching the original data table structure, the random sampling data and the target data table structure based on the heterogeneous data mapping table.

[0069] In specific application scenarios, in order to improve the model accuracy of heterogeneous data mapping models, such as Figure 4 As shown, the device further includes: an updating module 37;

[0070] The updating module 37 can be used to update the training set according to the randomly sampled data under the original data table structure and the target data content under the target data table structure, so as to iteratively train the first heterogeneous data mapping model and the second heterogeneous data mapping model according to the updated training set.

[0071] It should be noted that for other corresponding descriptions of the functional units involved in the heterogeneous data mapping device based on machine learning algorithm provided in this embodiment, please refer to Figure 1 to Figure 2 The corresponding description will not be repeated here.

[0072] Based on the above Figure 1 to Figure 2 The method shown in the embodiment, accordingly, also provides a storage medium, which can be volatile or non-volatile, and stores computer-readable instructions, which are executed by the processor to implement the above-mentioned Figure 1 to Figure 2 The heterogeneous data mapping method based on machine learning algorithm is shown.

[0073] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0074] Based on the above Figure 1 to Figure 2 The method shown and Figure 3 , Figure 4 In order to achieve the above-mentioned purpose, the present embodiment further provides a computer device, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 to Figure 2 The heterogeneous data mapping method based on machine learning algorithm is shown.

[0075] Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0076] Those skilled in the art will appreciate that the computer device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0077] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned computer device and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the information processing entity device.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or can be implemented by hardware.

[0079] By applying the technical solution of the present application, compared with the current existing technology, after receiving a heterogeneous data mapping request carrying the original data table structure, random sampling data of the original data table structure, and the target data table structure to be mapped, the present application can perform feature preprocessing on the random sampling data according to the original data table structure, and then input the original data table structure, the preprocessed random sampling data, and the target data table structure into the pre-trained heterogeneous data mapping model to obtain the predicted mapping result; finally, according to the predicted mapping result, the target data content after the random sampling data is mapped from the original data table structure to the target data table structure is determined. Through the technical solution in the present application, the machine learning algorithm can be used to identify and reduce misunderstandings or human errors that occur during manual mapping, improve the accuracy of heterogeneous data mapping, and use automated data mapping, which can greatly improve work efficiency, reduce technical personnel input, and shorten the overall task cycle. In addition, the entire mapping process does not require the intervention of a large number of professionals, and the business skills requirements for professionals are relatively low.

[0080] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0081] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A heterogeneous data mapping method based on machine learning algorithm, characterized in that: include: receiving a heterogeneous data mapping request, wherein the heterogeneous data mapping request carries an original data table structure, randomly sampled data of the original data table structure, and a target data table structure to be mapped; Performing feature processing on the randomly sampled data according to the original data table structure to obtain original data features, wherein the feature processing includes at least one of data cleaning, data screening, data residual filling and feature engineering; Inputting the original data table structure, the original data features and the target data table structure into a pre-trained heterogeneous data mapping model to obtain a predicted mapping result, wherein the heterogeneous data mapping model is used to perform classification prediction on the heterogeneous data mapping result, output a table probability and a table confidence for a preset table, and output a field probability and a field confidence for a preset field; Determine, according to the predicted mapping result, the target data content of the randomly sampled data after being mapped from the original data table structure to the target data table structure; Performing feature processing on the randomly sampled data according to the original data table structure to obtain original data features includes: If the randomly sampled data is determined to be structured or unstructured long text data according to the original data table structure, a data conversion format matching the original data table structure and a corresponding feature vector space dimension are obtained; Converting the randomly sampled data into a vector feature matrix matching the dimension of the feature vector space according to the data conversion format; The inputting the original data table structure, the original data features, and the target data table structure into the pre-trained heterogeneous data mapping model to obtain a predicted mapping result includes: Inputting the original data table structure, the original data features, and the target data table structure into a pre-trained first heterogeneous data mapping model to obtain a first predicted mapping result about the target table; Inputting the original data table structure, the original data features, and the target data table structure into the pre-trained second heterogeneous data mapping model to obtain a second prediction mapping result of the target field in the target table; The determining, according to the predicted mapping result, target data content after the random sampling data is mapped from the original data table structure to the target data table structure comprises: Screening out a preset number of prediction mapping tables matching the target data table structure from the first prediction mapping results in descending order of table confidence; Filtering a preset number of prediction mapping fields from the second prediction mapping result in descending order of field confidence; The prediction mapping fields under the prediction mapping table are outputted so that the target data content under the target data table structure can be manually determined according to the prediction mapping fields under the prediction mapping table.

2. The method according to claim 1, characterized in that The heterogeneous data mapping model includes a first heterogeneous data mapping model for table matching and a second heterogeneous data mapping model for field matching; Before inputting the original data table structure, the original data features and the target data table structure into the pre-trained heterogeneous data mapping model to obtain the predicted mapping result, the method further includes: A training set is constructed based on historical heterogeneous data mapping results, and the first heterogeneous data mapping model and the second heterogeneous data mapping model are trained respectively using the training set, so that the first heterogeneous data mapping model and the second heterogeneous data mapping model meet preset training standards, and the pre-training is determined to be completed.

3. The method according to claim 1, characterized in that The method further comprises: A heterogeneous data mapping table is generated according to the randomly sampled data under the original data table structure and the target data content under the target data table structure, so as to screen the target data content matching the original data table structure, the randomly sampled data and the target data table structure based on the heterogeneous data mapping table.

4. The method according to claim 2, characterized in that: The method further comprises: The training set is updated according to the randomly sampled data under the original data table structure and the target data content under the target data table structure, so as to iteratively train the first heterogeneous data mapping model and the second heterogeneous data mapping model according to the updated training set.

5. A heterogeneous data mapping device based on machine learning algorithm, characterized in that: include: A receiving module, configured to receive a heterogeneous data mapping request, wherein the heterogeneous data mapping request carries an original data table structure, randomly sampled data of the original data table structure, and a target data table structure to be mapped; A processing module, used for performing feature processing on the randomly sampled data according to the original data table structure to obtain original data features, wherein the feature processing includes at least one of data cleaning, data screening, data residual filling and feature engineering; An input module, used to input the original data table structure, the original data features and the target data table structure into a pre-trained heterogeneous data mapping model to obtain a predicted mapping result, wherein the heterogeneous data mapping model is used to perform classification prediction on the heterogeneous data mapping result, output a table probability and a table confidence for a preset table, and output a field probability and a field confidence for a preset field; A determination module, configured to determine, according to the predicted mapping result, target data content of the randomly sampled data after the randomly sampled data is mapped from the original data table structure to the target data table structure; The processing module is specifically used to obtain a data conversion format matching the original data table structure and a corresponding feature vector space dimension if the random sampled data is determined to be structured or unstructured long text data according to the original data table structure; convert the random sampled data into a vector feature matrix matching the feature vector space dimension according to the data conversion format; The input module is specifically used to input the original data table structure, original data features and target data table structure into a pre-trained first heterogeneous data mapping model to obtain a first prediction mapping result about the target table; input the original data table structure, original data features and target data table structure into a pre-trained second heterogeneous data mapping model to obtain a second prediction mapping result of a target field in the target table; The determination module is specifically used to filter out a preset number of prediction mapping tables that match the target data table structure in the first prediction mapping result in order of table confidence from high to low; filter out a preset number of prediction mapping fields in the second prediction mapping result in order of field confidence from high to low; and output the prediction mapping fields under the prediction mapping table so that the target data content under the target data table structure can be manually determined based on the prediction mapping fields under the prediction mapping table.

6. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the heterogeneous data mapping method based on a machine learning algorithm described in any one of claims 1 to 4 is implemented.

7. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the program, the heterogeneous data mapping method based on the machine learning algorithm described in any one of claims 1 to 4 is implemented.

Images