Data cleaning model training method, data cleaning model using method, data cleaning model training device, data cleaning model using device, equipment and medium

Abstract

The invention provides a data cleaning model training method and device, a data cleaning model using method and device, equipment and a medium, and relates to the technical field of data processing. The method comprises the steps of obtaining multiple pieces of first uncleaned data, and preprocessing the multiple pieces of first uncleaned data to obtain a training data set; training a preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model; screening out a plurality of pieces of first target data from the plurality of pieces of first uncleaned data, and obtaining mark data corresponding to each piece of first target data; and training the basic model through the plurality of marked data to obtain a data cleaning model based on the basic model. According to the method, the model is trained by using the training data, manual marking data does not need to be used in the model training process, and only a small amount of manual marking data is used in the subsequent model adjusting process. According to the method, the use quantity of manual marking data is reduced, and the training time and cost of the data cleaning model are reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a training method, a use method, a device, equipment and a medium for a data cleaning model. Background Art

[0002] Data cleaning refers to the use of a series of techniques to identify, correct or delete erroneous or redundant data in uncleaned data, with the aim of improving the quality and consistency of data, thereby ensuring the accuracy of data analysis and decision-making. Data cleaning plays an important role in various applications. For example, in data mining and machine learning, cleaned data can improve the accuracy of results.

[0003] When training a data cleaning model, the existing technology first collects a training data set and manually marks the erroneous or abnormal data in it. Then, an appropriate model architecture is selected, such as a decision tree, random forest, or neural network, and the model is trained using the marked training data to enable it to identify abnormal data. The new data set is input into the trained model, and the trained model can automatically mark the erroneous or abnormal data in the new data set.

[0004] However, existing technologies have a high dependence on the quantity and quality of labeled data in model training, which means that a large amount of high-quality manually labeled data is required to train an accurate model. However, manually labeling a large amount of data will significantly increase the training time and cost of the data cleaning model. Summary of the invention

[0005] The present application provides a data cleaning model training method, use method, device, equipment and medium to solve the problems of long training time and high cost of data cleaning models in the prior art.

[0006] In a first aspect, the present application provides a method for training a data cleaning model, comprising:

[0007] Acquire a plurality of first uncleaned data, and preprocess the plurality of first uncleaned data to obtain a training data set;

[0008] Training a preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model;

[0009] Filtering out a plurality of first target data from the plurality of first uncleaned data, and acquiring label data corresponding to each of the first target data; wherein each of the label data is data obtained by labeling the corresponding first target data;

[0010] The basic model is trained by using a plurality of the labeled data to obtain a data cleaning model based on the basic model; wherein the data cleaning model is used to perform data cleaning on uncleaned data.

[0011] In a possible design, the training data set includes a plurality of training data, and the step of training a preset hybrid expert model by using the training data set to obtain a basic model based on the hybrid expert model includes:

[0012] Iteratively training the hybrid expert model through the multiple training data until the error of the model result output by the iteratively trained hybrid expert model is less than a preset threshold;

[0013] The hybrid expert model after the iterative training is determined as the basic model.

[0014] In one possible design, the target training time is any training time in the training iteration;

[0015] For the target training, iteratively training the hybrid expert model through the multiple training data includes:

[0016] Input the plurality of training data into the first hybrid expert model to obtain a loss value corresponding to each of the training data output by the first hybrid expert model; wherein the first hybrid expert model refers to the hybrid expert model before the target number of trainings;

[0017] When the multiple loss values ​​are not less than the preset threshold, a hybrid expert second model is obtained; wherein the hybrid expert second model is the hybrid expert first model after adjusting the model parameters according to the multiple loss values.

[0018] In a possible design, the acquiring the label data corresponding to each first target data includes:

[0019] Inputting the plurality of first target data into a preset data processing model to obtain first cleaned data corresponding to each of the first target data output by the data processing model;

[0020] According to the multiple first target data and the first cleaned data corresponding to each of the first target data, the marked data corresponding to each of the first target data is obtained.

[0021] In a possible design, the second target data is any one of the plurality of first target data;

[0022] For the second target data, obtaining the marked data corresponding to each of the first target data according to the plurality of first target data and the first cleaned data corresponding to each of the first target data includes:

[0023] When the loss value corresponding to the second target data is within a first preset value range, obtaining the marked data corresponding to the second target data according to the second target data and the first cleaned data corresponding to the second target data;

[0024] When the loss value corresponding to the second target data is within a second preset numerical range, obtaining the marking data corresponding to the second target data according to the second target data, the first cleaned data corresponding to the second target data, and the first deletion reason; wherein the minimum value of the second preset numerical range is greater than the maximum value of the first preset numerical range, and the first deletion reason is used to indicate that the second target data is duplicate data;

[0025] When the loss value corresponding to the second target data is within a third preset numerical range, the marking data corresponding to the second target data is obtained based on the second target data, and the first cleaned data and the second deletion reason corresponding to the second target data; wherein the minimum value of the third preset numerical range is greater than the maximum value of the second preset numerical range, and the second deletion reason is used to indicate that the second target data is sensitive data.

[0026] In a possible design, preprocessing the plurality of first uncleaned data to obtain a training data set includes:

[0027] Perform word segmentation on each of the first uncleaned data to obtain a plurality of segmented text data;

[0028] Performing word frequency statistics for the multiple segmented text data to obtain the frequency of each word appearing in the multiple segmented text data;

[0029] Creating a vocabulary list based on the frequency of each of the words;

[0030] According to the vocabulary, the plurality of segmented text data are subjected to vocabulary indexing to obtain the training data.

[0031] In a possible design, after training the basic model with the plurality of labeled data to obtain a data cleaning model based on the basic model, the method further includes:

[0032] Inputting the plurality of first target data into the data cleaning model to obtain second cleaned data output by the data cleaning model;

[0033] Acquire third cleaned data corresponding to each of the first target data; wherein each of the third cleaned data is data obtained after manually cleaning the corresponding first target data;

[0034] The deviation between each of the second cleaned data and the corresponding third cleaned data is calculated, and the data cleaning model is iteratively optimized according to the plurality of the deviations.

[0035] In a second aspect, the present application provides a method for using a data cleaning model, including:

[0036] Acquire the second uncleaned data;

[0037] The second uncleaned data is input into a preset data cleaning model to obtain the second uncleaned data after data cleaning output by the data cleaning model; wherein the data cleaning model is a model trained by the method described in any one of the first aspects.

[0038] In a third aspect, the present application provides a training device for a data cleaning model, comprising:

[0039] A first acquisition module, used for acquiring a plurality of first uncleaned data, and preprocessing the plurality of first uncleaned data to obtain a training data set;

[0040] A training module, used for training a preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model;

[0041] A screening module, used to screen out a plurality of first target data from the plurality of first uncleaned data, and obtain marking data corresponding to each of the first target data; wherein each of the marking data is data obtained by marking the corresponding first target data;

[0042] The adjustment module is used to train the basic model through a plurality of the labeled data to obtain a data cleaning model based on the basic model; wherein the data cleaning model is used to perform data cleaning for uncleaned data.

[0043] In a possible design, the training data set includes multiple training data, and the training module includes:

[0044] A training unit, configured to iteratively train the hybrid expert model using the plurality of training data until an error of a model result output by the iteratively trained hybrid expert model is less than a preset threshold;

[0045] A determination unit is used to determine the hybrid expert model after the iterative training as the basic model.

[0046] In one possible design, the target training time is any training time in the training iteration;

[0047] For the target training, the training unit includes:

[0048] A computing component, used for inputting the plurality of training data into a first hybrid expert model, and obtaining a loss value corresponding to each of the training data output by the first hybrid expert model; wherein the first hybrid expert model refers to the hybrid expert model before the target number of trainings;

[0049] An acquisition component is used to acquire a hybrid expert second model when a plurality of the loss values ​​are not less than the preset threshold; wherein the hybrid expert second model is a hybrid expert first model after adjusting model parameters according to the plurality of the loss values.

[0050] In a possible design, the screening module includes:

[0051] A first cleaned data calculation unit, configured to input the plurality of first target data into a preset data processing model, and obtain first cleaned data corresponding to each of the first target data output by the data processing model;

[0052] The marked data acquisition unit is used to obtain the marked data corresponding to each of the first target data according to the multiple first target data and the first cleaned data corresponding to each of the first target data.

[0053] In a possible design, the second target data is any one of the plurality of first target data;

[0054] For the second target data, the marked data acquiring unit includes:

[0055] a first judgment component, configured to obtain, when the loss value corresponding to the second target data is within a first preset value range, label data corresponding to the second target data according to the second target data and the first cleaned data corresponding to the second target data;

[0056] a second judgment component, configured to obtain, when the loss value corresponding to the second target data is within a second preset numerical range, marking data corresponding to the second target data according to the second target data, the first cleaned data corresponding to the second target data, and a first deletion reason; wherein the minimum value of the second preset numerical range is greater than the maximum value of the first preset numerical range, and the first deletion reason is used to indicate that the second target data is duplicate data;

[0057] A third judgment component is used to obtain marking data corresponding to the second target data based on the second target data, and the first cleaned data and the second deletion reason corresponding to the second target data when the loss value corresponding to the second target data is within a third preset numerical range; wherein the minimum value of the third preset numerical range is greater than the maximum value of the second preset numerical range, and the second deletion reason is used to indicate that the second target data is sensitive data.

[0058] In a possible design, the first acquisition module includes:

[0059] A word segmentation unit, used for performing word segmentation on each of the first uncleaned data to obtain a plurality of segmented text data;

[0060] A word frequency statistics unit, used for performing word frequency statistics on the multiple segmented text data to obtain the frequency of each word appearing in the multiple segmented text data;

[0061] A vocabulary creation unit, used to create a vocabulary according to the frequency corresponding to each of the words;

[0062] The vocabulary indexing unit is used to perform vocabulary indexing on the plurality of segmented text data according to the vocabulary table to obtain the training data.

[0063] In a possible design, the training device of the data cleaning model further includes:

[0064] A second cleaned data calculation module, used for inputting the plurality of first target data into the data cleaning model to obtain second cleaned data output by the data cleaning model;

[0065] A third acquisition module is used to acquire third cleaned data corresponding to each of the first target data; wherein each of the third cleaned data is data obtained after manually cleaning the corresponding first target data;

[0066] The optimization module is used to calculate the deviation between each of the second cleaned data and the corresponding third cleaned data, and iteratively optimize the data cleaning model according to the multiple deviations.

[0067] In a fourth aspect, the present application provides a device for using a data cleaning model, including:

[0068] A second acquisition module, used for acquiring second uncleaned data;

[0069] A cleaning module is used to input the second uncleaned data into a preset data cleaning model to obtain the second uncleaned data after data cleaning output by the data cleaning model; wherein the data cleaning model is a model trained by the method described in any one of the first aspects.

[0070] In a fifth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0071] The memory stores computer-executable instructions;

[0072] When the processor executes the computer-executable instructions stored in the memory, it is used to implement the training method of the data cleaning model as described in any one of the first aspects, or to implement the method of using the data cleaning model as described in the second aspect.

[0073] In a sixth aspect, the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the training method of the data cleaning model as described in any one of the first aspects, or to implement the method of using the data cleaning model as described in the second aspect.

[0074] In a seventh aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the training method of the data cleaning model as described in any one of the first aspects, or to implement the method of using the data cleaning model as described in the second aspect.

[0075] The present application provides a training method, a method of using, a device, an equipment and a medium for a data cleaning model, wherein the training method comprises: obtaining a plurality of first uncleaned data, and preprocessing the plurality of first uncleaned data to obtain a training data set; training a preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model; screening a plurality of first target data from the plurality of first uncleaned data, and obtaining the labeled data corresponding to each of the first target data; wherein each of the labeled data is data obtained by data labeling for the corresponding first target data; training the basic model through the plurality of labeled data to obtain a data cleaning model based on the basic model; wherein the data cleaning model is used to perform data cleaning for uncleaned data. The training method for a data cleaning model of the present application preprocesses the uncleaned data to obtain a training data set, uses the training data set to perform model training to obtain a basic model, and then uses a small amount of labeled data to adjust the basic model to obtain a data cleaning model. In the process of model training, there is no need to use a large amount of manually labeled data, and only a small amount of manually labeled data is needed to adjust the model, which reduces the amount of manually labeled data used in the training process of the data cleaning model, and reduces the time and cost of data cleaning model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0077] Figure 1 A schematic diagram of an application scenario of a training method for a data cleaning model provided in an embodiment of the present application;

[0078] Figure 2 Schematic diagram of the training method of the data cleaning model provided in the embodiment of the present application Figure 1 ;

[0079] Figure 3 Schematic diagram of the training method of the data cleaning model provided in the embodiment of the present application Figure 2 ;

[0080] Figure 4 A flowchart of a method for using a data cleaning model provided in an embodiment of the present application;

[0081] Figure 5 A schematic diagram of the structure of a training device for a data cleaning model provided in an embodiment of the present application;

[0082] Figure 6 A schematic diagram of the structure of a device for using a data cleaning model provided in an embodiment of the present application;

[0083] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0085] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference. It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way. In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more.

[0086] It should be noted that the "at..." in the embodiments of the present application can be the instant when a certain situation occurs, or it can be a period of time after a certain situation occurs, and the embodiments of the present application do not specifically limit this. In addition, the training method and use method of a data cleaning model provided in the embodiments of the present application are only used as examples, and the training method and use method of a data cleaning model can also include more or less content.

[0087] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0088] Data cleaning: is a key step in data analysis, which improves data quality and reliability by identifying, correcting or removing inaccurate, incomplete or inconsistent data in the data set. This process usually includes handling missing values, correcting erroneous data, standardizing data formats and removing duplicate data.

[0089] Data cleaning models: refers to systems or frameworks that use algorithms and rules to automatically identify and correct errors, inconsistencies, or incomplete information in data sets. These models use machine learning to detect anomalies, fill in missing values, standardize data formats, or remove duplicates to improve data quality.

[0090] Hybrid Expert Model: A machine learning framework that solves complex problems by combining multiple expert models, each of which focuses on a specific subtask or data feature. The framework usually includes a gating network that dynamically assigns input data to the most appropriate expert model, thereby improving the overall prediction performance and the ability to handle complex data.

[0091] Word segmentation: refers to the process of dividing a continuous text string into independent words or phrases, aiming to provide structured word units for subsequent text analysis, information retrieval, machine translation and other applications. Word segmentation can be achieved through rule-based methods or deep learning technology to improve the accuracy and efficiency of text processing.

[0092] Vocabulary: refers to a collection of all unique words in a specific language or text corpus, which provides a standardized vocabulary basis for text analysis and helps with operations such as vocabulary matching, feature extraction, and vectorization during text processing. Vocabularies can be customized according to usage scenarios and contain common words, proper nouns, or specific domain terms to support more accurate language model construction and text analysis.

[0093] Lexical indexing: refers to the process of converting words in a text into numerical indices so that computers can efficiently process and analyze text data. By assigning a unique integer index to each unique word, lexical indexing converts the text into a series of numerical values, which helps to standardize text data and improve computing efficiency.

[0094] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0095] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0096] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced in detail. Data cleaning model training refers to the process of building a model through machine learning technology so that it can automatically identify and correct errors, inconsistencies or missing information in the data set. Through model training, the data cleaning model can learn to extract features and patterns from labeled data and master how to correct and standardize data in different situations. The trained data cleaning model can automatically perform cleaning tasks on large-scale data sets, reduce the need for manual intervention, and ensure the accuracy and consistency of the data in subsequent analysis and application. Therefore, it is necessary to train the data cleaning model.

[0097] At present, the training method of data cleaning models is usually to collect and label data sets containing known errors and correct data, and then select a suitable machine learning model for training. During the training process, the model learns to identify and correct errors in the data, detects model performance through cross-validation and test sets, and iterates and optimizes based on the results. Ultimately, the optimized model can automatically identify and correct errors in new data, thereby improving data quality and consistency. Existing technologies are highly dependent on the quantity and quality of labeled data in model training, which means that a large amount of manually labeled data is required when training data cleaning models. However, manually labeling a large amount of data will increase the time and cost of the data cleaning model. Therefore, the current training methods of data cleaning models are time-consuming and costly.

[0098] Therefore, in order to solve the problem of time-consuming and high-cost training methods for data cleaning models in existing technologies, the study found that in order to solve this problem, it is necessary to adjust the architecture of the model and reduce the amount of manually labeled data used: ① The data cleaning task can be decomposed into multiple subtasks and executed in parallel on multiple processors or computing nodes to reduce the training time. Parallel computing uses the multi-core and distributed characteristics of modern computing architectures to accelerate the model training process and reduce the load on a single computing node; ② Before data cleaning, the data can be pre-processed and converted into a data type suitable for model reading, reducing the model's data processing steps and reducing the complexity of the model; ③ Design and use lightweight model architectures, such as lightweight neural networks or compressed networks, which are designed for low computing resource environments and have fewer parameters and computing requirements. Lightweight models can reduce training time and costs while maintaining good performance.

[0099] Specific:

[0100] You can select a model that combines multiple sub-models for data cleaning, with one sub-model processing one sub-task. At the same time, pre-process the data before model training, convert it into a data type suitable for model reading, and then use the converted data to train the model.

[0101] The training method of the data cleaning model of the embodiment of the present application uses a hybrid expert model for data cleaning. First, the uncleaned data is preprocessed to form a training data set. The training data set is used to train a preset hybrid expert model to obtain a basic model. Then, the target data is filtered out from the uncleaned data, and the corresponding labeled data is obtained for these target data. Finally, the basic model is further trained using these labeled data to obtain the final data cleaning model. The hybrid expert model includes multiple expert models, each of which focuses on processing a specific type of data or task, and each expert model can be deeply optimized in a specific field. Multiple expert models of the hybrid expert model can be calculated simultaneously, and this parallel calculation reduces the training time and cost of the hybrid expert model. In the model training process, there is no need to use a large amount of manually labeled data, only a small amount of manually labeled data is needed for model adjustment, which reduces the amount of manually labeled data used in the training process of the data cleaning model and reduces the time and cost of data cleaning model training.

[0102] Based on the above creative findings, the technical solution of the present application is proposed.

[0103] The following introduces the application scenarios of the training method of the data cleaning model provided by the embodiment of the present invention. Figure 1 Schematic diagram of application scenarios of the training method of the data cleaning model provided in the embodiment of the present application. Figure 1 As shown, the application scenario includes a server 101, a first user terminal 102 and a second user terminal 103, and the first user terminal 102 and the second user terminal 103 can be a mobile phone, a tablet computer or a computer, etc. The first user terminal 102 and the second user terminal 103 use the training method of the data cleaning model to perform model training, and then use the trained data cleaning model to perform data cleaning, and finally send the cleaned data to the server 101.

[0104] The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0105] Figure 2 Schematic diagram of the training method of the data cleaning model provided in the embodiment of the present application Figure 1 .like Figure 2 As shown, in this embodiment, the execution subject of the embodiment of the present invention is the first user terminal or the second user terminal. Then the training method of the data cleaning model provided in this embodiment includes the following steps:

[0106] S201 . Acquire a plurality of first uncleaned data, and preprocess the plurality of first uncleaned data to obtain a training data set.

[0107] Specifically, a large amount of raw, uncleaned data can be collected from data sources, which may contain noisy, incomplete, or inconsistent information. Preprocessing steps can include data deduplication, missing value filling, outlier detection, or data format standardization. Through preprocessing, the raw data is converted into a training dataset with a consistent structure, providing a high-quality data foundation for subsequent model training. This processing can provide a high-quality input dataset for the training of the hybrid expert model.

[0108] S202: Train a preset hybrid expert model using a training data set to obtain a basic model based on the hybrid expert model.

[0109] Specifically, a hybrid expert model architecture can be defined, which consists of multiple expert models, each of which focuses on different data features. Then, a gating network is used to dynamically select and weight the outputs of these expert models according to the characteristics of the input data to generate a comprehensive output result. This process is repeated through iteration and optimization, so that the base model based on the hybrid expert model can effectively learn and generalize complex data patterns. The diversity and flexibility of the hybrid expert model are used to build a basic model with broad adaptability. By combining the advantages of multiple expert models, the basic model can more accurately understand and process complex data features, providing a strong foundation for subsequent data cleaning models, thereby improving the effectiveness and efficiency of data cleaning.

[0110] S203: Filter out a plurality of first target data from the plurality of first uncleaned data, and obtain label data corresponding to each first target data.

[0111] Specifically, screening criteria or rules can be defined to identify uncleaned data that is representative or important for model training. These criteria can be based on the quality, completeness or relevance of the data. The screened first target data is then used in a data labeling process, usually performed manually, to ensure that each target data has accurate labeled data. These labeled data provide supervision information for the model, helping the model learn how to extract and correct erroneous or inconsistent information from uncleaned data. By training the basic model with labeled data, the model is able to identify erroneous data, so that similar cleaning tasks can be automatically performed when applied to new uncleaned data. This not only improves the efficiency of data processing, but also ensures the reliability and consistency of the data, providing a solid foundation for subsequent data analysis and decision-making. Among them, each labeled data is the data obtained by data labeling for the corresponding first target data.

[0112] S204: Train a basic model using multiple labeled data to obtain a data cleaning model based on the basic model.

[0113] Specifically, the labeled data can be combined with the basic model for supervised learning. The labeled data provides the correct input-output pairs to guide the model to adjust its parameters during the training process to minimize the error between the predicted output and the labeled data. During the training process, the model continuously optimizes its weights and biases through the back-propagation algorithm, so that it can accurately identify and correct errors or inconsistencies in the uncleaned data. After multiple rounds of iterative training, the basic model gradually evolves into a data cleaning model that can perform data cleaning tasks. The data cleaning model can automatically process uncleaned data, identify and correct erroneous data therein, thereby improving the quality of the data. Among them, the data cleaning model is used to clean the uncleaned data.

[0114] The present embodiment provides a method for training a data cleaning model, the method comprising: obtaining a plurality of first uncleaned data, and preprocessing the plurality of first uncleaned data to obtain a training data set; training a preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model; screening a plurality of first target data from the plurality of first uncleaned data, and obtaining the labeled data corresponding to each first target data; wherein each labeled data is data obtained by data labeling the corresponding first target data; training a basic model through the plurality of labeled data to obtain a data cleaning model based on the basic model; wherein the data cleaning model is used to perform data cleaning for the uncleaned data. A method for training a data cleaning model achieves the following technical effects: by using a training data set that has not been manually labeled to train a preset hybrid expert model to form a basic model, the need for manually labeled data is avoided in the initial stage. Subsequently, although manually labeled data is still used in the adjustment process of the model, the required amount is greatly reduced, and only a small amount of labeled data is required to complete the optimization of the model. This method effectively reduces the dependence on manually labeled data, while speeding up the training process of the model, and ultimately achieves more efficient and lower-cost data cleaning model training.

[0115] In a possible design, the training data set includes multiple training data. Then, S202, training a preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model includes:

[0116] S2011. Iteratively train the hybrid expert model through multiple training data until the error of the model result output by the iteratively trained hybrid expert model is less than a preset threshold.

[0117] Specifically, the training data set can be input into the hybrid expert model in batches for training. In each iteration, the hybrid expert model makes predictions based on the input data, compares the predictions with the actual results, and calculates the error. Then, the model parameters are adjusted through the back-propagation algorithm to reduce the error. This process is repeated until the error drops below the preset threshold. This process is used to ensure that the hybrid expert model can effectively learn and capture the features and patterns in the training data, thereby improving the generalization ability and prediction accuracy of the model. Through iterative training, the model can gradually optimize its parameters so that it can maintain high performance even when processing unseen data. This lays a solid foundation for the subsequent construction of a data cleaning model based on the basic model, ensuring the reliability of the data cleaning process.

[0118] S2012. Determine the hybrid expert model after iterative training as the basic model.

[0119] Specifically, after the hybrid expert model has been trained for multiple iterations, the parameters of the model have been optimized, so that the error of the model on the training data set is reduced to below the preset threshold. At this point, the performance of the model has reached the design requirements, and it can be considered that the model has good generalization ability and prediction accuracy. Therefore, the parameters and structure of the hybrid expert model at this time are fixed and saved as the basic model. This process is used to provide an optimized and reliable basic model for the construction of the subsequent data cleaning model. As the core part of the data cleaning model, the basic model can effectively process and analyze the input data and provide necessary support for the data cleaning task. By using the iteratively optimized basic model, the data cleaning model can more accurately identify and process the noise and anomalies in the uncleaned data, thereby improving the quality and efficiency of data cleaning.

[0120] The technical effect of the scheme in this embodiment is: by iteratively training the hybrid expert model to achieve an error less than a preset threshold, an optimized basic model is ensured to improve the accuracy and reliability of the data cleaning model, so that it can more effectively handle noise and anomalies in uncleaned data, and improve the overall quality and efficiency of data cleaning.

[0121] In a possible design, the target training is any training in the iterative training; for the target training, the hybrid expert model is iteratively trained through multiple training data in S2011, including:

[0122] S20111. Input multiple training data into the first hybrid expert model to obtain the loss value corresponding to each training data output by the first hybrid expert model.

[0123] Specifically, each data sample in the training data set can be input into the hybrid expert first model for processing. The model predicts the input data based on its current parameters and structure. Then, the model's prediction results are compared with the actual results, and the loss value of each data sample is calculated. The loss value is usually measured by a loss function to measure the difference between the predicted result and the actual result. The loss function can be a task loss, an expert load balancing loss, or a sparsification loss. Task loss is a standard loss function for a specific task, usually a cross entropy loss, a mean square error, etc.; the expert load balancing loss promotes the expert selection probability distribution output by the gating network to be as uniform as possible by adding a load balancing term, which can prevent the gating network from always selecting certain experts and ignoring other experts. The expert load balancing loss aims to make all experts reasonably used during the training process; the sparsification loss is used to encourage the gating network to activate only a few experts on each sample to improve computational efficiency. This process is used to detect the performance of the hybrid expert first model in the current training stage. By calculating the loss value of each training data sample, the errors and deficiencies of the model in the prediction process can be identified. These loss values ​​provide a basis for subsequent model parameter adjustments, allowing the model to gradually optimize its parameters during iterative training and improve prediction accuracy and generalization ability. Among them, the hybrid expert first model refers to the hybrid expert model before the target training.

[0124] S20112. When the multiple loss values ​​are not less than a preset threshold, obtain a second hybrid expert model.

[0125] Specifically, the above-mentioned "error of the model result output by the hybrid expert model" is this loss value. When multiple loss values ​​are greater than the preset threshold, the error between the output result of the model and the expected result is large, indicating that the model has not yet reached the ideal performance level. In this case, the model parameters need to be further adjusted to improve the performance of the model. This process involves obtaining the hybrid expert second model and adjusting the parameters of the hybrid expert first model to reduce the loss value, thereby improving the accuracy and generalization ability of the model. This process ensures that the model is continuously optimized during the training process and gradually approaches the optimal solution. When multiple loss values ​​are less than the preset threshold, the error between the output result of the model and the expected result is small enough to meet or exceed the set performance standard. In this case, it can be considered that the model has been trained to a satisfactory state and no further parameter adjustment is required. Therefore, the iterative training can be stopped and the current hybrid expert model is determined as the basic model. This process is used to avoid overtraining, save computing resources, and ensure that the performance of the model has reached the expected standard. Among them, the hybrid expert second model is the hybrid expert first model after adjusting the model parameters according to multiple loss values.

[0126] The technical effect of the scheme in this embodiment is: through iterative training, the parameters of the hybrid expert model are dynamically adjusted to gradually reduce the loss value of the training data. When the loss value is not less than the preset threshold, the hybrid expert second model is obtained and adjusted to ensure that the model is continuously optimized during the training process. This method effectively improves the accuracy and robustness of the model, so that the final basic model can better adapt to and process uncleaned data, thereby laying a solid foundation for the subsequent data cleaning process.

[0127] In a possible design, obtaining the label data corresponding to each first target data in S203 includes:

[0128] S2031. Input a plurality of first target data into a preset data processing model to obtain first cleaned data corresponding to each first target data output by the data processing model.

[0129] Specifically, the calculation of the first cleaned data can be completed by using a preset data processing model. The model can be a rule-based system, a machine learning model, or a deep learning network, which is specifically used to identify and correct anomalies, errors, or inconsistencies in the data. By inputting the first target data, the model automatically performs a series of cleaning operations, such as missing value filling, format standardization, outlier detection and correction, etc., to generate more accurate and consistent first cleaned data. By converting the first target data into the first cleaned data, the accuracy and efficiency of the labeling process can be ensured, the cleaned data is closer to the real data distribution, and human intervention and errors in the labeling process are reduced.

[0130] S2032. Obtain marked data corresponding to each first target data according to the plurality of first target data and the first cleaned data corresponding to each first target data.

[0131] Specifically, the first target data and the first cleaned data corresponding to each first target data can be put into a two-dimensional array to obtain labeled data represented by a two-dimensional array. This process can provide high-quality supervision information for the training of the data cleaning model. By generating labeled data, the model can learn how to identify and correct errors or inconsistencies from uncleaned data. This not only improves the cleaning ability and accuracy of the model, but also reduces the dependence on manual labeling, allowing the model to work effectively on larger data sets. Labeled data, as the basis for training, ensures that the model can perform consistent and efficient cleaning operations when faced with new data.

[0132] The technical effect of the solution in this embodiment is: by inputting the first target data into the preset data processing model, generating the corresponding first cleaned data, and generating the labeled data based on this, the training quality and efficiency of the data cleaning model are improved. In this way, the model can more accurately learn and identify anomalies and errors in the data, thereby realizing a more automated and efficient data cleaning process in practical applications. This method not only reduces the need for manual intervention, but also improves the accuracy and reliability of the model when processing uncleaned data.

[0133] In a possible design, the second target data is any one of the plurality of first target data; for the second target data, S2032, according to the plurality of first target data and the first cleaned data corresponding to each first target data, obtaining the marked data corresponding to each first target data, includes:

[0134] S20321. When the loss value corresponding to the second target data is within a first preset value range, obtain the marked data corresponding to the second target data according to the second target data and the first cleaned data corresponding to the second target data.

[0135] Specifically, when the loss value corresponding to the second target data is within the first preset numerical range, it means that the data performs well in the prediction of the current model with a small error. At this time, it can be considered that the cleaning effect of the data is good, so the second target data and its corresponding first cleaned data can be directly used to generate labeled data. The second target data can be input into the data processing model to obtain the cleaned result, and then the original data and the cleaned data are combined to generate data labels for training. This process ensures that the model can identify and learn effective information features in the data, thereby improving the accuracy and reliability of the data cleaning model. Accurate labeled data can be provided for the data cleaning model so that the model can better learn how to extract and retain important information from uncleaned data. By generating labeled data with a low loss value, the model can more effectively learn the essential characteristics of the data and reduce the impact of noise and errors. This helps to improve the data cleaning ability of the model in practical applications, enabling it to more accurately process and clean new unlabeled data.

[0136] S20322. When the loss value corresponding to the second target data is within a second preset value range, the marking data corresponding to the second target data is obtained according to the second target data, and the first cleaned data and the first deletion reason corresponding to the second target data.

[0137] Specifically, when the loss value corresponding to the second target data is within the second preset numerical range, it indicates that there are certain problems in the performance of the data in the model, which may be caused by the repetitiveness of the data. At this time, it is necessary to combine the second target data and its corresponding first cleaned data, and introduce the first deletion reason to generate labeled data. It is possible to determine whether the data is duplicate data by analyzing the characteristics of the data and the results after cleaning, and record this information as part of the label. This process ensures that the model can identify and process duplicate data, thereby reducing errors caused by data redundancy during training. This process can help the data cleaning model effectively identify and process duplicate data to improve the generalization ability and accuracy of the model. By generating labeled data containing deletion reasons when the loss value is within the second preset numerical range, the model can learn how to detect and process duplicate data, thereby reducing the impact of data redundancy on the cleaning effect in practical applications. This helps to improve the efficiency and quality of data cleaning, so that the model can more accurately process and clean new unlabeled data. Among them, the minimum value of the second preset numerical range is greater than the maximum value of the first preset numerical range, and the first deletion reason is used to indicate that the second target data is duplicate data.

[0138] S20323. When the loss value corresponding to the second target data is within a third preset value range, the marking data corresponding to the second target data is obtained according to the second target data, and the first cleaned data and the second deletion reason corresponding to the second target data.

[0139] Specifically, when the loss value corresponding to the second target data is within the third preset numerical range, this indicates that there are significant problems in the performance of the data in the model, which may be caused by the sensitivity of the data. At this time, it is necessary to combine the second target data and its corresponding first cleaned data, and introduce the second deletion reason to generate labeled data. By analyzing the characteristics and context of the data, possible sensitive information can be identified and recorded as part of the label. This process ensures that the model can identify and process sensitive data, thereby avoiding errors and deviations caused by sensitive information during the training process. This process can help the data cleaning model effectively identify and process sensitive data to improve the security and compliance of the model. By generating labeled data containing deletion reasons when the loss value is within the third preset numerical range, the model can learn how to detect and process sensitive data, thereby protecting privacy and sensitive information in practical applications. This helps to improve the security of data cleaning and enables the model to follow relevant privacy and security standards when processing and cleaning new unlabeled data. Among them, the minimum value of the third preset numerical range is greater than the maximum value of the second preset numerical range, and the second deletion reason is used to indicate that the second target data is sensitive data.

[0140] The technical effect of the scheme in this embodiment is: by segmenting the loss value of the second target data, the data can be accurately identified and classified as normal, duplicate or sensitive data, and the corresponding labeled data can be generated accordingly. This segmentation processing method improves the accuracy and efficiency of the data cleaning model, enabling it to more effectively handle different types of data problems, thereby reducing the impact of redundant data and protecting sensitive information. This process not only improves the cleaning ability of the model, but also enhances its reliability and security in practical applications.

[0141] In a possible design, the preprocessing of the plurality of first uncleaned data in S201 to obtain a training data set includes:

[0142] S2011. Segment each first uncleaned data to obtain a plurality of segmented text data.

[0143] Specifically, you can use word segmentation algorithms or tools (such as regular expressions, dictionary matching, or machine learning models) to break down continuous text strings into independent words or phrases. These tools can segment sentences into meaningful vocabulary units based on the grammatical and lexical features of the language. The result of word segmentation is to convert the original uncleaned data into a vocabulary list that is easier to analyze and process, providing a basis for subsequent data cleaning and model training. By breaking down the text into words, the model can better understand and process the information in the text and identify important features.

[0144] S2012: Perform word frequency statistics on the multiple segmented text data to obtain the frequency of each word appearing in the multiple segmented text data.

[0145] Specifically, you can traverse all the text data after word segmentation and use a hash table or dictionary data structure to record the number of occurrences of each word. Then, divide the number of occurrences of each word by the total number of words in the text data set to get its frequency of occurrence. This process can be implemented using built-in functions in programming languages. By calculating the frequency of occurrence of words, the model can understand which words are more representative or informative in the data set. This is crucial for training data cleaning models, because high-frequency words usually contain more semantic information, which helps the model understand and process text data more accurately. In addition, word frequency statistics can also be used to create word lists and index words, providing a basis for subsequent model training and data cleaning.

[0146] S2013. Create a vocabulary according to the frequency of each word.

[0147] Specifically, the words can be sorted from high to low in frequency, and then a word list can be created according to the set word list size. This word list can be stored in a data structure such as a list or an array, where each word is assigned a unique index. Creating a word list can provide a basis for the subsequent word indexing step, so that text data can be converted into a numerical form that can be processed by the model. The high-frequency words contained in the word list are usually the most representative words in the data set, which can help the model better capture important features and patterns in the text. By using a word list, the model can convert text data into an index sequence, so that the data can be processed and analyzed more efficiently during the training process. This process helps to improve the accuracy and efficiency of the data cleaning model.

[0148] S2014. According to the vocabulary, the plurality of segmented text data are indexed to obtain training data.

[0149] Specifically, for each word segmented text data, traverse each word in it, find the position of the word in the vocabulary, and replace it with the corresponding index value. This process can be efficiently implemented through dictionaries or mapping data structures in programming languages. Vocabulary indexing can convert text data into numerical form for easy processing by machine learning models. By replacing words with indexes, the model can process text data in a standardized way, identify and learn patterns and features. The training data after vocabulary indexing provides structured input for the model, which helps to improve the accuracy and efficiency of the model.

[0150] The technical effect of the scheme in this embodiment is: by performing word segmentation, word frequency statistics, word list creation and vocabulary indexing on the uncleaned data, the original text data is converted into a structured numerical form. This enables the data cleaning model to process and analyze text data more efficiently, identify important features and patterns therein, thereby improving the training effect and cleaning accuracy of the model. Through this preprocessing method, the model can more accurately remove noise and redundant information, and improve the overall efficiency and reliability of data cleaning.

[0151] Figure 3 Schematic diagram of the training method of the data cleaning model provided in the embodiment of the present application Figure 2 In this embodiment, Figure 2 Based on the provided embodiment, the training method of the data cleaning model is further explained. The training method of the data cleaning model includes:

[0152] S301 , obtaining a plurality of first uncleaned data, and preprocessing the plurality of first uncleaned data to obtain a training data set.

[0153] S302: Train a preset hybrid expert model using a training data set to obtain a basic model based on the hybrid expert model.

[0154] S303, screening out a plurality of first target data from a plurality of first uncleaned data, and acquiring marked data corresponding to each first target data; wherein each marked data is data obtained by marking the corresponding first target data.

[0155] S304, training a basic model through multiple labeled data to obtain a data cleaning model based on the basic model; wherein the data cleaning model is used to clean the uncleaned data.

[0156] S301 - S304 are similar to S201 - S204 and will not be described in detail in this embodiment.

[0157] S305 , inputting a plurality of first target data into a data cleaning model to obtain second cleaned data output by the data cleaning model.

[0158] Specifically, using a trained data cleaning model to automatically clean uncleaned data can improve the efficiency and consistency of data cleaning. By inputting the target data into the model and obtaining the cleaned output, noise, redundancy or irrelevant information in the data can be quickly identified and removed. This not only reduces the workload of manual cleaning, but also improves the reliability and quality of the cleaning results, making the data more suitable for subsequent analysis and application.

[0159] S306: Obtain third cleaned data corresponding to each first target data.

[0160] Specifically, manual data cleaning refers to the detailed inspection and processing of the first target data by data experts or manual labelers to remove noise, correct errors or supplement missing information. Each processed data instance is the third cleaned data, which represents high-quality data after manual verification and cleaning. This process can provide a high-quality benchmark data set for the model to optimize the performance of the data cleaning model. By comparing the second cleaned data output by the model with the third cleaned data obtained by manual cleaning, the deviation can be calculated to identify possible deficiencies in the model during the cleaning process. This feedback mechanism helps to iteratively optimize the model and improve the accuracy and reliability of the model. Among them, each third cleaned data is the data obtained after the first target data corresponding to the manual data cleaning.

[0161] S307: Calculate the deviation between each second cleaned data and the corresponding third cleaned data, and iteratively optimize the data cleaning model according to the multiple deviations.

[0162] Specifically, by comparing the second cleaned data with the third cleaned data, a specific error metric can be used to calculate the deviation. These deviations reflect the difference between the model output and the manual standard. Using this deviation information, the model parameters are adjusted through backpropagation or other optimization algorithms to reduce the deviation. This process is usually carried out in multiple iterations to gradually optimize the performance of the model. The accuracy and efficiency of the data cleaning model are continuously improved through the feedback mechanism. By calculating the deviation and performing iterative optimization, the model can learn a cleaning strategy that is closer to the manual cleaning standard, thereby improving the quality of automated cleaning. This optimization process helps to enhance the robustness and reliability of the model when processing complex and diverse data sets, so that it can provide high-quality data cleaning results in practical applications.

[0163] The technical effect of the solution in this embodiment is: by comparing the output of the data cleaning model with the manual cleaning results, calculating the deviation and performing iterative optimization, the model can continuously adjust and improve its cleaning strategy. This process improves the accuracy and robustness of the model, making it closer to the standard of manual cleaning in automated data cleaning, thereby improving the quality and consistency of the cleaning results. Through this feedback optimization mechanism, the model can provide more reliable and efficient cleaning services when processing complex and diverse data sets.

[0164] Figure 4 A flow chart of a method for using a data cleaning model provided in an embodiment of the present application. Figure 4 As shown, in this embodiment, the execution subject of the embodiment of the present invention is the first user terminal or the second user terminal. Then the method for using the data cleaning model provided in this embodiment includes:

[0165] S401, obtaining second uncleaned data.

[0166] Specifically, the second uncleaned data and the first uncleaned data may be the same or different. Raw data may be collected from various data sources, which may contain noisy, incomplete, or inconsistent information. Data sources may include databases, sensors, log files, user input, or web crawlers, etc. Obtaining the second uncleaned data can provide input data for the data cleaning model so that the model can perform data cleaning operations. Uncleaned data usually contains errors, duplications, or irrelevant information that needs to be processed. By inputting this data into a preset data cleaning model, the model can automatically identify and correct problems in the data, thereby improving the quality and consistency of the data.

[0167] S402, input the second uncleaned data into a preset data cleaning model, and obtain the second uncleaned data after data cleaning output by the data cleaning model; wherein the data cleaning model is based on Figure 2 and Figure 3The model is trained by any of the methods in .

[0168] Specifically, you can use programming languages ​​and related machine learning libraries to write code to format the second uncleaned data into an input format acceptable to the model. Call the prediction function of the model, input the formatted data into the model, and the model will automatically perform the cleaning operation and output the cleaned data. By using a trained data cleaning model, you can effectively identify and correct errors in the data, remove duplicates, process missing values, or filter out irrelevant information. This not only saves time and energy in manually cleaning data, but also improves the consistency and accuracy of data processing, thereby providing a more reliable foundation for subsequent business analysis, machine learning model training, or other data-driven decisions.

[0169] The technical effect of the scheme in this embodiment is: by using a trained data cleaning model to automatically process uncleaned data, the quality and consistency of the data are improved. The method can effectively identify and correct errors in the data, remove duplicates, process missing values, or filter out irrelevant information, thereby reducing the need for manual intervention and saving time and resources. This automated data cleaning process provides a more reliable and accurate data foundation for subsequent data analysis, machine learning model training, or business decision-making, thereby improving the efficiency and effectiveness of the overall data processing process.

[0170] Figure 5 A schematic diagram of the structure of a training device for a data cleaning model provided in an embodiment of the present application. Figure 5 As shown, the training device of the data cleaning model includes:

[0171] The first acquisition module 501 is used to acquire a plurality of first uncleaned data and pre-process the plurality of first uncleaned data to obtain a training data set.

[0172] The training module 502 is used to train the preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model.

[0173] The screening module 503 is used to screen out a plurality of first target data from a plurality of first uncleaned data, and obtain the marking data corresponding to each first target data; wherein each marking data is data obtained by marking the corresponding first target data.

[0174] The adjustment module 504 is used to train the basic model through multiple labeled data to obtain a data cleaning model based on the basic model; wherein the data cleaning model is used to clean the uncleaned data.

[0175] In a possible design, the training data set includes multiple training data, and the training module 502 includes:

[0176] The training unit is used to iteratively train the hybrid expert model through multiple training data until the error of the model result output by the hybrid expert model after the iterative training is less than a preset threshold.

[0177] The determination unit is used to determine the hybrid expert model after iterative training as the basic model.

[0178] In one possible design, the target training round is any training round in the training iteration.

[0179] For target training, training units include:

[0180] A computing component is used to input multiple training data into a first hybrid expert model to obtain a loss value corresponding to each training data output by the first hybrid expert model; wherein the first hybrid expert model refers to the hybrid expert model before the target number of trainings.

[0181] An acquisition component is used to obtain a hybrid expert second model when multiple loss values ​​are not less than a preset threshold; wherein the hybrid expert second model is a hybrid expert first model after adjusting model parameters according to multiple loss values.

[0182] In a possible design, the screening module 503 includes:

[0183] The first cleaned data calculation unit is used to input a plurality of first target data into a preset data processing model to obtain first cleaned data corresponding to each first target data output by the data processing model.

[0184] The marked data acquisition unit is used to obtain the marked data corresponding to each first target data according to the multiple first target data and the first cleaned data corresponding to each first target data.

[0185] In a possible design, the second target data is any one of a plurality of first target data.

[0186] For the second target data, a marking data acquisition unit includes:

[0187] The first judgment component is used to obtain the marking data corresponding to the second target data according to the second target data and the first cleaned data corresponding to the second target data when the loss value corresponding to the second target data is within a first preset value range.

[0188] The second judgment component is used to obtain the marking data corresponding to the second target data based on the second target data, the first cleaned data corresponding to the second target data, and the first deletion reason when the loss value corresponding to the second target data is within a second preset numerical range; wherein the minimum value of the second preset numerical range is greater than the maximum value of the first preset numerical range, and the first deletion reason is used to indicate that the second target data is duplicate data.

[0189] The third judgment component is used to obtain the marking data corresponding to the second target data according to the second target data, the first cleaned data corresponding to the second target data, and the second deletion reason when the loss value corresponding to the second target data is within a third preset numerical range; wherein the minimum value of the third preset numerical range is greater than the maximum value of the second preset numerical range, and the second deletion reason is used to indicate that the second target data is sensitive data.

[0190] In a possible design, the first acquisition module 501 includes:

[0191] The word segmentation unit is used to perform word segmentation on each first uncleaned data to obtain a plurality of segmented text data.

[0192] The word frequency statistics unit is used to perform word frequency statistics on multiple word segmented text data to obtain the frequency of each word appearing in the multiple word segmented text data.

[0193] The vocabulary creation unit is used to create a vocabulary according to the frequency corresponding to each word.

[0194] The vocabulary indexing unit is used to perform vocabulary indexing on a plurality of segmented text data according to a vocabulary list to obtain training data.

[0195] In a possible design, the training device of the data cleaning model also includes:

[0196] The second cleaned data calculation module 505 is used to input a plurality of first target data into the data cleaning model to obtain second cleaned data output by the data cleaning model.

[0197] The third acquisition module 506 is used to acquire third cleaned data corresponding to each first target data; wherein each third cleaned data is data obtained after manually cleaning the corresponding first target data.

[0198] The optimization module 507 is used to calculate the deviation between each second cleaned data and the corresponding third cleaned data, and iteratively optimize the data cleaning model according to the multiple deviations.

[0199] The training device of the data cleaning model provided in this embodiment can execute Figure 2-Figure 3The technical solution of the method embodiment shown in the figure has the same implementation principle and technical effect as Figure 2-Figure 3 The method embodiments shown are similar and will not be described in detail here.

[0200] Figure 6 This is a schematic diagram of the structure of the device for using the data cleaning model provided in the embodiment of the present application. Figure 6 As shown, the device for using the data cleaning model includes:

[0201] The second acquisition module 601 is used to acquire second uncleaned data.

[0202] The cleaning module 602 is used to input the second uncleaned data into a preset data cleaning model to obtain the second uncleaned data after data cleaning output by the data cleaning model; wherein the data cleaning model is a model trained by any method of the first aspect.

[0203] The device for using the data cleaning model provided in this embodiment can execute Figure 4 The technical solution of the method embodiment shown in the figure has the same implementation principle and technical effect as Figure 4 The method embodiments shown are similar and will not be described in detail here.

[0204] Figure 7 The hardware structure diagram of the electronic device provided in the embodiment of the present application is shown in FIG. Figure 7 As shown, the electronic device includes: at least one processor 710 and a memory 720. The electronic device also includes a communication component 730. The processor 710, the memory 720 and the communication component 730 are connected via a bus 740.

[0205] In the specific implementation process, at least one processor 710 executes the computer execution instructions stored in the memory 720, so that the at least one processor 710 is used to implement the training method of the data cleaning model or the use method of the data cleaning model in the above-mentioned embodiment.

[0206] The specific implementation process of the processor 710 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0207] In the above embodiments, it should be understood that the processor 710 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0208] The memory 720 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0209] The bus 740 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 740 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 740 in the drawings of the present application is not limited to only one bus or one type of bus.

[0210] The above-mentioned functions implemented by the electronic device and the main control device introduce the scheme provided by the embodiment of the present invention. It can be understood that in order to implement the above-mentioned functions, the electronic device or the main control device includes a hardware structure and / or software module corresponding to the execution of each function. In combination with the units and algorithm steps of each example described in the embodiment disclosed in the embodiment of the present invention, the embodiment of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present invention.

[0211] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the training method or use method of the data cleaning model of the above embodiment is implemented. In the specific implementation of the aforementioned multi-protocol label switching message transmission method, each module can be implemented as a processor.

[0212] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0213] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in an electronic device or a main control device.

[0214] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the training method of the data cleaning model or the method of using the data cleaning model of the above-mentioned embodiment.

[0215] The computer program is stored in a readable storage medium. At least one processor can read the computer program from the readable storage medium. At least one processor executes the computer program to execute the solution provided in any of the above embodiments.

[0216] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions. The above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0217] So far, the technical solution of the present application has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments, and the above embodiments are only used to illustrate the technical solution of the present application rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a data cleaning model, characterized in that: The method comprises: Acquire a plurality of first uncleaned data, and preprocess the plurality of first uncleaned data to obtain a training data set; Training a preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model; Filtering out a plurality of first target data from the plurality of first uncleaned data, and acquiring label data corresponding to each of the first target data; wherein each of the label data is data obtained by labeling the corresponding first target data; The basic model is trained by using a plurality of the labeled data to obtain a data cleaning model based on the basic model; wherein the data cleaning model is used to perform data cleaning on uncleaned data.

2. The data cleaning model training method according to claim 1, characterized in that: The training data set includes a plurality of training data, and the training of a preset hybrid expert model by the training data set to obtain a basic model based on the hybrid expert model includes: Iteratively training the hybrid expert model through the multiple training data until the error of the model result output by the iteratively trained hybrid expert model is less than a preset threshold; The hybrid expert model after the iterative training is determined as the basic model.

3. The training method of the data cleaning model according to claim 2, characterized in that: The target training time is any training time in the iterative training; For the target training, iteratively training the hybrid expert model through the multiple training data includes: Input the plurality of training data into the first hybrid expert model to obtain a loss value corresponding to each of the training data output by the first hybrid expert model; wherein the first hybrid expert model refers to the hybrid expert model before the target number of trainings; When the multiple loss values ​​are not less than the preset threshold, a hybrid expert second model is obtained; wherein the hybrid expert second model is the hybrid expert first model after adjusting the model parameters according to the multiple loss values.

4. The training method of the data cleaning model according to claim 3, characterized in that: The acquiring the label data corresponding to each of the first target data includes: Inputting the plurality of first target data into a preset data processing model to obtain first cleaned data corresponding to each of the first target data output by the data processing model; According to the multiple first target data and the first cleaned data corresponding to each of the first target data, the marked data corresponding to each of the first target data is obtained.

5. The data cleaning model training method according to claim 4, characterized in that: The second target data is any one of the plurality of first target data; For the second target data, obtaining the marked data corresponding to each of the first target data according to the plurality of first target data and the first cleaned data corresponding to each of the first target data includes: When the loss value corresponding to the second target data is within a first preset value range, obtaining the marked data corresponding to the second target data according to the second target data and the first cleaned data corresponding to the second target data; When the loss value corresponding to the second target data is within a second preset numerical range, obtaining the marking data corresponding to the second target data according to the second target data, the first cleaned data corresponding to the second target data, and the first deletion reason; wherein the minimum value of the second preset numerical range is greater than the maximum value of the first preset numerical range, and the first deletion reason is used to indicate that the second target data is duplicate data; When the loss value corresponding to the second target data is within a third preset numerical range, the marking data corresponding to the second target data is obtained based on the second target data, and the first cleaned data and the second deletion reason corresponding to the second target data; wherein the minimum value of the third preset numerical range is greater than the maximum value of the second preset numerical range, and the second deletion reason is used to indicate that the second target data is sensitive data.

6. The method for training a data cleaning model according to any one of claims 1 to 5, characterized in that: The preprocessing of the plurality of the first uncleaned data to obtain a training data set includes: Perform word segmentation on each of the first uncleaned data to obtain a plurality of segmented text data; Performing word frequency statistics for the multiple segmented text data to obtain the frequency of each word appearing in the multiple segmented text data; Creating a vocabulary list based on the frequency of each of the words; According to the vocabulary, the plurality of segmented text data are subjected to vocabulary indexing to obtain the training data.

7. The data cleaning model training method according to claim 1, characterized in that: After the basic model is trained by the plurality of labeled data to obtain a data cleaning model based on the basic model, the method further includes: Inputting the plurality of first target data into the data cleaning model to obtain second cleaned data output by the data cleaning model; Acquire third cleaned data corresponding to each of the first target data; wherein each of the third cleaned data is data obtained after manually cleaning the corresponding first target data; The deviation between each of the second cleaned data and the corresponding third cleaned data is calculated, and the data cleaning model is iteratively optimized according to the plurality of the deviations.

8. A method for using a data cleaning model, characterized in that: The method comprises: Acquire the second uncleaned data; The second uncleaned data is input into a preset data cleaning model to obtain the second uncleaned data after data cleaning output by the data cleaning model; wherein the data cleaning model is a model trained by the method according to any one of claims 1 to 7.

9. A training device for a data cleaning model, characterized in that: include: A first acquisition module, used for acquiring a plurality of first uncleaned data, and preprocessing the plurality of first uncleaned data to obtain a training data set; A training module, used for training a preset hybrid expert model through the training data set to obtain a basic model based on the hybrid expert model; A screening module, used to screen out a plurality of first target data from the plurality of first uncleaned data, and obtain marking data corresponding to each of the first target data; wherein each of the marking data is data obtained by marking the corresponding first target data; The adjustment module is used to train the basic model through a plurality of the labeled data to obtain a data cleaning model based on the basic model; wherein the data cleaning model is used to perform data cleaning for uncleaned data.

10. A device for using a data cleaning model, characterized in that: include: A second acquisition module, used for acquiring second uncleaned data; A cleaning module is used to input the second uncleaned data into a preset data cleaning model to obtain the second uncleaned data after data cleaning output by the data cleaning model; wherein the data cleaning model is a model trained by the method described in any one of claims 1 to 7.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; When the processor executes the computer-executable instructions stored in the memory, it is used to implement the training method of the data cleaning model as described in any one of claims 1 to 7, or to implement the method for using the data cleaning model as described in claim 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the training method of the data cleaning model as described in any one of claims 1 to 7, or to implement the method of using the data cleaning model as described in claim 8.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program is used to implement the training method of the data cleaning model according to any one of claims 1 to 7, or the method for using the data cleaning model according to claim 8.

Images