Data processing method and device

Abstract

The invention provides a data processing method and device, and the method comprises the steps: obtaining target data, the target data comprises at least one missing class and at least one normal class, a plurality of pieces of data in the at least one missing class are in one-to-one correspondence with a plurality of pieces of data in the at least one normal class, and the at least one missing class has data missing; performing data filling on the target data by adopting a plurality of data filling models to obtain complete data corresponding to each data filling model; training the target filling model by adopting each piece of complete data to obtain a to-be-tested filling model corresponding to each piece of complete data; taking data in at least one missing class and corresponding data in at least one normal class in the target data as a test standard and input of each to-be-tested filling model, and obtaining a prediction accuracy parameter corresponding to each to-be-tested filling model; and determining final complete data based on the to-be-tested filling model with the highest prediction accuracy parameter. And the filling accuracy of the missing data in the target data is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method, a data processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the advent of the artificial intelligence era, the development of many businesses cannot be separated from high-quality big data. High-quality big data can be used to train highly accurate models, and then use highly accurate models to achieve accurate business predictions. In addition, high-quality big data can be used to achieve accurate data mining and analysis, thereby maximizing the potential value of data.

[0003] Due to the long time of data generation, immature data collection technology, improper data transmission and storage, etc., there will be missing values ​​or erroneous values ​​in the data, which will reduce the data quality. In order to improve the data quality, for the missing values ​​in the data, it is usually necessary to average the normal values ​​of the category where the missing values ​​are located to obtain the average value, and then fill the missing values ​​in the data with the average value.

[0004] However, in some cases, there is no direct correlation between the actual value of the missing value and the normal value of the same category. For example: in the user-age data table, user a-20 years old, user b-30 years old, user c-null. The age value of user c is missing. There is no direct correlation between the ages of each user. If the average of the ages of user a and user b is 25 years old, and the actual age of user c is 40 years old, it will lead to an error in the age filling of user c. It can be seen that using the average method to fill in missing data will cause filling errors, thereby reducing the accuracy of data filling. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a data processing method, a data processing device, an electronic device, and a computer-readable storage medium to improve the accuracy of data filling.

[0006] In order to solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0007] The first aspect of the present application provides a data processing method, which includes: acquiring target data, the target data including at least one missing class and at least one normal class, multiple data in the at least one missing class corresponding one-to-one to multiple data in the at least one normal class, and at least one missing class has missing data; using a plurality of data filling models to fill the target data to obtain complete data corresponding to each data filling model; using each complete data to train the target filling model to obtain a filling model to be tested corresponding to each complete data; using the data in at least one missing class and the corresponding data in at least one normal class in the target data as the test standard and input for each filling model to be tested to obtain the prediction accuracy parameter corresponding to each filling model to be tested; and determining the final complete data based on the filling model to be tested with the highest prediction accuracy parameter.

[0008] Compared with the prior art, the data processing method provided in the first aspect of the present application fills the target data with different data filling models, and then uses each filled complete data to train the target filling model to obtain multiple filling models to be tested. Each filling model to be tested has fully learned the relevant knowledge in the complete data after the corresponding data filling model is filled, and can accurately and comprehensively characterize the corresponding filled complete data. The normal class data in the target data is input into each filling model to be tested, and each filling model to be tested can output the filling data. By comparing the filled data output by each filling model to be tested with the corresponding known data in the missing class in the target data, it is possible to accurately lock the filling model to be tested with the highest prediction accuracy parameter, thereby selecting the corresponding complete data and improving the accuracy of filling missing data in the target data.

[0009] In some changed implementations of the first aspect of the present application, each complete data is used to train the target filling model to obtain the filling model to be tested corresponding to each complete data, including: using each complete data to train the network architecture of the target filling model to obtain an intermediate model; using the missing class data and the corresponding data of the normal class in each complete data as output and input, respectively training the intermediate model to obtain the filling model to be tested corresponding to each complete data.

[0010] First, all complete data are used to train the network architecture of the target filling model, so that the network architecture of the target filling model is more suitable for this data filling, and then the missing data and normal data in each complete data are used to train the trained target filling model again, so that the retrained target filling model can more accurately characterize the rules of each complete data, thereby improving the accuracy of the final complete data selection.

[0011] In some modified implementations of the first aspect of the present application, each complete data is used to train the network architecture of the target filling model, including: averaging the filling data in each complete data to obtain average data; filling the average data into the target data to obtain training data; and using the training data to train the network architecture of the target filling model.

[0012] The data filled with different filling methods are averaged, and then the averaged data is used to train the network architecture of the target filling model. This can not only make the network architecture of the target filling model applicable to all filling methods, but also reduce the amount of data for network architecture training, thereby improving the training efficiency of the target filling model.

[0013] In some modified implementations of the first aspect of the present application, the target data includes multiple normal classes; before using each complete data to train the target filling model, the method also includes: performing data feature association analysis on the multiple normal classes and at least one missing class respectively, to obtain an association index between each normal class and at least one missing class; determining the normal class whose association index reaches a preset threshold as an associated class of at least one missing class, so as to use the associated class in each complete data and the data in at least one missing class to train the target filling model respectively.

[0014] When the target data includes multiple normal classes, selecting one of the normal classes that is most relevant to the missing class and then training the model together with the data of the missing class can reduce the amount of data for model training and improve the efficiency of model training. In addition, using the data of the normal class that is most relevant to the missing class for model training allows the trained model to more accurately characterize the corresponding filling method, thereby improving the accuracy of the correct filling data selection and thus improving the accuracy of data filling.

[0015] In some changed implementation modes of the first aspect of the present application, the target data is respectively filled with multiple data filling models to obtain complete data corresponding to each data filling model, including: for each data filling model, the data in the missing class of the target data and the corresponding data in the normal class are used as the output and input of the data filling model to train the data filling model; the corresponding data of the missing data in the normal class of the target data is input into the trained data filling model to obtain the filled data; the filled data is filled into the target data to obtain the complete data corresponding to the corresponding data filling model.

[0016] When filling the target data, the known data in the target data is first used to train the data filling model, so that the trained model is more consistent with the rules in the target data. Then, the trained model is used to predict the unknown data, which can improve the accuracy of filling the target data.

[0017] In some changed implementation modes of the first aspect of the present application, multiple data filling models are decision tree type machine learning models, and the decision tree type machine learning models include a search space and a search strategy, and the search space of each data filling model is different, but the search strategy is the same; the corresponding data of the missing data in the target data in the normal class is input into the trained data filling model to obtain the filling data, including: the corresponding data of the missing data in the target data in the normal class is processed using the search space and search strategy in the trained data filling model to obtain the filling data.

[0018] Different data filling models use different search spaces and the same search strategies under the decision tree machine learning model to achieve simple and effective generation of various data filling models, thereby improving the efficiency and accuracy of data filling.

[0019] In some modified implementations of the first aspect of the present application, the final complete data is determined based on the filling model to be tested with the highest prediction accuracy parameter, including: determining the complete data corresponding to the filling model to be tested with the highest prediction accuracy parameter as the final complete data.

[0020] Directly determining the complete data corresponding to the tested filling model with the highest prediction accuracy parameter as the final complete data can shorten the process of model-based reprocessing, improve the efficiency of final complete data determination, and thus improve data filling efficiency.

[0021] In some modified implementations of the first aspect of the present application, before the target data is filled with multiple data filling models respectively, the method also includes: performing at least one of the following: deleting abnormal data in the target data; converting characters in the target data into numerical values ​​with preset meanings; deleting data of a specified class in the target data, where the specified class is a category in which each data is different.

[0022] Before filling the target data, the abnormal data in the target data is deleted, the target data is standardized, and the data of no practical use in the target data is deleted to improve the validity of the target data and thus improve the accuracy of data filling.

[0023] According to a second aspect of the present application, there is provided a data processing device, which includes: an acquisition module for acquiring target data, wherein the target data includes at least one missing class and at least one normal class, wherein multiple data in the at least one missing class correspond one-to-one to multiple data in the at least one normal class, and there is data missing in at least one missing class; a filling module for filling the target data using multiple data filling models respectively to obtain complete data corresponding to each data filling model; a training module for training the target filling model using each complete data to obtain a filling model to be tested corresponding to each complete data; a testing module for using the data in at least one missing class in the target data and the corresponding data in at least one normal class as the test standard and input for each filling model to be tested to obtain the prediction accuracy parameter corresponding to each filling model to be tested; and a determination module for determining the final complete data based on the filling model to be tested with the highest prediction accuracy parameter.

[0024] A third aspect of the present application provides an electronic device, which includes a processor, a memory and a bus. The processor and the memory communicate with each other through the bus, and the processor is used to call program instructions in the memory to execute the method in the first aspect.

[0025] A fourth aspect of the present application provides a computer-readable storage medium, which includes a stored program, and when the program is executed, controls a device where the computer-readable storage medium is located to execute the method in the first aspect.

[0026] A fifth aspect of the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a device, the method in the first aspect is implemented.

[0027] The data processing device provided in the second aspect, the electronic device provided in the third aspect, the computer-readable storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect of the present application have the same or similar beneficial effects as the data processing method provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0029] Figure 1 This is a schematic diagram of the scenario architecture of the data processing method in the embodiment of the present application;

[0030] Figure 2 The data processing method in the embodiment of the present application is shown in FIG. Figure 1 ;

[0031] Figure 3 This is a schematic diagram of the overall architecture of the data processing method in the embodiment of the present application;

[0032] Figure 4 The data processing method in the embodiment of the present application is shown in FIG. Figure 2 ;

[0033] Figure 5 The data processing method in the embodiment of the present application is shown in FIG. Figure 3 ;

[0034] Figure 6 Schematic diagram of the architecture and training process of various data filling models in the embodiments of the present application;

[0035] Figure 7 This is a schematic diagram of the data feature association analysis results between various types of target data in the embodiment of the present application;

[0036] Figure 8 A schematic diagram of the process of model training, model testing, and filling data selection in an embodiment of the present application;

[0037] Fig. 9 The structure of the data processing device in the embodiment of the present application is shown in FIG. Figure 1 ;

[0038] Fig.10 The structure of the data processing device in the embodiment of the present application is shown in FIG. Figure 2 ;

[0039] Fig.11 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0041] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0042] Currently, for missing data in the target data, the average value of the context data of the missing part is usually used to fill in the missing data. However, sometimes the context data has no direct connection with the missing data, and the filled average value is very different from the actual missing data, which reduces the accuracy of data filling.

[0043] In view of this, the embodiments of the present application provide a data processing method, a data processing device, an electronic device, a computer-readable storage medium, and a computer program product. First, the target data is filled in different filling methods to obtain multiple complete data. Then, a model is trained using each complete data, and different models can characterize different filling methods. Finally, the known data of the missing class in the target data and the data corresponding to the known data of the missing class in the normal class are used to verify the different characterization models to test the accuracy of each filling method, and then the final complete data is determined according to the best filling method tested, thereby achieving accurate filling of the data.

[0044] First, the application scenario of the data processing method provided in the embodiment of the present application is described.

[0045] Figure 1 This is a schematic diagram of the scenario architecture of the data processing method in the embodiment of the present application, see Figure 1 As shown, the architecture may include: a database 11 and a data processing device 12 .

[0046] The database 11 is used to store various data.

[0047] When some data (i.e., target data) in the database 11 is needed for model training, potential value analysis, etc., but it is found that there is data missing in the target data, the data processing device 12 obtains the target data from the database 11. The data processing device 12 fills the target data in different ways, trains the same model with different filled data, and performs a prediction accuracy test on the trained model, etc., to fill the missing data in the target data, and then transmits the filled data back to the database 11.

[0048] In practical applications, the data processing device may be a computer program product, a computer-readable storage medium, or a computer hardware device (ie, an electronic device). The specific form of the data processing device is not limited here.

[0049] It should be noted here that the target data and its processing process involved in the embodiments of the present application have obtained relevant authorization in advance and are legal and compliant.

[0050] Next, the data processing method provided in the embodiment of the present application is described in detail.

[0051] Figure 2 The data processing method in the embodiment of the present application is shown in FIG. Figure 1 , see Figure 2 As shown, the method may include:

[0052] S21: Acquire target data.

[0053] The target data here may refer to a data set in which there are missing data.

[0054] The target data includes at least one missing class and at least one normal class. Multiple data in the at least one missing class correspond to multiple data in the at least one normal class one by one. There is data missing in the at least one missing class.

[0055] The missing class and normal class here both refer to data categories in the target data, such as age class, gender class, home address class, etc. Each category is composed of multiple data, for example, the age class includes 20 years old, 18 years old, 40 years old, etc. A category that does not have missing data is a normal class, and a category that has missing data is a missing class. The method in the embodiment of the present application is used to accurately fill in the missing data in the missing class.

[0056] In the target data, the data in the missing class and the normal class are all corresponding. For example: the target data includes user identification class, age class, gender class and home address class. The user identification class includes user 1, user 2 and user 3. The age class includes 20 years old, empty and 40 years old. The gender class includes male, female and male. The home address includes xx community, xx street, xx city, xx province, xx street, yy community, xx street, xx city, xx province, and ww community, zz street, xx city, xx province. It can be seen that user 1 is 20 years old, male and his home address is xx community, xx street, xx city, xx province. User 2's age is unknown, his gender is female and his home address is yy community, xx street, xx city, xx province. User 3 is 40 years old, male and his home address is ww community, zz street, xx city, xx province.

[0057] Table 1 below is an example of target data.

[0058] Table 1 Target data

[0059] User ID Age (years) gender Home address User 1 20 male xx District, xx Street, xx City, xx Province User 2 female yy Community, xx Street, xx City, xx Province User 3 40 male xx Province xx City zz Street ww Community

[0060] In Table 1, user ID, gender, and home address are three normal classes, and age is one missing class. In the missing class, the age data of user 2 is missing.

[0061] Generally speaking, the target data is stored in a database. The method in the embodiment of the present application can use select*fromtable or other known equivalent methods to establish a connection with the database, read the target data, and then convert it into structured data that is easy to process by the method in the embodiment of the present application.

[0062] S22: The target data is filled with data using a plurality of data filling models to obtain complete data corresponding to each data filling model.

[0063] There are missing data in the target data. In addition to using the average value of the context of the missing data for data filling, various data filling models can also be used for data filling.

[0064] The various data filling models used here can be various currently disclosed algorithms for data filling, including but not limited to LightGBM, XGboost, Catboost, etc. The specific algorithms and numbers of the various data filling models are not limited here.

[0065] For each data filling model, the target data is filled once. For example, data filling model a is used to fill the missing data in the target data, and the corresponding complete data 1 is obtained. Data filling model b is used to fill the missing data in the target data, and the corresponding complete data 2 is obtained. Data filling model c is used to fill the missing data in the target data, and the corresponding complete data 3 is obtained. The number of complete data that can be obtained here depends on the number of data filling models used.

[0066] Different data filling models have different accuracy for different data to be filled. Since some data in the target data is missing, it is impossible to verify the accuracy of each data filling model by the accuracy of the filled data. Therefore, the data filled by each data filling model can only be represented in a unified way, and then the known data in the target data can be used to verify the corresponding representation content of each data filling model.

[0067] S23: Use each complete data to train the target filling model to obtain the filling model to be tested corresponding to each complete data.

[0068] The target filling model here is the specific object used when the data after filling each data filling model is characterized in a unified manner. The target filling model can be one of the multiple data filling models mentioned above, or a data filling model after the multiple data filling models are fused together, or a data filling model other than the multiple data filling models mentioned above. The specific content of the target filling model is not limited here.

[0069] Since each complete data contains filling data generated by a corresponding data filling model, the data filled in each complete data is different. Each complete data is used to train the target filling model, and each model obtained after training is also different. Each model can fully learn and characterize the corresponding filling method of the previously generated filling data. Therefore, whether the prediction of the data by the filling model to be tested corresponding to each complete data is accurate can characterize whether the corresponding filled data before is accurate.

[0070] In the specific training, for each complete data, each data of the normal class in the complete data is used as input, and the corresponding data of the missing class data in the complete data is used as output, and the target filling model is trained to obtain the trained filling model to be tested.

[0071] For example, the target data includes a normal class and a missing class, the normal class includes data 1 and data 2, and the missing class only includes data a corresponding to data 1. Complete data 1 includes data b1 corresponding to the target data and data 2, and complete data 2 includes data b2 corresponding to the target data and data 2. In other words, data b1 and data b2 are different filling data generated by two different data filling models. In the training of the target filling model, for complete data 1, data 1 and data a are used as the input and output of the first model training, and data 2 and data b1 are used as the input and output of the second model training. After all the training is completed, the filling model 1 to be tested is obtained. For complete data 2, data 1 and data a are used as the input and output of the first model training, and data 2 and data b2 are used as the input and output of the second model training. After all the training is completed, the filling model 2 to be tested is obtained. As many filling models to be tested as there are complete data.

[0072] Since each filling model to be tested represents the generation method of the corresponding data filling model when generating filling data, when the actual missing data cannot be known, the known data in the normal class and the missing class in the target data are used to test the accuracy of each filling model to generate missing class data. This can determine which filling model to be tested is more accurate, thereby obtaining the most accurate filling data.

[0073] S24: Using data in at least one missing class and corresponding data in at least one normal class in the target data as test criteria and input for each filling model to be tested, and obtaining prediction accuracy parameters corresponding to each filling model to be tested.

[0074] The data in the missing class of the target data is generally not completely missing, but partially missing. For example: a class of the target data should originally include data 1-data 9, but data 3 is missing due to some factors. Therefore, this class is the missing class of the target data. The missing class lacks data 3, while data 1-2 and 4-9 still exist, which are all data in the missing class. Therefore, the data in the missing class refers to the data that currently exists in the missing class and is not missing.

[0075] In the target data, multiple data under each category exist in correspondence. For example, the target data category a includes data a1 and data a2, and the target data category b includes data b1 and data b2. Here, data a1 corresponds to data b1, and data a2 corresponds to data b2. Since there are missing data in the target data, for example, data a1 is missing. Then, the known data in the missing class and the normal class for testing the prediction accuracy of each filling model to be tested in step S23 are the data in the missing class and its corresponding data in the normal class, for example, the known data of the missing class selects data a2, and the known data of the normal class selects b2.

[0076] During the specific test, for each filling model to be tested, the corresponding data of the normal class in the target data is input into the filling model to be tested, and the filling model to be tested outputs a filling data based on the previous training. Then, the output filling data is compared with the missing class data to obtain the accuracy parameter predicted by the filling model to be tested. The similarity between the output filling data and the missing class data is the accuracy parameter predicted by the filling model to be tested.

[0077] After obtaining the prediction accuracy parameter corresponding to each filling model to be tested, the higher the prediction accuracy parameter, the more accurate the prediction of the corresponding filling model to be tested is, so that accurate filling of the target data can be achieved based on the most accurate filling model to be tested.

[0078] S25: Determine the final complete data based on the filling model to be tested with the highest prediction accuracy parameter.

[0079] After determining the filling model with the highest prediction accuracy parameter, it means that the filling model has the highest accuracy in predicting missing target data, and thus the missing data in the target data can be accurately filled based on the model.

[0080] Specifically, since the filling model to be tested is trained based on the complete data after filling, the filling data in the complete data used in training the filling model to be tested can be directly used as the final filling data, and then the final filling data can be filled into the target data to obtain the final complete data.

[0081] Alternatively, the corresponding data of the missing data in the target data in the normal class can be input again into the to-be-tested filling model with the highest prediction accuracy parameter. The output of the model is the final filling data, and then the final filling data is filled into the target data to obtain the final complete data. The specific implementation method of determining the final complete data based on the to-be-tested filling model with the highest prediction accuracy parameter can be selected according to actual needs and is not limited here.

[0082] It can be seen from the above content that the data processing method provided in the embodiment of the present application uses different data filling models to fill the target data, and then uses each filled complete data to train the target filling model to obtain multiple filling models to be tested. Each filling model to be tested has fully learned the relevant knowledge in the complete data after the corresponding data filling model is filled, and can accurately and comprehensively characterize the corresponding filled complete data. The normal class data in the target data is input into each filling model to be tested, and each filling model to be tested can output the filling data. By comparing the filled data output by each filling model to be tested with the corresponding known data in the missing class in the target data, it is possible to accurately lock the filling model to be tested with the highest prediction accuracy parameter, thereby selecting the corresponding complete data and improving the accuracy of filling missing data in the target data.

[0083] Furthermore, as a Figure 2 As a refinement and extension of the method shown, an embodiment of the present application also provides a data processing method.

[0084] Figure 3 This is a schematic diagram of the overall architecture of the data processing method in the embodiment of the present application, see Figure 3 As shown, the architecture may include: a data governance automation system based on machine learning, which may include: a database communication module, a data set management module, a data governance automated machine learning (Automated Machine Learning, AutoML) module and a platform management module.

[0085] The database communication module includes a database reading module and a database writing module. The database reading module is used to read data from the database. The database writing module is used to write data back to the database. When reading and writing data, various reading and writing methods suitable for the database can be used, such as Structured Query Language (SQL).

[0086] The dataset management module includes a dataset construction module, a dataset archiving module, and a data processing module. The data processing module is used to delete abnormal data and enumeration data, and perform data standardization. The dataset construction module is used to convert data into data that can be used for machine learning model training. The dataset archiving module is used to divide the data into training sets, validation sets, and test sets.

[0087] The data governance AutoML module includes a data completion model center, a data feature association analysis center, and a data verification model center. The data completion model center is used to use different data filling models to fill in data and obtain different complete data. The data feature association analysis center is used to calculate the data feature association index between each class in the data and select the associated class with the highest association index for the missing class in the data. The data verification model center is used to characterize the missing class and associated class in each complete data using the target filling model. For different characterized models, the original data in the complete data is used for testing, so as to select the final filling data based on the test results.

[0088] The platform management module, including task management and task scheduling, is used to provide creation and scheduling management of various tasks in the entire platform.

[0089] Figure 4 The data processing method in the embodiment of the present application is shown in FIG. Figure 2 , see Figure 4 As shown, in Figure 3 During the operation of each module of the system shown, the target data in the database is read out by the database reading module, and the database reading module sends the target data to the data processing module. The data processing module deletes abnormal data and enumeration data in the target data, and standardizes the target data, and then sends the processed data to the data set construction module. The data set construction module further processes the data processed by the data processing module into a data set that can be used for model training, and sends the data set to the data set archiving module. The data set archiving module splits the data set into a training set, a test set, and a validation set, and sends them to the data completion model center. The data completion model center optimizes multiple data filling models based on the training set and the validation set, and inputs the test set into each optimized data filling model, and obtains multiple complete data accordingly.

[0090] At the same time, the dataset construction module also sends the dataset to the data feature association analysis center. The data feature association analysis center performs association analysis on the data features between the various classes in the dataset, generates association indexes between the various classes, and selects the class with the highest association index with the missing class from among the multiple normal classes in the dataset, i.e., the association class.

[0091] Next, the multiple complete data output by the data completion model center and the associated classes output by the data feature association analysis center are input into the data verification model center together. For each complete data, the data verification model center uses the complete data to train the target filling model to obtain the filling model to be tested corresponding to each complete data, and then uses the accuracy of the prediction of the data in the normal class corresponding to the known data in the missing class by the filling model to be tested to select the complete data corresponding to the filling model to be tested with the highest prediction accuracy, so as to determine the filling data in the selected complete data as the final filling data.

[0092] Finally, the database write-back module writes the final filled data back to the target data in the database.

[0093] Figure 5 The data processing method in the embodiment of the present application is shown in FIG. Figure 3 , see Figure 5 As shown, the method may include:

[0094] S51: Acquire target data.

[0095] The target data here is the data set with missing data read from the database.

[0096] S52: Standardize the target data.

[0097] The target data may contain some erroneous data or data that is not suitable for subsequent machine learning model processing. The standardization process here is to delete some unreasonable data in the target data and convert the target data into data that can be processed by the subsequent machine learning model.

[0098] Specifically, S52 may include:

[0099] Step A1: Delete abnormal data in the target data.

[0100] Abnormal data refers to data that is inconsistent with common sense or has a wrong format.

[0101] Deleting abnormal data from the target data can improve the accuracy of the target data, thereby improving the accuracy of subsequent machine learning model training and ultimately improving the accuracy of data filling.

[0102] Table 2 below is an example of deleting abnormal data in the target data.

[0103] Table 2 Comparison of abnormal data before and after deletion in target data

[0104] Age category (before deletion) Age category (after deletion) 20 20 19 19 16 16 23 23 1044 NAN

[0105] In Table 2, for the age category before deletion, 20, 19, etc. are all in the normal age range, while 1044 obviously does not meet the normal age range, so 1044 is deleted as abnormal data. The corresponding position in the age category after deletion is replaced by missing data.

[0106] Step A2: Convert characters in the target data into numerical values ​​with preset meanings.

[0107] The target data sometimes contains some Chinese characters. However, the machine learning model cannot learn these Chinese characters very well. Therefore, it is necessary to convert these Chinese characters into numerical values ​​that can be used by the machine learning model for more accurate learning. In other words, different Chinese characters are converted into different numerical values. The Chinese character corresponding to the numerical value is its corresponding preset meaning.

[0108] Table 3 below shows examples of character conversion values ​​in target data.

[0109] Table 3 Comparison of character values ​​before and after conversion in target data

[0110] Gender (before conversion) Gender (after conversion) male 0 male 0 female 1 male 0 female 1

[0111] In Table 3, for males, they are all converted to 0. For females, they are all converted to 1. Subsequent machine learning models can learn better based on values ​​such as 0 and 1. Regardless of whether the model finally outputs 0 or 1, it can also characterize male and female genders.

[0112] Step A3: Delete the data of the specified class in the target data, where the specified class is a category in which all data are different.

[0113] There may be a category of data in the target data. Each data in this category is different. This is of little significance to the subsequent learning and training of machine learning models, and will increase the amount of data for model training. This category is the designated category. The designated category data in the target data can be deleted to improve the efficiency and accuracy of subsequent machine learning model training.

[0114] Table 4 below is an example of deleting data of a specified type from target data.

[0115] Table 4 Comparison before and after deleting the specified class data in the target data

[0116]

[0117] In Table 4, the address of each data is different. For the subsequent machine learning model, the possibility of learning relevant knowledge is low. Therefore, the household registration data in the target data is deleted.

[0118] Of course, if the granularity of household registration data is located in a larger area such as the city or province level, and these partitions have a certain relevance for filling in missing data, the household registration data can be retained. Data of similar categories to household registration data can also be handled in this way.

[0119] It should be noted that at least one of the above steps A1, A2, and A3 can be selected for execution, or all of them can be executed. The execution order of steps A1, A2, and A3 is not limited here.

[0120] After the target data is preprocessed, the data becomes more accurate and standardized. Then the data filling model can be used to fill in the missing parts of the target data.

[0121] Although the data filling model can fill in the missing items in the data, the data filling model is essentially a machine learning model, and different data have different contents and different missing parts. Directly using the data filling model to fill in the missing items in the target data will reduce the accuracy of the filling. Therefore, the corresponding data of the missing class and the normal class in the target data can be used to train the model first, and the normal class data corresponding to the missing data can be input into the trained model to obtain more accurate filling data.

[0122] S53: For each data filling model, the data in the missing class of the target data and the corresponding data in the normal class are used as the output and input of the data filling model to train the data filling model.

[0123] S54: Input the corresponding data of the missing data in the target data in the normal class into the trained data filling model to obtain the filling data.

[0124] S55: Fill the target data with the filling data to obtain the complete data corresponding to the corresponding data filling model.

[0125] In model training, the data set needs to be divided into training set, validation set and test set. The proportion of training set, validation set and test set in the data set can be 70%, 20%, 10% respectively, or other proportions determined according to actual conditions. Since there is missing data in the missing class of the target data, and the data filling model is required to fill the data, therefore, in the training of the data filling model, the missing data and its corresponding data in the normal class are no longer involved in this model training. In other words, the missing data and its corresponding data in the normal class are used as the test set for data filling. And the data other than the target data are used as the training set and validation set for model training.

[0126] In the training set and validation set, the data in the missing class corresponds to the data in the normal class. The data in the normal class is used as the model input, and the corresponding data in the missing class is used as the model output, so that the model can learn the target data and optimize the various parameters in the model. After training, the model has learned the association knowledge between the normal class and the missing class in the target data.

[0127] Although different data filling models are trained with the same training set and validation set in the target data, the specific algorithms and architectures used by different data filling models are different, so there are still differences between the models after training.

[0128] After the model training is completed, the normal class data in the test set is input, and the output of the model is to fill in the missing data in the missing class.

[0129] For each data filling model, the above training method needs to be used for model training, and then the trained model is used to predict missing data. There are as many data filling models as there are trained models. Then, the corresponding data of the missing data in the normal class is input into each trained model, and the number of filled data can be obtained. These filled data are filled into the missing positions of the target data, and the number of complete data can be obtained.

[0130] Since there must be some regularity in the original feature distribution of the target data, there will also be some similarities between different classes of target data. In order to improve the accuracy of model filling, various data filling models can adopt decision tree-type machine learning models.

[0131] In the decision tree machine learning model, you can configure the search space and search strategy. The search space of each data filling model can be the same or different. The content and number of search strategies of each data filling model can be the same or different. As long as the search space and search strategy of each data filling model are not completely consistent.

[0132] Accordingly, the above step S54 may include: using the search space and search strategy in the trained data filling model to process the corresponding data of the missing data in the target data in the normal class to obtain the filling data.

[0133] In practical applications, in order to improve the construction efficiency of each data filling model, the search space of each data filling model can be different, but the search strategy can be the same.

[0134] Figure 6 The architecture of various data filling models and the schematic diagram of the training process in the embodiments of the present application are shown in FIG. Figure 6As shown in the figure, taking the construction of three data filling models, LightGBM, XGboost and Catboost, as examples, the search spaces of the LightGBM model, XGboost model and Catboost model correspond to the LightGBM search space, XGboost search space and Catboost search space respectively. The search strategies of the LightGBM model, XGboost model and Catboost model all correspond to the evolutionary algorithm search strategy, early stopping search strategy, etc.

[0135] The normal class data of the training set and the validation set in the target data are respectively input into the LightGBM search space, XGboost search space, and Catboost search space, and the outputs of the LightGBM search space, XGboost search space, and Catboost search space are respectively subjected to data search using evolutionary algorithm search strategy, early stopping search strategy, etc. to obtain search results, and then the search results are compared with the corresponding data of the missing class, so as to adjust the parameters in the LightGBM search space, XGboost search space, Catboost search space, evolutionary algorithm search strategy, early stopping search strategy, etc. according to the comparison results, and the LightGBM optimal model, XGboost optimal model, and Catboost optimal model are obtained.

[0136] The normal class data in the target data test set are input into the LightGBM optimal model, the XGboost optimal model, and the Catboost optimal model respectively, and three types of filling data for the missing data in the target data are obtained, that is, multiple complete data after the target data is filled in different ways are obtained.

[0137] For the target data, several filling methods can be used to obtain more or less complete data. Among multiple complete data, the filling data at the same position can be averaged to obtain another complete data.

[0138] Table 5 below is an example when the target data is not filled.

[0139] Table 5 Unfilled target data

[0140] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 5 23 0.4 1 334 26 0.5 1 412

[0141] In Table 5, features 1 and 5 are missing classes, and features 2, 3, and 4 are normal classes.

[0142] Tables 6, 7, 8, and 9 below are examples after the target data is filled.

[0143] Table 6 Target data after filling

[0144] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 LGB filling data 5 23 0.4 1 334 LGB filling data 26 0.5 1 412

[0145] Table 7 Target data after filling II

[0146] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 XGB of padding data 5 23 0.4 1 334 XGB of padding data 26 0.5 1 412

[0147] Table 8 Target data after filling

[0148] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 CATB filling data 5 23 0.4 1 334 CATB filling data 26 0.5 1 412

[0149] Table 9 Target data after filling

[0150] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 Average data 5 23 0.4 1 334 Average data 26 0.5 1 412

[0151] The next step is to characterize multiple complete data through the same model training, and then select the final complete data from multiple complete data based on the accuracy of the trained model prediction.

[0152] As shown in Table 5-9 above, the target data or complete data may include multiple normal classes. Among these normal classes, there is usually one normal class that is most relevant to the missing class. Finding the normal class that is most relevant to the missing class and then using the data in the normal class and the missing class to train the model can not only reduce the amount of training data and improve the model training efficiency, but also improve the accuracy of model training, thereby improving data filling efficiency.

[0153] S56: Perform data feature association analysis on the multiple normal classes and the at least one missing class respectively to obtain an association index between each normal class and the at least one missing class.

[0154] That is to say, for the target data, each class therein can be analyzed for data feature association with other classes. In this way, the association index between the missing class in the target data and each normal class can be obtained. In the specific analysis, the association index between each of the various categories can be counted and stored in advance, and then the association index corresponding to the missing class in the target data and each normal class can be found in these stored two-by-two categories and their corresponding association indexes. Alternatively, for the data under the missing class and the data under the normal class in the target data, any known association analysis algorithm is used to calculate the association index. Known association analysis algorithms include but are not limited to Apriori algorithm, FP-growth algorithm, etc.

[0155] The association index between each class in the target data and other classes is calculated in the above manner, and the association index between each class in the target data is obtained. The association index between each class in the target data can be displayed through a chart.

[0156] Figure 7 This is a schematic diagram of the data feature association analysis results between various types of target data in the embodiment of this application, see Figure 7As shown in the figure, after performing data association analysis on features 1 to 5 in Table 5, the association index between each feature in features 1 to 5 is obtained. The association index between the features themselves is the highest, and the association index between feature 1 and feature 1, feature 2 and feature 2, feature 3 and feature 3, feature 4 and feature 4, and feature 5 and feature 5 are all 1.00. The association index between feature 1 and feature 2 is 0.54, the association index between feature 1 and feature 3 is 0.15, and so on.

[0157] S57: Determine the normal class whose association index reaches a preset threshold as the associated class of at least one missing class, so as to respectively train the target filling model using the associated class in each complete data and the data in at least one missing class.

[0158] For each missing class in the target data, its corresponding associated class needs to be found.

[0159] When determining specifically, the normal class corresponding to the highest association index can be directly selected from the association indexes of all normal classes corresponding to the missing class as the associated class of the missing class. The preset threshold at this time is a dynamic value, that is, the association index corresponding to the associated class.

[0160] In the target data of Table 5, there are two missing classes, namely feature 1 and feature 5. For feature 1, Figure 7 Find the feature with the highest correlation index, that is, feature 2 with a correlation index of 0.54 with feature 1. For feature 5, Figure 7 Find the feature with the highest correlation index, that is, feature 3, whose correlation index with feature 5 is 0.71.

[0161] When multiple associated classes need to be determined for a missing class, or when an associated class with an associated missing class needs to be found, the preset threshold can be a fixed value, such as 0.3, 0.5, 0.6, etc. The value of the preset threshold can be determined according to the actual situation. For this missing class, among the association indexes of all corresponding normal classes, the normal class corresponding to the association index greater than or equal to the preset threshold is selected as the associated class of the missing class.

[0162] It should be noted here that the acquisition of multiple complete data based on the target data in the above steps S53-S55 and the determination of the associated class based on the target data in the above steps S56-S57 can be performed simultaneously to improve the data filling efficiency.

[0163] After multiple complete data of the target data and the associated classes corresponding to the missing class are determined, the data in the missing class and the normal class in the multiple complete data can be used to fully learn the same model, that is, train it, so that each trained model can represent the target data and various filling methods respectively, so as to select the final complete data from multiple complete data through the accuracy of the prediction of the known data in the missing class by each trained model based on the data in the associated class.

[0164] S58: Use each complete data to train the network architecture of the target filling model to obtain an intermediate model.

[0165] The target filling model here is the same model mentioned in the previous paragraph. Only through the model of the same architecture can the filling methods of different data filling models be characterized under the same benchmark, ensuring that subsequent judgments are under the same benchmark, thereby improving the accuracy of the final complete data selection and thus improving the accuracy of data filling.

[0166] In practical applications, the target filling model can be any one of the multiple data filling models used previously, or a new data filling model in addition to the multiple data filling models, or a model that is a fusion of multiple data filling models.

[0167] For a target filling model that is a fusion of multiple data filling models, assuming that the multiple data filling models are LightGBM model, XGboost model and Catboost model, the LightGBM model, XGboost model and Catboost model can be fused into an Ensemble model based on the advantages of each model through Ensemble and other methods.

[0168] When preparing training data, for each complete data, divide the data in the missing class and associated class in the complete data into a test set, a validation set, and a training set. The amount of data in the training set, validation set, and test set can account for 70%, 20%, and 10% of the total amount of data in the missing class and associated class, respectively. In addition, the test set cannot contain filler data in the missing class and corresponding data in the associated class. The test set is used to test the accuracy of the model prediction after training. It does not participate in training, but participates in model testing. What is needed is the known data in the missing class and the corresponding data in the associated class. In addition, other proportions can be adjusted according to actual conditions.

[0169] The training set and validation set participate in model training. During specific training, the data in the missing class and the associated class in all the complete data can be used as output and input respectively for the network architecture training of the target filling model. The so-called network architecture training can refer to adjusting only the network architecture parameters in the model through training, while other parameters, such as optimizer parameters, loss function parameters, etc., do not participate in this training. The network architecture parameters here specifically refer to the number of leaf nodes, hidden layer size and width, etc.

[0170] In order to improve the training efficiency of the model network architecture, the amount of data used for training can be reduced. At the same time, the training data needs to use the filling data of various data filling models. Therefore, the filling data filled in the same position of various data filling models can be averaged, and then the missing class and associated class data containing the average data can be used to train the model network architecture.

[0171] Specifically, the above step S58 may include:

[0172] Step B1: average the filled data in each complete data to obtain average data.

[0173] The specific process of averaging processing can be found in the above Table 9, which will not be repeated here.

[0174] Step B2: Fill the target data with the average data to obtain training data.

[0175] Step B3: Use the training data to train the network architecture of the target filling model.

[0176] Taking the data in Table 9 above as an example, Table 9 is the target data after filling the average data, that is, the training data here. The training data includes missing classes and multiple normal classes. In order to further reduce the amount of training data, only the data of the missing class and the associated class can be used as output and input respectively to train the model network architecture. That is, feature 1 and feature 2 are used as output and input, and feature 5 and feature 3 are used as output and input to train the model network architecture. At this time, an intermediate model containing various information is obtained.

[0177] S59: Take the missing class data and the corresponding data of the normal class in each complete data as output and input, train the intermediate models respectively, and obtain the filling model to be tested corresponding to each complete data.

[0178] The filling data in different complete data are generated by using different data filling models. At this time, it is necessary to train the intermediate models respectively with different filling data to obtain various filling models to be tested that characterize the filling methods of different data filling models.

[0179] During specific training, for each complete data, the missing class and associated class in the previously divided training set and validation set can be used as output and input respectively to train the intermediate model, and obtain the filling model to be tested corresponding to each complete data that can characterize the filling method of each data filling model.

[0180] S510: Using data in at least one missing class and corresponding data in at least one normal class in the target data as test criteria and input for each filling model to be tested, and obtaining prediction accuracy parameters corresponding to each filling model to be tested.

[0181] For each filling model to be tested, the associated class data in the previously divided test set is input into the filling model to be tested. The filling model to be tested can output the corresponding data in the predicted missing class through the previous learning of the filling data predicted by the corresponding data filling model. The data in at least one missing class is actually the known data in the missing class. By comparing the data predicted by the filling model to be tested with the known data in the missing class, the accuracy of the missing data predicted by the filling model to be tested can be known, that is, the prediction accuracy parameter of the filling model to be tested is determined. The number of prediction accuracy parameters corresponding to the filling models to be tested can be obtained as many as the filling models to be tested.

[0182] S511: Determine the complete data corresponding to the filling model to be tested with the highest prediction accuracy parameter as the final complete data.

[0183] The prediction accuracy parameter is a numerical representation of whether the filling model to be tested can achieve accurate prediction. Each filling model to be tested is trained with complete data. Each complete data is obtained by various data filling models based on the missing calculations of the target data. Therefore, the complete data corresponding to the filling model to be tested with the highest prediction accuracy parameter is the most accurately calculated filled target data, which is then determined as the final complete data.

[0184] Finally, the process of model training, model testing and filling data selection in the above steps S58-S511 is clearly explained again.

[0185] Figure 8 This is a schematic diagram of the process of model training, model testing and filling data selection in the embodiment of the present application, see Figure 8As shown in the figure, the multiple data filling models are LightGBM model, XGboost model and Catboost model, and the target filling model is the Ensemble model after the fusion of LightGBM model, XGboost model and Catboost model. At this time, the target data has been filled by LightGBM model, XGboost model and Catboost model to obtain LGB complete data, XGB complete data and CATB complete data. The filled data of the same missing bits in LGB complete data, XGB complete data and CATB complete data are averaged to obtain average complete data. The data in the missing class and the associated class in the average complete data are divided into an average training set and an average validation set.

[0186] In the model training, the average training set and the average validation set are used to train the Ensemble model to obtain the optimal Ensemble model. The optimal Ensemble model here is the intermediate model in the aforementioned embodiment.

[0187] The missing and associated data in the LGB complete data are divided into LGB training set, LGB validation set and LGB test set. The XGB complete data and CATB complete data are also divided in this way to obtain the XGB training set, XGB validation set and XGB test set, and the CATB training set, CATB validation set and CATB test set. The LGB test set, XGB test set and CATB test set do not contain the filled data and their corresponding data in the associated class.

[0188] In the model test, the optimal Ensemble model was trained using the LGB training set and the LGB validation set to obtain the LGB model. The optimal Ensemble model was trained using the XGB training set and the XGB validation set to obtain the XGB model. The optimal Ensemble model was trained using the CATB training set and the CATB validation set to obtain the CATB model.

[0189] Next, in the filling data selection, the LGB test set is used to test the LGB model to obtain the prediction accuracy parameters of the LGB model. The XGB test set is used to test the XGB model to obtain the prediction accuracy parameters of the XGB model. The CATB test set is used to test the CATB model to obtain the prediction accuracy parameters of the CATB model.

[0190] Finally, the prediction accuracy parameter can be used to lock the model with the best prediction performance from the LGB model, XGB model and CATB model, so that the filling data in the complete data corresponding to the model can be used as the final filling data.

[0191] At this point, the data processing method provided in the embodiments of the present application has been fully described.

[0192] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a data processing device.

[0193] Fig. 9 The structure of the data processing device in the embodiment of the present application is shown in FIG. Figure 1 , see Fig. 9 As shown, the device may include: an acquisition module 91, a filling module 92, a training module 93, a testing module 94 and a determination module 95.

[0194] The acquisition module 91 is used to acquire target data, where the target data includes multiple data in at least one missing class and at least one normal class. The at least one missing class corresponds to multiple data in at least one normal class, and there is missing data in at least one missing class.

[0195] The filling module 92 is used to fill the target data with multiple data filling models to obtain complete data corresponding to each data filling model.

[0196] The training module 93 is used to train the target filling model using each complete data to obtain the filling model to be tested corresponding to each complete data.

[0197] The testing module 94 is used to use the data in at least one missing class and the corresponding data in at least one normal class in the target data as the testing standard and input of each filling model to be tested, and obtain the prediction accuracy parameter corresponding to each filling model to be tested.

[0198] The determination module 95 is used to determine the final complete data based on the filling model to be tested with the highest prediction accuracy parameter.

[0199] Furthermore, as a Fig. 9 As a refinement and expansion of the device shown, an embodiment of the present application also provides a data processing device.

[0200] Fig.10 The structure of the data processing device in the embodiment of the present application is shown in FIG. Figure 2 , see Fig.10 As shown, the device may include: an acquisition module 101 , a standardization module 102 , a filling module 103 , an association analysis module 104 , a training module 105 , a testing module 106 and a determination module 107 .

[0201] The acquisition module 101 is used to acquire target data, where the target data includes multiple data in at least one missing class and at least one normal class, where the at least one missing class corresponds one-to-one to multiple data in at least one normal class, and there is data missing in at least one missing class.

[0202] The standardization module 102 is used to perform at least one of the following: deleting abnormal data in the target data; converting characters in the target data into numerical values ​​with preset meanings; deleting data of a specified class in the target data, where the specified class is a category in which each data is different.

[0203] The filling module 103 includes: a model training unit 1031 , a prediction unit 1032 and a filling unit 1033 .

[0204] The model training unit 1031 is used to train the data filling model by taking the data in the missing class of the target data and the corresponding data in the normal class as the output and input of the data filling model for each data filling model.

[0205] The prediction unit 1032 is used to input the corresponding data of the missing data in the target data in the normal class into the trained data filling model to obtain the filling data.

[0206] When the multiple data filling models are decision tree-type machine learning models, the decision tree-type machine learning models include a search space and a search strategy, and the search space of each data filling model is different, but the search strategy is the same, the prediction unit 1032 is specifically used to process the corresponding data of the missing data in the target data in the normal class using the search space and search strategy in the trained data filling model to obtain the filling data.

[0207] The filling unit 1033 is used to fill the filling data into the target data to obtain the complete data corresponding to the corresponding data filling model.

[0208] The association analysis module 104 is used to perform data feature association analysis on multiple normal classes and at least one missing class respectively, to obtain an association index between each normal class and at least one missing class; a normal class whose association index reaches a preset threshold is determined as an associated class of at least one missing class, so as to use the associated class in each complete data and the data in at least one missing class to train the target filling model respectively.

[0209] The training module 105 includes: an architecture training unit 1051 and a model generation unit 1052 .

[0210] The architecture training unit 1051 is used to train the network architecture of the target filling model using each complete data to obtain an intermediate model.

[0211] The architecture training unit 1051 is specifically used to average the filling data in each complete data to obtain average data; fill the average data into the target data to obtain training data; and use the training data to train the network architecture of the target filling model.

[0212] The model generation unit 1052 is used to take the missing class data and the corresponding data of the normal class in each complete data as output and input, respectively train the intermediate model, and obtain the to-be-tested filling model corresponding to each complete data.

[0213] The testing module 106 is used to use the data in at least one missing class and the corresponding data in at least one normal class in the target data as the testing standard and input of each filling model to be tested, and obtain the prediction accuracy parameter corresponding to each filling model to be tested.

[0214] The determination module 107 is used to determine the complete data corresponding to the filling model to be tested with the highest prediction accuracy parameter as the final complete data.

[0215] It should be noted here that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0216] Based on the same inventive concept, an embodiment of the present application also provides an electronic device.

[0217] Fig.11 This is a schematic diagram of the structure of the electronic device in the embodiment of the present application, see Fig.11 As shown, the electronic device may include: a processor 111, a memory 112, and a bus 113. The processor 111 and the memory 112 communicate with each other through the bus 113. The processor 111 is used to call program instructions in the memory 112 to execute the methods in one or more of the above embodiments.

[0218] It should be noted that the description of the above electronic device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the electronic device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0219] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, which may include: a stored program that controls the device where the storage medium is located to execute the method in one or more of the above embodiments when the program is running.

[0220] It should be noted here that the description of the above computer-readable storage medium embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the computer-readable storage medium embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0221] Based on the same inventive concept, an embodiment of the present application further provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by the device where they are located, the methods in one or more of the above embodiments are implemented.

[0222] It should be noted here that the description of the above computer program product embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the computer program product embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0223] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire target data, wherein the target data includes at least one missing class and at least one normal class, a plurality of data in the at least one missing class corresponds one-to-one to a plurality of data in the at least one normal class, and the at least one missing class has missing data; The target data is respectively filled with a plurality of data filling models to obtain complete data corresponding to each data filling model; Each complete data is used to train the target filling model to obtain the filling model to be tested corresponding to each complete data; Using the data in the at least one missing class and the corresponding data in the at least one normal class in the target data as the test criteria and input of each filling model to be tested, and obtaining the prediction accuracy parameter corresponding to each filling model to be tested; The final complete data is determined based on the tested filling model with the highest prediction accuracy parameter.

2. The method according to claim 1, characterized in that The method of using each complete data to train the target filling model to obtain the filling model to be tested corresponding to each complete data includes: Each complete data is used to train the network architecture of the target filling model to obtain an intermediate model; The data of the missing class and the corresponding data of the normal class in each complete data are used as output and input, and the intermediate model is trained respectively to obtain the filling model to be tested corresponding to each complete data.

3. The method according to claim 2, characterized in that The network architecture of the target filling model is trained using each complete data, including: The filling data in each complete data are averaged to obtain average data; Filling the target data with the average data to obtain training data; The training data is used to train the network architecture of the target filling model.

4. The method according to any one of claims 1 to 3, characterized in that The target data includes multiple normal classes; Before using each complete data to train the target filling model, the method further includes: Performing data feature association analysis on the multiple normal classes and the at least one missing class respectively to obtain an association index between each normal class and the at least one missing class; The normal class whose association index reaches a preset threshold is determined as the associated class of the at least one missing class, so that the associated class in each complete data and the data in the at least one missing class are used to train the target filling model respectively.

5. The method according to any one of claims 1 to 3, characterized in that The target data is respectively filled with a plurality of data filling models to obtain complete data corresponding to each data filling model, including: For each data filling model, the data in the missing class of the target data and the corresponding data in the normal class are used as the output and input of the data filling model to train the data filling model; Inputting the corresponding data of the missing data in the target data in the normal class into the trained data filling model to obtain filling data; The filling data is filled into the target data to obtain complete data corresponding to the corresponding data filling model.

6. The method according to claim 5, characterized in that The multiple data filling models are decision tree-type machine learning models, and the decision tree-type machine learning models include a search space and a search strategy. The search space of each data filling model is different, but the search strategy is the same; the corresponding data of the missing data in the target data in the normal class is input into the trained data filling model to obtain the filling data, including: The corresponding data of the missing data in the target data in the normal class is processed using the search space and search strategy in the trained data filling model to obtain the filling data.

7. The method according to any one of claims 1 to 3, characterized in that The final complete data is determined based on the filling model to be tested with the highest prediction accuracy parameter, including: The complete data corresponding to the tested filling model with the highest prediction accuracy parameter is determined as the final complete data.

8. The method according to any one of claims 1 to 3, characterized in that Before the target data is filled with data using a plurality of data filling models respectively, the method further includes: Do at least one of the following: Deleting abnormal data in the target data; Converting characters in the target data into numerical values ​​with preset meanings; The data of a specified class in the target data is deleted, and the specified class is a category in which each data is different.

9. A data processing device, characterized in that: The device comprises: an acquisition module, configured to acquire target data, wherein the target data includes at least one missing class and at least one normal class, a plurality of data in the at least one missing class corresponds one-to-one to a plurality of data in the at least one normal class, and the at least one missing class has data missing; A filling module, used to fill the target data with a plurality of data filling models to obtain complete data corresponding to each data filling model; A training module, used to train the target filling model using each complete data to obtain a filling model to be tested corresponding to each complete data; A testing module, used to use the data in the at least one missing class and the corresponding data in the at least one normal class in the target data as the testing criteria and input of each filling model to be tested, and obtain the prediction accuracy parameter corresponding to each filling model to be tested; The determination module is used to determine the final complete data based on the filling model to be tested with the highest prediction accuracy parameter.

10. An electronic device, characterized in that: The electronic device comprises a processor, a memory and a bus, wherein the processor and the memory communicate with each other via the bus, and the processor is used to call program instructions in the memory to execute the method as claimed in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, and when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are executed by a device, the method according to any one of claims 1 to 8 is implemented.

Images