Data processing method and apparatus

Abstract

The application provides a data processing method and device. The method comprises: obtaining target data, the target data comprising at least one missing class and at least one normal class, a plurality of data in the at least one missing class corresponding to a plurality of data in the at least one normal class one by one, and the at least one missing class having data missing; filling the target data using a plurality of data filling models respectively to obtain complete data corresponding to each data filling model; training a target filling model using each complete data to obtain a to-be-tested filling model corresponding to each complete data; taking data in the at least one missing class and corresponding data in the at least one normal class in the target data as a test standard and input of each to-be-tested filling model to obtain a prediction accuracy parameter corresponding to each to-be-tested filling model; and determining final complete data based on a to-be-tested filling model having the highest prediction accuracy parameter. The accuracy of filling missing data in the target data is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a data processing method, a data processing device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] With the advent of the artificial intelligence era, many businesses cannot be carried out without high-quality big data. Through high-quality big data, a highly precise model can be trained, and then precise prediction of the business is realized through the high-precision model. In addition, through high-quality big data, precise data mining and analysis are realized, so as to maximize the potential value of data.

[0003] Due to the long time of data generation, the immaturity of data acquisition technology, improper data transmission and storage and other reasons, missing values or error values exist in the data, thereby reducing the data quality. In order to improve the data quality, for the missing values in the data, the normal values in the category of the missing values are usually averaged to obtain an average value, and then the average value is filled in the data as the missing value.

[0004] However, in some cases, the actual value of the missing value is not directly related to the normal value of the same category. For example, in a user-age data table, user a is 20 years old, user b is 30 years old, and user c is empty. The age value of user c is missing. There is no direct relationship between the ages of the users. If the average of the ages of user a and user b is taken, 25 years old is obtained, and the actual age of user c is 40 years old, which will cause the age filling error of user c. It can be seen that the average method for filling the missing data will cause the filling error problem, thereby reducing the accuracy of data filling. SUMMARY

[0005] The embodiments of the present application aim to provide a data processing method, a data processing device, an electronic device and a computer readable storage medium to improve the accuracy of data filling.

[0006] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0007] The first aspect of the present application provides a data processing method, which comprises: obtaining target data, the target data comprising at least one missing class and at least one normal class, a plurality of data in the at least one missing class corresponding to a plurality of data in the at least one normal class one by one, and the at least one missing class having data missing; filling the target data using a plurality of data filling models respectively to obtain complete data corresponding to each data filling model; training a target filling model using each complete data to obtain a to-be-tested filling model corresponding to each complete data; taking the data in the at least one missing class and the corresponding data in the at least one normal class in the target data as a test standard and an input of each to-be-tested filling model to obtain a prediction accuracy parameter corresponding to each to-be-tested filling model; and determining a final complete data based on the to-be-tested filling model with the highest prediction accuracy parameter.

[0008] Compared with the prior art, the data processing method provided by the first aspect of the present application fills the target data using different data filling models, and then trains a target filling model using each complete data after filling to obtain a plurality of to-be-tested filling models. Each to-be-tested filling model has fully learned the relevant knowledge in the complete data after filling by the corresponding data filling model, and can accurately and comprehensively represent the corresponding complete data after filling. The data in the normal class in the target data is input into each to-be-tested filling model, and each to-be-tested filling model can output the filled data. By comparing the filled data output by each to-be-tested filling model with the known data in the missing class in the target data, the to-be-tested filling model with the highest prediction accuracy parameter can be accurately locked, and the corresponding complete data can be selected, thereby improving the accuracy of filling the missing data in the target data.

[0009] In some modified embodiments of the first aspect of the present application, training the target filling model using each complete data to obtain a to-be-tested filling model corresponding to each complete data comprises: training the network architecture of the target filling model using each complete data to obtain an intermediate model; and training the intermediate model using the data in the missing class and the corresponding data in the normal class in each complete data as the output and the input, respectively, to obtain the to-be-tested filling model corresponding to each complete data.

[0010] The network architecture of the target filling model is first trained using all the complete data, so that the network architecture of the target filling model is more suitable for the data filling this time. Then, the trained target filling model is trained again using the missing class data and the normal class data in each complete data, so that the retrained target filling model can more accurately represent the rules of each complete data, thereby improving the accuracy of selecting the final complete data.

[0011] In some modified embodiments of the first aspect of the present application, the network architecture of the target padding model is trained using each complete data, including: performing average processing on the padding data in each complete data to obtain average data; padding the average data to the target data to obtain training data; and training the network architecture of the target padding model using the training data.

[0012] The data padded by different padding methods is subjected to average processing, and the network architecture of the target padding model is trained using the average processed data, which can make the network architecture of the target padding model applicable to all padding methods, and can reduce the amount of data for training the network architecture, thereby improving the training efficiency of the target padding model.

[0013] In some modified embodiments of the first aspect of the present application, the target data includes a plurality of normal classes; before the target padding model is trained using each complete data, the method further includes: performing data feature correlation analysis on the plurality of normal classes and at least one missing class respectively to obtain a correlation index of each normal class and at least one missing class; determining a normal class with a correlation index reaching a preset threshold as a correlation class of the at least one missing class, so as to train the target padding model using the data of the correlation class and the at least one missing class in each complete data.

[0014] In the case where the target data includes a plurality of normal classes, selecting one normal class most relevant to the missing class and then training the model together with the data of the missing class can reduce the amount of data for model training, improve the model training efficiency, and using the data of the normal class most relevant to the missing class for model training enables the trained model to more accurately represent the corresponding padding method, thereby improving the accuracy of selecting correct padding data and improving the accuracy of data padding.

[0015] In some modified embodiments of the first aspect of the present application, the target data is padded using a plurality of data padding models respectively to obtain complete data corresponding to each data padding model, including: for each data padding model, taking the data in the missing class and the corresponding data in the normal class of the target data as the output and input of the data padding model, and training the data padding model; inputting the corresponding data of the missing data in the target data in the normal class into the trained data padding model to obtain padding data; and padding the padding data to the target data to obtain complete data corresponding to the corresponding data padding model.

[0016] When padding the target data, the data padding model is first trained using the known data in the target data, so that the trained model is more in line with the rules in the target data. Then, the unknown data is predicted by the trained model, which can improve the accuracy of padding the target data.

[0017] In some modified embodiments of the first aspect of the application, the plurality of data filling models are decision tree type machine learning models, the decision tree type machine learning model comprises a search space and a search strategy, the search space of each data filling model is different, and the search strategy is the same; inputting the corresponding data of the missing data in the target data in the normal class into the trained data filling model to obtain the filling data, comprising: inputting the corresponding data of the missing data in the target data in the normal class into the search space and the search strategy in the trained data filling model to obtain the filling data.

[0018] Different data filling models use different search spaces and the same search strategy under the decision tree type machine learning model, so that various data filling models are simply and effectively generated, and the efficiency and accuracy of data filling are improved.

[0019] In some modified embodiments of the first aspect of the application, the final complete data is determined based on the data filling model with the highest prediction accuracy parameter, comprising: determining the complete data corresponding to the data filling model with the highest prediction accuracy parameter as the final complete data.

[0020] The complete data corresponding to the data filling model with the highest prediction accuracy parameter is directly determined as the final complete data, which can reduce the process of re-processing based on the model, improve the efficiency of determining the final complete data, and further improve the data filling efficiency.

[0021] In some modified embodiments of the first aspect of the application, before filling the target data with the plurality of data filling models, the method further comprises: performing at least one of the following: deleting abnormal data in the target data; converting characters in the target data into numerical values with a predetermined meaning; deleting data of a specified class in the target data, the specified class being a class in which each data is different.

[0022] Before filling the target data, the abnormal data in the target data is deleted, the target data is standardized, and the data without actual use in the target data is deleted, so as to improve the effectiveness of the target data and further improve the accuracy of data filling.

[0023] The second aspect of the present application provides a data processing device, the device comprising: an acquisition module configured to acquire target data, the target data comprising at least one missing class and at least one normal class, a plurality of data in the at least one missing class corresponding to a plurality of data in the at least one normal class one by one, and the at least one missing class having data missing; a filling module configured to fill data in the target data using a plurality of data filling models respectively to obtain complete data corresponding to each data filling model; a training module configured to train the target filling model using each complete data to obtain a to-be-tested filling model corresponding to each complete data; a testing module configured to use data in the at least one missing class and corresponding data in the at least one normal class in the target data as a testing standard and input of each to-be-tested filling model to obtain a prediction accuracy parameter corresponding to each to-be-tested filling model; and a determination module configured to determine final complete data based on a to-be-tested filling model having the highest prediction accuracy parameter.

[0024] The third aspect of the present application provides an electronic device, the electronic device comprising a processor, a memory and a bus, the processor and the memory being in communication with each other through the bus, and the processor being configured to invoke program instructions in the memory to execute the method in the first aspect.

[0025] The fourth aspect of the present application provides a computer-readable storage medium, the computer-readable storage medium comprising a stored program, the program being configured to control a device in which the computer-readable storage medium is located to execute the method in the first aspect when the program is running.

[0026] The fifth aspect of the present application provides a computer program product, the computer program product comprising a computer program or instructions, the computer program or instructions being configured to implement the method in the first aspect when executed by a device in which the computer program product is located.

[0027] The data processing device provided by the second aspect of the present application, the electronic device provided by the third aspect of the present application, the computer-readable storage medium provided by the fourth aspect of the present application and the computer program product provided by the fifth aspect of the present application have the same or similar beneficial effects as the data processing method provided by the first aspect of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0028] The above and other objects, features and advantages of the present application will become readily apparent upon a reading of the following detailed description in conjunction with the drawings, in which exemplary embodiments of the present application are shown. In the drawings, several embodiments of the present application are shown in example, but not limiting, manners, like or corresponding reference numerals referring to like or corresponding parts, wherein:

[0029] Figure 1 A scene architecture schematic diagram of the data processing method in the embodiments of the present application;

[0030] Figure 2 A flowchart of the data processing method in the embodiments of the present application Figure 1 ;

[0031] Figure 3 Fig. 1 is a schematic diagram of an overall architecture of a data processing method in an embodiment of the present application;

[0032] Figure 4 Fig. 2 is a schematic diagram of a flow of the data processing method in the embodiment of the present application; Figure 2

[0033] Figure 5 Fig. 3 is a schematic diagram of a flow of the data processing method in the embodiment of the present application; Figure 3

[0034] Figure 6 Fig. 4 is a schematic diagram of an architecture and a training process of various data filling models in the embodiment of the present application;

[0035] Figure 7 Fig. 5 is a schematic diagram of a result of data feature correlation analysis between various types in the target data in the embodiment of the present application;

[0036] Figure 8 Fig. 6 is a schematic diagram of a process of model training, model testing and filling data selection in the embodiment of the present application;

[0037] Figure 9 Fig. 7 is a schematic diagram of a structure of a data processing apparatus in the embodiment of the present application; Figure 1

[0038] Fig. 8 is a schematic diagram of a structure of the data processing apparatus in the embodiment of the present application; Figure 10 Figure 2 Fig. 9 is a schematic diagram of a structure of an electronic device in the embodiment of the present application.

[0039] DETAILED DESCRIPTION Figure 11 The exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0040] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as their common meanings by those skilled in the art to which the present application pertains.

[0041] Currently, for the missing data in the target data, the average value of the context data of the missing part is usually used for filling. However, sometimes the context data has no direct connection with the missing data, and the filled average value is quite different from the actual missing data, thereby reducing the accuracy of data filling.

[0042] Currently, for the missing data in the target data, the average value of the context data of the missing part is usually used for filling. However, sometimes the context data has no direct connection with the missing data, and the filled average value is quite different from the actual missing data, thereby reducing the accuracy of data filling. ​​​

[0043] Therefore, the embodiments of the present application provide a data processing method, a data processing apparatus, an electronic device, a computer readable storage medium and a computer program product. The target data is filled in different filling ways to obtain a plurality of complete data. Then, one model is trained using each complete data, and different models can represent different filling ways. Finally, the known data of the missing class in the target data and the data corresponding to the known data of the missing class in the normal class are used to test the accuracy of each filling way by verifying the different models representing the filling ways, and then the optimal filling way is determined according to the test to obtain the final complete data, so as to realize accurate filling of data.

[0044] Firstly, the application scenario of the data processing method provided by the embodiments of the present application is described.

[0045] Figure 1 For the scenario architecture diagram of the data processing method in the embodiments of the present application, refer to Figure 1 , the architecture can include a database 11 and a data processing apparatus 12.

[0046] The database 11 is used to store various data.

[0047] When some data (i.e. target data) in the database 11 needs to be used for model training, potential value analysis, etc., but it is found that there is data missing in the target data, the data processing apparatus 12 obtains the target data from the database 11. The data processing apparatus 12 fills the data in different ways for the target data, trains different filled data for the same model, and tests the prediction accuracy of the trained model, etc. The missing data in the target data is filled, and the filled data is returned to the database 11.

[0048] In actual application, the data processing apparatus can be a computer program product, a computer readable storage medium, or a computer hardware device (i.e. an electronic device). The specific form of the data processing apparatus is not limited here.

[0049] It should be noted that the target data and its processing process involved in the embodiments of the present application are pre-authorized and legal and compliant.

[0050] Next, the data processing method provided by the embodiments of the present application is described in detail.

[0051] Figure 2 For the flowchart of the data processing method in the embodiments of the present application, refer to Figure 1 , the method can include: Figure 2

[0052] ​S21: obtaining target data.

[0053] The target data herein can refer to a data set in which data is missing.

[0054] In the target data, at least one missing class and at least one normal class are included. The data in the at least one missing class corresponds to the data in the at least one normal class one by one. The at least one missing class has data missing.

[0055] The missing class and the normal class herein both refer to the data class in the target data, for example, the age class, the gender class, the home address class, etc. Each class is composed of multiple data, for example, the age class includes 20 years old, 18 years old, 40 years old, etc. The class in which no data is missing is the normal class, and the class in which data is missing is the missing class. The method in the embodiments of the present application is used to accurately fill in the missing data in the missing class.

[0056] In the target data, the data in the missing class and the normal class are corresponding. For example, the target data includes the user identification class, the age class, the gender class and the home address class, the user identification class includes user 1, user 2 and user 3, the age class includes 20 years old, null, 40 years old, the gender class includes male, female, male, and the home address includes xx province xx city xx street xx community, xx province xx city xx street yy community, xx province xx city zz street ww community. As can be seen, the age of user 1 is 20 years old, the gender is male, and the home address is xx province xx city xx street xx community. The age of user 2 is unknown, the gender is female, and the home address is xx province xx city xx street yy community. The age of user 3 is 40 years old, the gender is male, and the home address is xx province xx city zz street ww community.

[0057] Table 1 below is an example of target data.

[0058] Table 1 target data

[0059] User ID Age (years) Gender Home Address User 1 20 Male xx Province xx City xx Street xx Community User 2 Female xx Province xx City xx Street yy Community User 3 40 Male xx Province xx City zz Street ww Community

[0060] In Table 1, the user identification, the gender and the home address are three normal classes, and the age is one missing class. In the missing class, the age data of user 2 is missing.

[0061] Generally, the target data is stored in a database. The method in the embodiments of the present application can use select*from table or other known equivalent methods to establish a connection with the database, read out the target data, and then convert it into structured data which is convenient for processing by the method in the embodiments of the present application.

[0062] S22: Fill the target data with multiple data filling models respectively to obtain complete data corresponding to each data filling model.

[0063] There are missing data in the target data. In addition to filling the data with the average value of the context of the missing data, various data filling models can also be used to fill the data.

[0064] The multiple data filling models used here can be various algorithms for filling data currently disclosed. For example: including but not limited to LightGBM, XGboost, Catboost, etc. The specific algorithm and number of the multiple data filling models are not limited here.

[0065] For each data filling model, it is filled with the target data. For example: fill the missing data in the target data with data filling model a to obtain complete data 1. Fill the missing data in the target data with data filling model b to obtain complete data 2. Fill the missing data in the target data with data filling model c to obtain complete data 3. The more data filling models are used, the more complete data can be obtained.

[0066] Different data filling models have different accuracy for different data to be filled. Since some data in the target data is missing, the accuracy of the filled data cannot be used to test the accuracy of each data filling model, so only the data filled by each data filling model can be represented in a unified way, and then the known data in the target data can be used to check the corresponding representation of each data filling model.

[0067] S23: Train each complete data on the target filling model to obtain a to-be-tested filling model corresponding to each complete data.

[0068] The target filling model here is the specific object used when the data filled by each data filling model is represented in a unified way. The target filling model can be one of the above-mentioned multiple data filling models, or a data filling model fused from the above-mentioned multiple data filling models, or a data filling model other than the above-mentioned multiple data filling models. The specific content of the target filling model is not limited here.

[0069] Since each complete data contains the filled data generated by using a corresponding data filling model, the filled data in each complete data is different, each model obtained by training the target filling model using each complete data is also different, and each model can fully learn and represent the corresponding filling mode of the filled data generated before, therefore, whether the prediction of the data by the to-be-tested filling model corresponding to each complete data is accurate can represent whether the corresponding filled data before is accurate.

[0070] In the specific training, for each complete data, each data of the normal class in the complete data is taken as the input, and the corresponding data of the missing class data in the complete data is taken as the output, the target filling model is trained, and the trained to-be-tested filling model is obtained.

[0071] For example, the target data includes a normal class and a missing class, the normal class includes data 1 and data 2, and the missing class only includes data a corresponding to data 1. Complete data 1 includes target data and data b1 corresponding to data 2, and complete data 2 includes target data and data b2 corresponding to data 2. That is, data b1 and data b2 are different filled data generated by using two different data filling models. In the training of the target filling model, for complete data 1, data 1 and data a are taken as the input and output of the first model training, data 2 and data b1 are taken as the input and output of the second model training, and after all the training is completed, to-be-tested filling model 1 is obtained. For complete data 2, data 1 and data a are taken as the input and output of the first model training, data 2 and data b2 are taken as the input and output of the second model training, and after all the training is completed, to-be-tested filling model 2 is obtained. As many complete data as there are, as many to-be-tested filling models are obtained.

[0072] Since each to-be-tested filling model represents the generation mode of the filled data generated by the corresponding data filling model, in the case that the missing actual data cannot be known, the known data in the normal class and the missing class in the target data are used to test the accuracy of each to-be-tested filling model in generating the missing class data, so that it can be known which to-be-tested filling model is more accurate, and the most accurate filled data is obtained.

[0073] S24: taking the data in at least one missing class and the corresponding data in at least one normal class in the target data as the test standard and input of each to-be-tested filling model, obtaining the prediction accuracy parameter corresponding to each to-be-tested filling model.

[0074] The data in the missing class of the target data is generally not all missing, but partially missing. For example, the data 1-9 should originally be included under a class of the target data, but data 3 is missing due to some factors, and thus the class is a missing class of the target data, which lacks data 3, and data 1-2 and 4-9 still exist and are data in the missing class. Therefore, the data in the missing class refers to the data in the missing class that currently exists and is not missing.

[0075] In the target data, multiple data under each class correspondingly exist. For example, data a1 and data a2 are included under a class a of the target data, and data b1 and data b2 are included under a class b of the target data. Here, data a1 corresponds to data b1, and data a2 corresponds to data b2. Since the data in the target data is missing, for example, data a1 is missing. Then, the known data in the missing class and the normal class for testing the prediction accuracy of each to-be-tested filling model in step S23 is the data in the missing class and the corresponding data in the normal class, for example, the known data of the missing class is data a2, and the known data of the normal class is b2.

[0076] In the specific test, for each to-be-tested filling model, the corresponding data in the normal class of the target data is input into the to-be-tested filling model, and the to-be-tested filling model outputs a filling data based on the previous training. Then, the output filling data is compared with the data in the missing class, and the prediction accuracy parameter of the to-be-tested filling model can be obtained. The similarity between the output filling data and the data in the missing class is the prediction accuracy parameter of the to-be-tested filling model.

[0077] After obtaining the prediction accuracy parameter corresponding to each to-be-tested filling model, the higher the prediction accuracy parameter, the more accurate the prediction of the corresponding to-be-tested filling model, and thus the most accurate to-be-tested filling model can be used to realize the accurate filling of the target data.

[0078] S25: determining the final complete data based on the to-be-tested filling model with the highest prediction accuracy parameter.

[0079] After determining the to-be-tested filling model with the highest prediction accuracy parameter, it is indicated that the to-be-tested filling model has the highest prediction accuracy for the missing data of the target data, and thus the accurate filling of the missing data in the target data can be realized based on the model.

[0080] Specifically, since the to-be-tested filling model is trained based on the complete data after filling, the filling data in the complete data used for training the to-be-tested filling model can be directly used as the final filling data, and thus the final filling data is filled in the target data to obtain the final complete data.

[0081] Or, the corresponding data of the missing data in the target data in the normal class can also be input into the to-be-tested filling model with the highest prediction accuracy parameter again, and the output of the model is the final filling data, and then the final filling data is filled in the target data, so that the final complete data is obtained. The specific implementation manner of determining the final complete data based on the to-be-tested filling model with the highest prediction accuracy parameter can be selected according to actual needs, which is not limited here.

[0082] From the above, the data processing method provided by the embodiments of the present application fills the target data by using different data filling models, and then trains the target filling model by using each filled complete data, to obtain a plurality of to-be-tested filling models. Each to-be-tested filling model fully learns the related knowledge in the complete data filled by the corresponding data filling model, and can accurately and comprehensively represent the corresponding filled complete data. The data of the normal class in the target data is input into each to-be-tested filling model, and each to-be-tested filling model can output filling data. Comparing the filling data output by each to-be-tested filling model with the corresponding known data in the missing class in the target data can accurately lock the to-be-tested filling model with the highest prediction accuracy parameter, so as to select the corresponding complete data and improve the accuracy of filling the missing data in the target data.

[0083] Further, as a refinement and extension of the method shown in Figure 2 The embodiments of the present application also provide a data processing method.

[0084] Figure 3 For the overall architecture of the data processing method in the embodiments of the present application, refer to Figure 3 The architecture can include a data governance automation system based on machine learning, which can include a database communication module, a data set management module, an automated machine learning (AutoML) module and a platform management module.

[0085] The database communication module includes a database reading module and a database write-back module. The database reading module is used to read data from the database. The database write-back module is used to write data back to the database. When data is read and written, various read and write methods suitable for databases such as structured query language (SQL) can be used.

[0086] The data set management module includes a data set construction module, a data set archiving module and a data processing module. The data processing module is used for deleting abnormal data and enumeration class data and performing data standardization processing. The data set construction module is used for converting data into data available for training of a machine learning model. The data set archiving module is used for dividing data into a training set, a validation set and a test set.

[0087] The data governance AutoML module includes a data completion model center, a data feature correlation analysis center and a data verification model center. The data completion model center is used for filling data by using different data filling models to obtain different complete data. The data feature correlation analysis center is used for calculating a data feature correlation index between each class in data and selecting a class with the highest correlation index as a correlation class for a missing class in the data. The data verification model center is used for performing representation on the missing class and the correlation class in each complete data by using a target filling model, testing different models after representation by using original data in the complete data, and selecting a final filling data according to a test result.

[0088] The platform management module includes task management and task scheduling and is used for providing creation and scheduling management of various tasks of the entire platform.

[0089] Figure 4 A flowchart of a data processing method in an embodiment of the present application Figure 2 , as shown in Figure 4 , in the operation of each module of the system shown in Figure 3 , target data in a database is read out by a database reading module, and the database reading module sends the target data to a data processing module. The data processing module deletes abnormal data and enumeration class data in the target data and performs standardization processing on the target data, and then sends the processed data to a data set construction module. The data set construction module further processes the data processed by the data processing module into a data set available for model training and sends the data set to a data set archiving module. The data set archiving module splits the data set into a training set, a test set and a validation set and sends them to a data completion model center. The data completion model center optimizes multiple data filling models based on the training set and the validation set, inputs the test set into each optimized data filling model, and correspondingly obtains multiple complete data.

[0090] At the same time, the data set construction module also sends the data set to a data feature correlation analysis center. The data feature correlation analysis center performs correlation analysis on data features between classes in the data set, generates correlation indexes between the classes, and thus selects a class with the highest correlation index as a correlation class from multiple normal classes in the data set.

[0091] Next, the plurality of complete data output by the data completion model center and the association classes output by the data feature association analysis center are input into the data verification model center. For each complete data, the data verification model center trains the target filling model using the complete data, obtains a to-be-tested filling model corresponding to each complete data, and selects the complete data corresponding to the to-be-tested filling model with the highest prediction accuracy for the data in the normal class corresponding to the known data in the missing class, so as to determine the filling data in the selected complete data as the final filling data.

[0092] Finally, the database write-back module writes the final filling data back to the target data in the database.

[0093] Figure 5 The flowchart of the data processing method in the embodiment of the present application Figure 3 , see Figure 5 , the method can include:

[0094] S51: Obtain target data.

[0095] The target data here is a data set with data missing read from the database.

[0096] S52: Standardize the target data.

[0097] In the target data, there may be some incorrect or unsuitable data for subsequent machine learning model processing. The standardization here is to delete some unreasonable data in the target data and convert the target data into data that can be processed by the subsequent machine learning model.

[0098] Specifically, S52 can include:

[0099] Step A1: Delete abnormal data in the target data.

[0100] Abnormal data can refer to data that does not conform to common sense or is in the wrong format.

[0101] Deleting abnormal data in the target data can improve the accuracy of the target data, and thus improve the accuracy of subsequent machine learning model training, and ultimately improve the accuracy of data filling.

[0102] Table 2 below is an example of deleting abnormal data in the target data.

[0103] Table 2 Comparison of target data before and after deleting abnormal data

[0104] Age Category (Before Deletion) Age Category (After Deletion) 20 20 19 19 16 16 23 23 1044 NAN

[0105] In Table 2, for the age class before deletion, 20, 19, etc. all conform to the normal age interval, while 1044 does not conform to the normal age interval, so 1044 is deleted as abnormal data. In the age class after deletion, the corresponding position is replaced by missing data.

[0106] Step A2: converting characters in the target data into numerical values with preset meanings.

[0107] In the target data, sometimes there are some Chinese characters. However, the machine learning model cannot learn these Chinese characters well. Therefore, it is necessary to convert these Chinese characters into numerical values that can be used for more accurate learning by the machine learning model. That is, different Chinese characters are converted into different numerical values. The Chinese character corresponding to the numerical value is the corresponding preset meaning.

[0108] The following Table 3 is an example of converting characters in the target data into numerical values.

[0109] Table 3 Comparison before and after converting characters in the target data into numerical values

[0110] Gender Category (Before Conversion) Gender Category (After Conversion) Male 0 Male 0 Female 1 Male 0 Female 1

[0111] In Table 3, for males, all are converted to 0. For females, all are converted to 1. The subsequent machine learning model can better learn based on numerical values such as 0 and 1. The model can ultimately represent male and female gender regardless of whether it outputs 0 or 1.

[0112] Step A3: deleting data of a specified class in the target data, the specified class being a class in which each data is different.

[0113] In the target data, there may also be a class of data. In this class of data, each data is different. This is not meaningful for the subsequent machine learning model learning and training, and will also increase the data volume of model training. This is the specified class. The specified class data in the target data can be deleted to improve the efficiency and accuracy of subsequent machine learning model training.

[0114] The following Table 4 is an example of deleting specified class data in the target data.

[0115] Table 4 Comparison before and after deleting specified class data in the target data

[0116]

[0117] In Table 4, the address of each data is different, and the likelihood of learning related knowledge by the subsequent machine learning model is low, so the household registration class data in the target data is deleted.

[0118] Of course, if the granularity of the household registration data is at the city level or the province level, and these partitions have certain relevance to the filling of missing data, the household registration data can be retained. Similar data to household registration data can also be processed in this way.

[0119] It should be noted here that steps A1, A2, and A3 above can be executed at least one of them, or all of them. The execution order of steps A1, A2, and A3 is not limited here.

[0120] After the target data is preprocessed, the data is more accurate and standard. Then the missing data in the target data can be filled by using the data filling model.

[0121] Although the data filling model can fill the missing items in the data, the data filling model is essentially a machine learning model, and different data has different content and different missing parts. Directly using the data filling model to fill the missing data in the target data will reduce the accuracy of the filling. Therefore, the corresponding data of the missing class and the normal class in the target data can be used to train the model, and the corresponding data of the normal class in the missing data is input into the trained model to obtain more accurate filling data.

[0122] S53: For each data filling model, the data in the missing class and the corresponding data in the normal class of the target data are used as the output and input of the data filling model to train the data filling model.

[0123] S54: The corresponding data of the missing data in the normal class in the target data is input into the trained data filling model to obtain the filling data.

[0124] S55: The filling data is filled in the target data to obtain the complete data corresponding to the corresponding data filling model.

[0125] In model training, the data set needs to be divided into a training set, a validation set, and a test set. The proportions of the training set, the validation set, and the test set in the data set can be 70%, 20%, and 10% respectively, or other proportions determined according to actual conditions. Since the missing class of the target data has data missing, and the data filling model is needed to fill the data, in the training of the data filling model, the missing data and its corresponding data in the normal class are no longer involved in this model training. In other words, the missing data and its corresponding data in the normal class are used as the test set for data filling. The data other than the missing data and its corresponding data in the normal class in the target data is used as the training set and the validation set for model training.

[0126] In the training set and the validation set, the data in the missing class and the normal class are one-to-one corresponding. The data in the normal class is input into the model, and the corresponding data in the missing class is output from the model, so that the model learns for the target data and optimizes the parameters in the model. After training the model, the model learns the association knowledge between the normal class and the missing class in the target data.

[0127] Although different data filling models use the same training set and validation set in the target data for training, the specific algorithms and architectures used by different data filling models are different, so there are still differences between the trained models.

[0128] After the model training is completed, the normal class data in the test set is input again, and the output of the model is the filling of the missing data in the missing class.

[0129] For each data filling model, the above training method needs to be used for model training, and then the trained model is used to predict the missing data. Here, the number of data filling models can obtain the number of trained models. Then, the corresponding data of the missing data in the normal class is input into each trained model, and the number of filled data can be obtained. By filling these filled data into the missing positions of the target data, the number of complete data can be obtained.

[0130] Since there is certainly some regularity in the original feature distribution of the target data, there is also some similarity between different classes of the target data. In order to improve the accuracy of the model filling, various data filling models can use decision tree type machine learning models.

[0131] In the decision tree type machine learning model, the search space and the search strategy can be configured. The search space of each data filling model can be the same or different. The content and number of the search strategy of each data filling model can be the same or different. As long as the search space and the search strategy of each data filling model are not completely consistent.

[0132] Correspondingly, the above step S54 can include: processing the corresponding data of the missing data in the normal class in the target data using the search space and the search strategy in the trained data filling model to obtain the filled data.

[0133] In practical applications, in order to improve the construction efficiency of each data filling model, the search space of each data filling model can be different, and the search strategy can be the same.

[0134] Figure 6 For the architecture and training process of various data filling models in the embodiments of the present application, see Figure 6As shown, taking the construction of LightGBM, XGboost, and Catboost three data filling models as an example, the search space of the LightGBM model, the XGboost model, and the Catboost model corresponds to the LightGBM search space, the XGboost search space, and the Catboost search space respectively. The search strategy of the LightGBM model, the XGboost model, and the Catboost model corresponds to the evolutionary algorithm search strategy, the early stopping search strategy, and so on.

[0135] The normal class data in the training set and the validation set in the target data are input into the LightGBM search space, the XGboost search space, and the Catboost search space respectively, and the output of the LightGBM search space, the XGboost search space, and the Catboost search space is searched again by using the evolutionary algorithm search strategy, the early stopping search strategy, and so on. The search results are compared with the corresponding data of the missing class, and the parameters in the LightGBM search space, the XGboost search space, the Catboost search space, and the evolutionary algorithm search strategy, the early stopping search strategy, and so on are adjusted according to the comparison results, so as to obtain the LightGBM optimal model, the XGboost optimal model, and the Catboost optimal model.

[0136] The normal class data in the test set of the target data are input into the LightGBM optimal model, the XGboost optimal model, and the Catboost optimal model, so as to obtain three kinds of filling data of the missing data in the target data, that is, to obtain multiple complete data of the target data filled by different filling methods.

[0137] For the target data, several filling methods can fill in some complete data. Among the multiple complete data, the filling data at the same position can be averaged to obtain a complete data.

[0138] Table 5 below is an example of the target data without filling.

[0139] Table 5: Target data without filling

[0140] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 5 23 0.4 1 334 26 0.5 1 412

[0141] In Table 5, features 1 and 5 are missing classes, and features 2, 3, and 4 are normal classes.

[0142] Tables 6, 7, 8, and 9 below are examples of the target data after filling.

[0143] Table 6: Target data after filling

[0144] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 LGB Padding Data 5 23 0.4 1 334 LGB Padding Data 26 0.5 1 412

[0145] Table 7 filled target data two

[0146] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 XGB Padding Data 5 23 0.4 1 334 XGB Padding Data 26 0.5 1 412

[0147] Table 8 filled target data three

[0148] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 CATB Padding Data 5 23 0.4 1 334 CATB Padding Data 26 0.5 1 412

[0149] Table 9 filled target data four

[0150] Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 4 20 0.3 1 344 6 19 0.2 0 443 5 16 0.4 0 Average Data 5 23 0.4 1 334 Average Data 26 0.5 1 412

[0151] Next, a plurality of complete data are respectively characterized by the same model training, and then the accuracy of the prediction by the trained model is used to select the final complete data from the plurality of complete data.

[0152] As in the above tables 5-9, the target data or complete data can include a plurality of normal classes. Among these normal classes, there is generally one normal class most relevant to the missing class. Finding the normal class most relevant to the missing class, and then using the data in the normal class and the missing class to train the model, not only can reduce the amount of training data, improve the model training efficiency, but also can improve the accuracy of model training, thereby improving the data filling efficiency.

[0153] S56: Perform data feature correlation analysis on each normal class and at least one missing class to obtain the correlation index of each normal class and at least one missing class.

[0154] That is, for the target data, each class in it can be subjected to data feature correlation analysis with other classes. In this way, the correlation index of the missing class in the target data and each normal class can be obtained. In specific analysis, the correlation index between each two classes can be pre-calculated and stored, and then the correlation index corresponding to the missing class in the target data and each normal class is searched in the stored correlation index between each two classes. Alternatively, the correlation index can be calculated by using any known correlation analysis algorithm for the data under the missing class and the data under the normal class in the target data. The known correlation analysis algorithm includes but is not limited to the Apriori algorithm, the FP-growth algorithm, etc.

[0155] The correlation index between each two classes in the target data is obtained by calculating the correlation index of each class and other classes in the target data by using the above method. The correlation index between each two classes in the target data can be displayed by a chart.

[0156] Figure 7 For the correlation analysis result between each class in the target data in the embodiment of the present application, see Figure 7As shown, after the data correlation analysis is performed on the features 1-5 in Table 5, the correlation indexes between each of the features 1-5 are obtained. The correlation indexes between the features themselves are the highest, and the correlation indexes of feature 1 and feature 1, feature 2 and feature 2, feature 3 and feature 3, feature 4 and feature 4, and feature 5 and feature 5 are all 1.00. The correlation index of feature 1 and feature 2 is 0.54, the correlation index of feature 1 and feature 3 is 0.15, and so on.

[0157] S57: Determine the normal class with the correlation index reaching the preset threshold value as the correlation class of the at least one missing class, to train the target filling model by using the data in the correlation class and the at least one missing class in each complete data, respectively.

[0158] For each missing class in the target data, its corresponding correlation class needs to be found.

[0159] When determining specifically, the normal class corresponding to the highest correlation index can be directly selected from the correlation indexes of all normal classes corresponding to the missing class as the correlation class of the missing class. At this time, the preset threshold value is a dynamic value, that is, the correlation index corresponding to the correlation class.

[0160] In the target data in Table 5, there are two missing classes, that is, feature 1 and feature 5. For feature 1, the feature with the highest correlation index is found in the Figure 7 , that is, feature 2 with the correlation index of 0.54 with feature 1. For feature 5, the feature with the highest correlation index is found in the Figure 7 , that is, feature 3 with the correlation index of 0.71 with feature 5.

[0161] In the case that multiple correlation classes need to be determined for one missing class, or the correlation class needs to be found for the missing class, the preset threshold value can be a fixed value, for example: 0.3, 0.5, 0.6, etc. The size of the preset threshold value can be determined according to the actual situation. For this missing class, the normal classes corresponding to the correlation indexes greater than and equal to the preset threshold value are selected from the correlation indexes of all corresponding normal classes as the correlation classes of the missing class.

[0162] It should be noted that the above steps S53-S55 of obtaining multiple complete data based on the target data and the above steps S56-S57 of determining the correlation class based on the target data can be executed simultaneously to improve the data filling efficiency.

[0163] After the plurality of complete data of the target data and the corresponding associated classes of the missing classes are determined, the data in the missing classes and the normal classes in the plurality of complete data can be used to fully learn and train the same model, so that each trained model can represent the target data and various filling methods, respectively, so that the accuracy of the known data in the missing classes predicted by each trained model based on the data in the associated classes can select the final complete data from the plurality of complete data.

[0164] S58: Train the network architecture of the target filling model using each complete data to obtain an intermediate model.

[0165] The target filling model here is the same model referred to in the previous section. Only through the same architecture model can the filling methods of different data filling models be represented under the same benchmark, ensuring that the subsequent judgment is under the same benchmark, thereby improving the accuracy of the final complete data selection and the accuracy of the data filling.

[0166] In actual applications, the target filling model can be any of the previously used multiple data filling models, can be a new data filling model other than the multiple data filling models, or can be a model fused from the multiple data filling models.

[0167] For the target filling model being a model fused from the multiple data filling models, assuming that the multiple data filling models are LightGBM model, XGboost model and Catboost model, the LightGBM model, XGboost model and Catboost model can be fused into an Ensemble model based on the advantages of each model through Ensemble and the like.

[0168] When preparing the training data, for each complete data, the data in the missing classes and the associated classes in the complete data is divided into a test set, a validation set and a training set. Among them, the data amount in the training set, the validation set and the test set can account for 70%, 20% and 10% of the total data amount in the missing classes and the associated classes, respectively. Moreover, there cannot be filling data in the missing classes and corresponding data in the associated classes in the test set. The test set is used to test the accuracy of the prediction of the trained model and does not participate in the training, participates in the model test, and needs the known data in the missing classes and the corresponding data in the associated classes. In addition, other proportions can be adjusted according to actual conditions.

[0169] The training set and the validation set participate in model training. Specifically, during training, the data in the missing class and the associated class in all complete data can be used as output and input, respectively, for network architecture training of the target padding model. Network architecture training can refer to adjusting only the network architecture parameters in the model during training, while other parameters, such as optimizer parameters and loss function parameters, do not participate in this training. The network architecture parameters specifically refer to the number of leaf nodes, the size and width of the hidden layer, and the like.

[0170] To improve the training efficiency of the model network architecture, the amount of data used for training can be reduced. At the same time, the data used for training needs to use the padding data of various data padding models. Therefore, the padding data filled by various data padding models at the same position can be averaged, and then the missing class and the associated class data containing the average data are used to train the network architecture of the model.

[0171] Specifically, the above step S58 can include:

[0172] Step B1: average the padding data in each complete data to obtain average data.

[0173] The specific process of the average processing can be referred to in Table 9 described above, which will not be described here again.

[0174] Step B2: fill the average data into the target data to obtain training data.

[0175] Step B3: train the network architecture of the target padding model using the training data.

[0176] Taking the data in Table 9 described above as an example, Table 9 is the target data after padding the average data, i.e., the training data here. In the training data, there are a missing class and multiple normal classes. In order to further reduce the amount of data used for training, only the data in the missing class and the associated class can be used as output and input, respectively, for training of the model network architecture. That is, feature 1 and feature 2 are used as output and input, and feature 5 and feature 3 are used as output and input, for training of the model network architecture. At this time, an intermediate model containing various information is obtained.

[0177] S59: train the intermediate model using the data in the missing class and the corresponding data in the normal class in each complete data as output and input, respectively, to obtain a to-be-tested padding model corresponding to each complete data.

[0178] The padding data in different complete data is generated by using different data padding models. At this time, the intermediate model needs to be trained by using different padding data, respectively, to obtain various to-be-tested padding models representing the padding methods of different data padding models.

[0179] In the specific training, for each complete data, the missing class and the associated class in the training set and the validation set divided in advance can be taken as the output and the input respectively, and the intermediate model is trained to obtain the to-be-tested filling model corresponding to each complete data, which can characterize the filling mode of each data filling model.

[0180] S510: The data in the at least one missing class and the corresponding data in the at least one normal class in the target data are taken as the test standard and the input of each to-be-tested filling model, and the prediction accuracy parameter corresponding to each to-be-tested filling model is obtained.

[0181] For each to-be-tested filling model, the associated class data in the test set divided in advance is input into the to-be-tested filling model. Through the learning of the corresponding data filling model for predicting the filling data in advance, the to-be-tested filling model can output the predicted corresponding data in the missing class. The data in the at least one missing class is actually the known data in the missing class. By comparing the data predicted by the to-be-tested filling model with the known data in the missing class, the accuracy of the to-be-tested filling model in predicting the missing data can be known, that is, the prediction accuracy parameter of the to-be-tested filling model is determined. As many to-be-tested filling models as there are, as many prediction accuracy parameters corresponding to the to-be-tested filling models as there are.

[0182] S511: The complete data corresponding to the to-be-tested filling model with the highest prediction accuracy parameter is determined as the final complete data.

[0183] The prediction accuracy parameter is a numerical representation of whether the to-be-tested filling model can achieve accurate prediction. Each to-be-tested filling model is trained by each complete data. Each complete data is obtained by filling the missing data of the target data based on various data filling models. Therefore, the complete data corresponding to the to-be-tested filling model with the highest prediction accuracy parameter is the most accurate filling target data calculated, and is further determined as the final complete data.

[0184] Finally, the process of model training, model testing and filling data selection in the above steps S58-S511 is described again.

[0185] Figure 8 The process of model training, model testing and filling data selection in the embodiments of the present application is shown in the schematic diagram Figure 8As shown, taking the LightGBM model, the XGboost model and the Catboost model as the plurality of data filling models, and taking the Ensemble model obtained by fusing the LightGBM model, the XGboost model and the Catboost model as the target filling model as examples, the target data has been filled by the LightGBM model, the XGboost model and the Catboost model, to obtain LGB complete data, XGB complete data and CATB complete data. The filling data at the same missing positions in the LGB complete data, the XGB complete data and the CATB complete data are averaged to obtain average complete data. The data in the missing class and the associated class in the average complete data are divided into an average training set and an average validation set.

[0186] In model training, the average training set and the average validation set are used to train the Ensemble model to obtain an optimal Ensemble model. The optimal Ensemble model is the intermediate model in the foregoing embodiment.

[0187] The data in the missing class and the associated class in the LGB complete data are divided into an LGB training set, an LGB validation set and an LGB test set. The XGB complete data and the CATB complete data are also divided in this way to obtain an XGB training set, an XGB validation set and an XGB test set, and to obtain a CATB training set, a CATB validation set and a CATB test set. The LGB test set, the XGB test set and the CATB test set do not contain filling data and corresponding data in the associated class.

[0188] In model testing, the LGB training set and the LGB validation set are used to train the optimal Ensemble model to obtain an LGB model. The XGB training set and the XGB validation set are used to train the optimal Ensemble model to obtain an XGB model. The CATB training set and the CATB validation set are used to train the optimal Ensemble model to obtain a CATB model.

[0189] Then, in filling data selection, the LGB test set is used to test the LGB model to obtain a prediction accuracy parameter of the LGB model. The XGB test set is used to test the XGB model to obtain a prediction accuracy parameter of the XGB model. The CATB test set is used to test the CATB model to obtain a prediction accuracy parameter of the CATB model.

[0190] Finally, the prediction accuracy parameter can be used to lock the model with the optimal prediction performance from the LGB model, the XGB model and the CATB model, so that the filling data in the complete data corresponding to the model is used as the final filling data.

[0191] Thus far, the data processing method provided by the embodiments of the present application has been fully described.

[0192] Based on the same inventive concept, as an implementation of the above method, the embodiments of the present application further provide a data processing device.

[0193] Figure 9 For the structure of the data processing device in the embodiments of the present application Figure 1 , referring to Figure 9 , the device can include an acquisition module 91, a filling module 92, a training module 93, a test module 94, and a determination module 95.

[0194] The acquisition module 91 is configured to acquire target data, the target data including a plurality of data in at least one missing class and at least one normal class, the plurality of data in the at least one missing class corresponding one-to-one to the at least one normal class, and the at least one missing class having data missing.

[0195] The filling module 92 is configured to fill data in the target data using a plurality of data filling models respectively to obtain complete data corresponding to each data filling model.

[0196] The training module 93 is configured to train the target filling model using each complete data to obtain a to-be-tested filling model corresponding to each complete data.

[0197] The test module 94 is configured to use the data in the at least one missing class and the corresponding data in the at least one normal class in the target data as a test standard and input of each to-be-tested filling model to obtain a prediction accuracy parameter corresponding to each to-be-tested filling model.

[0198] The determination module 95 is configured to determine the final complete data based on the to-be-tested filling model with the highest prediction accuracy parameter.

[0199] Further, as a refinement and expansion of the device shown in Figure 9 , the embodiments of the present application further provide a data processing device.

[0200] Figure 10 For the structure of the data processing device in the embodiments of the present application Figure 2 , referring to Figure 10 , the device can include an acquisition module 101, a standardization module 102, a filling module 103, a correlation analysis module 104, a training module 105, a test module 106, and a determination module 107.

[0201] The acquisition module 101 is configured to acquire target data, the target data including a plurality of data in at least one missing class and at least one normal class, the plurality of data in the at least one missing class corresponding one-to-one to the at least one normal class, and the at least one missing class having data missing.

[0202] The standardization module 102 is configured to perform at least one of the following: deleting abnormal data in the target data; converting characters in the target data into numerical values with preset meanings; and deleting data of a specified class in the target data, the specified class being a class in which each data is different.

[0203] The filling module 103 includes a model training unit 1031, a prediction unit 1032, and a filling unit 1033.

[0204] The model training unit 1031 is configured to train a data filling model by taking data in a missing class of the target data and corresponding data in a normal class as the input and output of the data filling model.

[0205] The prediction unit 1032 is configured to input the corresponding data of the missing data in the normal class into the trained data filling model to obtain filled data.

[0206] In a case where the plurality of data filling models are decision tree type machine learning models, the decision tree type machine learning models include a search space and a search strategy, the search space of each data filling model is different, and the search strategy is the same, the prediction unit 1032 is specifically configured to process the corresponding data of the missing data in the normal class in the target data by using the search space and the search strategy in the trained data filling model to obtain the filled data.

[0207] The filling unit 1033 is configured to fill the filled data in the target data to obtain complete data corresponding to the target filling model.

[0208] The correlation analysis module 104 is configured to perform data feature correlation analysis on a plurality of normal classes and at least one missing class respectively to obtain a correlation index of each normal class and the at least one missing class; determine a normal class with a correlation index reaching a preset threshold as a correlation class of the at least one missing class, and train the target filling model by using the correlation class and data in the at least one missing class in each complete data respectively.

[0209] The training module 105 includes an architecture training unit 1051 and a model generation unit 1052.

[0210] The architecture training unit 1051 is configured to train a network architecture of the target filling model by using each complete data to obtain an intermediate model.

[0211] The architecture training unit 1051 is specifically configured to perform average processing on the filled data in each complete data to obtain average data; fill the average data in the target data to obtain training data; and train the network architecture of the target filling model by using the training data.

[0212] The model generation unit 1052 is configured to train an intermediate model by taking the data of the missing class and the corresponding data of the normal class in each complete data as input and output, respectively, to obtain a to-be-tested filling model corresponding to each complete data.

[0213] The test module 106 is configured to take the data in at least one missing class and the corresponding data in at least one normal class in the target data as a test standard and input of each to-be-tested filling model, to obtain a prediction accuracy parameter corresponding to each to-be-tested filling model.

[0214] The determination module 107 is configured to determine the complete data corresponding to the to-be-tested filling model with the highest prediction accuracy parameter as the final complete data.

[0215] It should be noted that the above description of the device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0216] Based on the same inventive concept, the embodiments of the present application further provide an electronic device.

[0217] Figure 11 For the structural schematic diagram of the electronic device in the embodiments of the present application, refer to Figure 11 As shown in the figure, the electronic device can include a processor 111, a memory 112, and a bus 113. The processor 111 and the memory 112 can communicate with each other through the bus 113. The processor 111 is configured to invoke program instructions in the memory 112 to execute the method in one or more of the above embodiments.

[0218] It should be noted that the above description of the electronic device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the electronic device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0219] Based on the same inventive concept, the embodiments of the present application further provide a computer readable storage medium, which can include a stored program. When the program runs, it controls the device where the storage medium is located to execute the method in one or more of the above embodiments.

[0220] It should be noted that the above description of the computer readable storage medium embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the computer readable storage medium embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0221] Based on the same inventive concept, the embodiment of the present application further provides a computer program product, which comprises computer programs or instructions, and when the computer programs or instructions are executed by a device, the method in one or more embodiments described above is implemented.

[0222] It should be noted here that the description of the above computer program product embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the computer program product embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0223] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain target data, which includes at least one missing class and at least one normal class, wherein multiple data in the at least one missing class correspond one-to-one with multiple data in the at least one normal class, and the at least one missing class has missing data; The target data is filled using various data filling models to obtain multiple complete data corresponding to the various data filling models. The various data filling models are various algorithms used for data filling. The target imputation model is trained using each complete dataset to obtain multiple imputation models to be tested corresponding to multiple complete datasets. The target imputation model is one of the multiple data imputation models, or a data imputation model fused from multiple data imputation models, or a data imputation model other than multiple data imputation models. The data in at least one missing class and the corresponding data in at least one normal class in the target data are used as the test standard and input for each imputation model to be tested, so as to obtain the prediction accuracy parameter corresponding to each imputation model to be tested. The data in the missing class are the data that still exist in the missing class and are not missing. For each imputation model to be tested, the corresponding data of the normal class in the target data is input into the imputation model to be tested, and the imputation model to be tested outputs imputation data. The output imputation data is compared with the data of the missing class to obtain the prediction accuracy parameter of the imputation model to be tested. The final complete data is determined based on the test imputation model with the highest prediction accuracy parameter; The step of training the target filling model using each complete dataset to obtain multiple filling models to be tested corresponding to multiple complete datasets includes: The network architecture of the target filling model is trained using each complete dataset to obtain an intermediate model; Using the missing class data and the corresponding normal class data in each complete dataset as output and input, the intermediate model is trained to obtain multiple test imputation models corresponding to multiple complete datasets. The target data includes multiple normal classes; before training the target imputation model using each complete data set, the method further includes: Data feature association analysis is performed on the multiple normal classes and the at least one missing class to obtain the association index between each normal class and the at least one missing class; The normal class whose correlation index reaches a preset threshold is determined as the correlation class of the at least one missing class, so that the target imputation model is trained using the data of the correlation class in each complete data and the data of the at least one missing class respectively.

2. The method according to claim 1, characterized in that, The process of training the network architecture of the target filling model using each complete dataset includes: The average data is obtained by averaging the filler data in each complete dataset. The average data is filled into the target data to obtain the training data; The network architecture of the target filling model is trained using the training data.

3. The method according to any one of claims 1 to 2, characterized in that, The step involves filling the target data using various data filling models to obtain multiple complete data sets corresponding to the different data filling models, including: For each data imputation model, the data in the missing class and the corresponding data in the normal class of the target data are used as the output and input of the data imputation model to train the data imputation model; The corresponding data of the missing data in the normal class in the target data is input into the trained data filling model to obtain the filled data; The filling data is used to fill the target data to obtain the complete data corresponding to the corresponding data filling model.

4. The method according to claim 3, characterized in that, The various data imputation models are decision tree-based machine learning models. Each decision tree-based machine learning model includes a search space and a search strategy. The search space is different for each data imputation model, but the search strategy is the same. The step of inputting the corresponding data of the missing data in the target data within the normal class into the trained data imputation model to obtain the imputed data includes: The missing data in the target data is matched with the corresponding data in the normal class. The search space and search strategy in the model are then used to fill in the missing data to obtain the filled data.

5. The method according to any one of claims 1 to 2, characterized in that, The final complete data is determined based on the test imputation model with the highest prediction accuracy parameter, including: The complete data corresponding to the test imputation model with the highest prediction accuracy parameter is determined as the final complete data.

6. The method according to any one of claims 1 to 2, characterized in that, Before filling the target data using various data filling models, the method further includes: Perform at least one of the following: Delete the abnormal data from the target data; Convert the characters in the target data into numerical values ​​with preset meanings; Delete data of a specified class from the target data, where the specified class is a category in which each data item is different.

7. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire target data, which includes at least one missing class and at least one normal class, wherein multiple data in the at least one missing class correspond one-to-one with multiple data in the at least one normal class, and the at least one missing class has missing data. The data filling module is used to fill the target data using various data filling models to obtain multiple complete data corresponding to the various data filling models. The various data filling models are various algorithms used for data filling. The training module is used to train the target imputation model using each complete data set to obtain multiple imputation models to be tested corresponding to multiple complete data sets. The target imputation model is one of the multiple data imputation models, or a data imputation model fused from multiple data imputation models, or a data imputation model other than multiple data imputation models. The testing module is used to take the data from at least one missing class and the corresponding data from at least one normal class in the target data as the test standard and input for each imputation model to be tested, and obtain the prediction accuracy parameter corresponding to each imputation model to be tested. The data in the missing class is the data that still exists in the missing class and is not missing. Specifically, for each imputation model to be tested, the corresponding data of the normal class in the target data is input into the imputation model to be tested, the imputation model to be tested outputs imputation data, and the output imputation data is compared with the data of the missing class to obtain the prediction accuracy parameter of the imputation model to be tested. The determination module is used to determine the final complete data based on the test imputation model with the highest prediction accuracy parameter. The training module includes: an architecture training unit and a model generation unit. The architecture training unit is used to train the network architecture of the target-filled model using each complete dataset to obtain an intermediate model; The model generation unit is used to train intermediate models by taking the missing class data and the corresponding normal class data in each complete dataset as output and input, respectively, to obtain multiple test imputation models corresponding to multiple complete datasets. The target data includes multiple normal categories; the device further includes: The association analysis module is used to perform data feature association analysis on multiple normal classes and at least one missing class to obtain the association index between each normal class and at least one missing class; the normal classes whose association index reaches a preset threshold are identified as the association classes of at least one missing class, so as to train the target imputation model using the association classes in each complete data and the data in at least one missing class respectively.

8. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a bus. The processor and the memory communicate with each other via the bus. The processor is used to call program instructions in the memory to execute the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program that, when executed, controls the device on which the computer-readable storage medium is located to perform the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed by the device, implement the method as described in any one of claims 1 to 6.

Images