A method, device and electronic device for determining training data

Abstract

The present application discloses a method, device and electronic device for determining training data, which can be applied to the field of artificial intelligence or finance. Using the replacement feature, the feature of the first training data is replaced to obtain the second training data, and the second training data is subjected to a data addition operation to obtain the third training data with the same number as the training data to be analyzed, and the second residual value corresponding to the third training data is determined, and when the difference between the first residual value and the second residual value is less than a preset threshold, the first training data is used as the training data used for model training. That is, in the present invention, through the above steps, the normal first training data is screened out from the training data to be analyzed, and the first training data is used as the training data used for model training, which ensures the accuracy of the model training data, thereby improving the accuracy of the neural network model trained based on the training data.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more specifically, to a method, device and electronic device for determining training data. Background Art

[0002] With the continuous development of technology, the usage rate of neural network models is increasing. Before using the neural network model, it is necessary to first use training data to train the neural network model. At present, in the stage of preparing training data, some users will use data cheating methods, such as modifying certain features of the training data to forge the training data, so that the accuracy of the neural network model trained based on this training data is low. Summary of the invention

[0003] In view of this, the present invention provides a method, device and electronic device for determining training data to solve the problem of low accuracy of a neural network model trained based on forged training data.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A method for determining training data, comprising:

[0006] Obtaining training data to be analyzed and model prediction results of the training data to be analyzed;

[0007] Performing an abnormal analysis on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results and use them as abnormal training data;

[0008] Deleting the abnormal training data from the training data to be analyzed to obtain first training data;

[0009] Determine a first association relationship between the feature value of the training data to be analyzed and the corresponding label value, and determine a second association relationship between the feature value of the first training data and the corresponding label value;

[0010] Determine a replacement feature based on the first association relationship and the second association relationship;

[0011] Using the replacement feature, replace the feature of the first training data to obtain second training data, and determine a first residual value corresponding to the first training data and the second training data;

[0012] Performing a data addition operation on the first training data to obtain third training data having the same number as the training data to be analyzed;

[0013] Determine a second residual value corresponding to the third training data, and when a difference between the first residual value and the second residual value is less than a preset threshold, use the first training data as training data for model training.

[0014] Further, an abnormal analysis is performed on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results and use them as abnormal training data, including:

[0015] Classify the model prediction results according to the label values ​​to obtain classification results;

[0016] The training data to be analyzed corresponding to the model prediction results whose values ​​satisfy the preset data interval range in the classification results are used as the first abnormal training data;

[0017] Obtaining user behavior data corresponding to the model prediction result, and performing cluster analysis on the user behavior data to obtain a clustering result;

[0018] Based on the clustering result, an abnormal user is determined, and the to-be-analyzed training data corresponding to the abnormal user is used as second abnormal training data;

[0019] The first abnormal training data and the second abnormal training data are combined to obtain abnormal training data.

[0020] Further, based on the clustering result, determining abnormal users includes:

[0021] Based on the clustering results, users whose user behavior data meets preset behavior rules are screened out and used as initial abnormal users;

[0022] Using the training data to be analyzed of the initial abnormal user as fourth training data, and using the training data to be analyzed of the non-initial abnormal user as fifth training data;

[0023] Calculate a third residual value of the eigenvalue of the fifth training data;

[0024] Performing feature replacement on the fourth training data to obtain sixth training data, and calculating a fourth residual value of the feature value of the sixth training data;

[0025] In a case where the third residual value and the fourth residual value satisfy a preset residual value rule, the initial abnormal user is regarded as an abnormal user.

[0026] Further, based on the first association relationship and the second association relationship, determining a replacement feature includes:

[0027] The feature value with the largest change in feature sequence value in the first association relationship and the second association relationship is used as a first replacement feature;

[0028] Randomly select a preset number of features from the feature values ​​with the largest non-sequential value changes in the first association relationship and the second association relationship, and use them as second replacement features;

[0029] The first replacement feature and the second replacement feature are used as replacement features.

[0030] Furthermore, it also includes:

[0031] When the difference between the first residual value and the second residual value is not less than a preset threshold, return to perform abnormal analysis on the model prediction result to determine the training data to be analyzed corresponding to the abnormal model prediction result and use it as abnormal training data. The step is performed sequentially until the difference between the first residual value and the second residual value is less than a preset threshold, and the first training data is used as the training data for model training and stops.

[0032] A device for determining training data, comprising:

[0033] A data acquisition module, used to acquire the training data to be analyzed and the model prediction results of the training data to be analyzed;

[0034] An abnormality analysis module is used to perform abnormality analysis on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results and use them as abnormal training data;

[0035] A data deletion module, used for deleting the abnormal training data from the training data to be analyzed to obtain first training data;

[0036] a relationship determination module, configured to determine a first association relationship between a feature value of the training data to be analyzed and a corresponding label value, and to determine a second association relationship between a feature value of the first training data and a corresponding label value;

[0037] A feature determination module, configured to determine a replacement feature based on the first association relationship and the second association relationship;

[0038] A replacement module, configured to use the replacement feature to replace the feature of the first training data to obtain second training data, and determine a first residual value corresponding to the first training data and the second training data;

[0039] A data adding module, used for performing a data adding operation on the first training data to obtain third training data having the same number as the training data to be analyzed;

[0040] The data determination module is used to determine the second residual value corresponding to the third training data, and when the difference between the first residual value and the second residual value is less than a preset threshold, use the first training data as training data for model training.

[0041] Furthermore, the abnormality analysis module includes:

[0042] A classification submodule is used to classify the model prediction results according to the label values ​​to obtain classification results;

[0043] A first data determination submodule is used to use the training data to be analyzed corresponding to the model prediction results whose values ​​satisfy the preset data interval range in the classification results as the first abnormal training data;

[0044] The clustering submodule is used to obtain user behavior data corresponding to the model prediction results, and perform cluster analysis on the user behavior data to obtain clustering results;

[0045] A second data determination submodule is used to determine abnormal users based on the clustering result, and use the to-be-analyzed training data corresponding to the abnormal users as second abnormal training data;

[0046] The third data determination submodule is used to combine the first abnormal training data and the second abnormal training data to obtain abnormal training data.

[0047] Further, the second data determination submodule includes:

[0048] A user screening unit, configured to screen out users whose user behavior data meets preset behavior rules based on the clustering result and use them as initial abnormal users;

[0049] a data determination unit, configured to use the to-be-analyzed training data of the initial abnormal user as fourth training data, and use the to-be-analyzed training data of the non-initial abnormal user as fifth training data;

[0050] A first residual calculation unit, used for calculating a third residual value of the eigenvalue of the fifth training data;

[0051] a second residual calculation unit, configured to perform feature replacement on the fourth training data to obtain sixth training data, and calculate a fourth residual value of a feature value of the sixth training data;

[0052] A user determination unit is configured to determine the initial abnormal user as an abnormal user if the third residual value and the fourth residual value satisfy a preset residual value rule.

[0053] Furthermore, the feature determination module includes:

[0054] A first feature determination unit, configured to use a feature value with the largest change in feature sequence value between the first association relationship and the second association relationship as a first replacement feature;

[0055] A feature screening unit, configured to randomly screen a preset number of features from the feature values ​​with the largest non-sequential value changes in the first association relationship and the second association relationship, and use the features as second replacement features;

[0056] The second feature determination unit is used to use the first replacement feature and the second replacement feature as replacement features.

[0057] An electronic device comprising: a memory and a processor;

[0058] Wherein, the memory is used to store programs;

[0059] The processor calls the program and is used to execute the above-mentioned method for determining training data.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The present invention provides a method, device and electronic device for determining training data, which obtain training data to be analyzed and model prediction results of the training data to be analyzed, perform abnormal analysis on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results, and use the abnormal training data as abnormal training data, delete the abnormal training data from the training data to be analyzed to obtain first training data, determine a first residual value corresponding to the first training data, determine a first association relationship between a feature value of the training data to be analyzed and a corresponding label value, and determine a second association relationship between the feature value of the first training data and the corresponding label value, determine a replacement feature based on the first association relationship and the second association relationship, use the replacement feature to replace the feature of the first training data to obtain second training data, perform a data addition operation on the second training data to obtain third training data with the same number as the training data to be analyzed, determine a second residual value corresponding to the third training data, and use the first training data as the training data used for model training when the difference between the first residual value and the second residual value is less than a preset threshold. That is, in the present invention, through the above steps, normal first training data are screened out from the training data to be analyzed, and the first training data are used as training data for model training, thereby ensuring the accuracy of the model training data, thereby improving the accuracy of the neural network model trained based on the training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0063] Figure 1 A method flow chart of a method for determining training data provided by an embodiment of the present invention;

[0064] Figure 2 A method flow chart of another method for determining training data provided by an embodiment of the present invention;

[0065] Figure 3 A method flow chart of another method for determining training data provided by an embodiment of the present invention;

[0066] Figure 4 A method flow chart of another method for determining training data provided by an embodiment of the present invention;

[0067] Figure 5 A schematic diagram of the structure of a device for determining training data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] With the continuous development of technology, the usage rate of neural network models is increasing. Before the neural network model is used, it is first necessary to use training data to train the neural network model. At present, in the stage of preparing training data, some users will use data cheating to modify the training data. For example, internal users of model training, such as own employees, technical instructors, and external aid, may use data cheating methods to modify the characteristics of training data to intervene in the model during the model training process (i.e., white box). External users of model training, such as loan users and credit risk assessment users, use black box methods to repeatedly iterate input and output of the model, thereby intervening in the model using data cheating methods such as adversarial case generation. Both methods make the trained neural network model less accurate, which seriously affects the normal operation of the machine learning model and causes abnormal business handling.

[0070] In order to solve this technical problem, the inventors have discovered through research that abnormal data can be identified from a large amount of training data and removed from the training data, thereby ensuring the accuracy of the final training data and thereby improving the accuracy of model training using the training data.

[0071] Specifically, an embodiment of the present invention provides a method, device and electronic device for determining training data, which obtains training data to be analyzed and a model prediction result of the training data to be analyzed, performs an abnormal analysis on the model prediction result to determine the training data to be analyzed corresponding to the abnormal model prediction result, and uses it as abnormal training data, deletes the abnormal training data from the training data to be analyzed to obtain first training data, and determines a first residual value corresponding to the first training data, determines a first association relationship between a feature value of the training data to be analyzed and a corresponding label value, and determines a second association relationship between the feature value of the first training data and the corresponding label value, determines a replacement feature based on the first association relationship and the second association relationship, uses the replacement feature to replace the feature of the first training data to obtain second training data, performs a data addition operation on the second training data to obtain third training data with the same number as the training data to be analyzed, determines a second residual value corresponding to the third training data, and uses the first residual value and the second residual value as The difference between the first residual value and the second residual value is less than a preset threshold value, Use the first training data as the training data used for model training. That is, in the present invention, through the above steps, normal first training data are screened out from the training data to be analyzed, and the first training data are used as training data for model training, thereby ensuring the accuracy of the model training data, thereby improving the accuracy of the neural network model trained based on the training data.

[0072] It should be noted that the method, device and electronic device for determining training data provided by the present invention can be used in the field of artificial intelligence or finance. The above is only an example and does not limit the application field of the method, device and electronic device for determining training data provided by the present invention.

[0073] Based on the above content, the embodiment of the present invention provides a method for determining training data, referring to Figure 1 , which may include:

[0074] S11, obtaining training data to be analyzed and model prediction results of the training data to be analyzed.

[0075] In this embodiment, training data is obtained when performing model training. Since it is necessary to filter out training data with higher reliability from these data later, the training data obtained at this time is referred to as training data to be analyzed.

[0076] Each training data to be analyzed corresponds to a label value. For example, if the training data to be analyzed is a picture, the label value is cat or dog.

[0077] The present invention can optimize the model during model training or using the model to provide external services. At this time, each training data to be analyzed can be input into the model to obtain a model prediction result. The model prediction result can be a numerical value and a classification result, such as a cat, with a numerical value of 90.

[0078] S12: Perform an abnormal analysis on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results, and use them as abnormal training data.

[0079] In this embodiment, based on the model prediction results, abnormal training data to be analyzed are screened out and used as abnormal training data.

[0080] S13: Delete the abnormal training data from the training data to be analyzed to obtain first training data.

[0081] After the abnormal training data is determined, the abnormal training data is deleted from the training data to be analyzed, and the remaining data in the training data to be analyzed is called the first training data, that is, data that is initially considered to be normal and not forged.

[0082] After the first training data is determined, the first training data may be verified through subsequent steps to determine whether the obtained first training data is accurate.

[0083] The verification method is to perform data redundancy and feature processing respectively. Data redundancy is to upsample the first training data to obtain a richer data set to calculate the residual value M. Feature processing is to perform feature screening, filtering, feature combination and other feature processing on the data set to calculate the residual value N of the data set. If the two residual values ​​are not much different, the first training data is accurate. The verification process will be described in detail later.

[0084] S14: Determine a first association relationship between the feature value of the training data to be analyzed and the corresponding label value, and determine a second association relationship between the feature value of the first training data and the corresponding label value.

[0085] In this embodiment, each training data to be analyzed has a corresponding characteristic value. Taking a cat as an example, the characteristic value may be the characteristic value corresponding to the eyes, ears and other parts. Then, each training data to be analyzed has a corresponding label value, which is introduced in the above corresponding content. Please refer to the above corresponding description.

[0086] Then, calculate the correlation between the feature values ​​of all the training data to be analyzed and the corresponding label values ​​to determine the correlation between each feature value and the label value, such as the cat's ear feature proves that the cat has the highest correlation, the cat's eye feature proves that the cat has the second highest correlation... The cat's tail feature proves that the cat has the lowest correlation. The first correlation relationship obtained is the sorting result of the correlation between each feature value and the label value, such as the first correlation relationship is:

[0087] 1-ears, 2-eyes...5-tail.

[0088] For the first training data, the second correlation relationship between the feature value of the first training data and the corresponding label value is also calculated. For example, the second correlation relationship is:

[0089] 1-tail, 2-ears, 3-eyes…

[0090] S15, based on the first association relationship and the second association relationship, determine the replacement feature.

[0091] Specifically, refer to Figure 2 , step S15 may include:

[0092] S21. Use the feature value with the largest change in feature sequence value in the first association relationship and the second association relationship as the first replacement feature.

[0093] Specifically, the feature serial number value may be 12 ... 5 in the above association relationship. At this time, the feature serial number values ​​of the same feature in the first association relationship and the second association relationship are subtracted, and the absolute value after the subtraction is taken, and the feature with the largest absolute value is used as the first replacement feature.

[0094] S22: Randomly select a preset number of features from the feature values ​​with the largest change in non-sequential values ​​in the first association relationship and the second association relationship, and use them as second replacement features.

[0095] After the first replacement feature is determined, a preset number, such as 2, of features are selected from the remaining features as the second replacement features.

[0096] S23. Use the first replacement feature and the second replacement feature as replacement features.

[0097] In this embodiment, the first replacement feature and the second replacement feature are selected as replacement features because some features are a combination of multiple sub-features, such as the tail feature can be a combination of hair color, tail length, hair glossiness, etc. The advantage of replacing features is that features can be combined and filtered by feature processing, preventing cheaters from cheating on data based on features, and improving the detection capability of the solution.

[0098] S16. Use the replacement feature to replace the feature of the first training data to obtain second training data, and determine a first residual value corresponding to the first training data and the second training data.

[0099] Specifically, the replacement feature in the first training data is replaced to obtain the second training data, the residual of the eigenvalue of the first training data is calculated, and the residual of the eigenvalue of the second training data is calculated, and the result of the weighted sum of the two is used as the first residual value.

[0100] The first residual value can represent the residual value of the normal sample. After filtering the abnormal records by statistical methods, the characteristics of the normal sample can be initially retained. The sample deviation level after removing the abnormal value can be calculated by residual value calculation.

[0101] In addition to the feature replacement described above, a feature filtering operation may also be performed. In this case, the replacement features in the first training data may be directly filtered to obtain second training data.

[0102] Specifically, data cheaters may be more clear about the details of the model because they may forge the features of the model. In this case, the detection model needs to process the features of the samples submitted by the users and calculate the residual value level of the samples multiple times through feature combination and feature filtering methods, which will help to find out which features are forged by customers.

[0103] S17: Perform a data addition operation on the first training data to obtain third training data having the same number as the training data to be analyzed.

[0104] Specifically, assuming that the number of training data to be analyzed is 1000, after removing the abnormal training data, the remaining first training data is 900. In order to keep the same number as the training data to be analyzed, in this embodiment, a data addition operation is performed on the first training data. At this time, the data distribution of the first training data can be calculated, and 100 new data are generated based on the data distribution. The two are combined to obtain 1000 data, which are called third training data.

[0105] S18. Determine a second residual value corresponding to the third training data, and when a difference between the first residual value and the second residual value is less than a preset threshold, use the first training data as training data for model training.

[0106] Specifically, the second residual value of the eigenvalue of the third training data is calculated, and then the difference between the first residual value and the second residual value is compared. If the difference is less than a preset threshold, it means that the third training data after data addition can still be regarded as normal data, that is, it is further verified that the first data used to obtain the third training data is normal data, thereby improving the reliability of normal data screening.

[0107] Thereafter, model training and optimization operations are performed using the first training data.

[0108] In this embodiment, training data to be analyzed and model prediction results of the training data to be analyzed are obtained, and an abnormal analysis is performed on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results, and the abnormal training data is deleted from the training data to be analyzed to obtain first training data, and a first residual value corresponding to the first training data is determined, a first association relationship between a feature value of the training data to be analyzed and a corresponding label value is determined, and a second association relationship between the feature value of the first training data and the corresponding label value is determined, and a replacement feature is determined based on the first association relationship and the second association relationship, and the feature of the first training data is replaced using the replacement feature to obtain second training data, and a data addition operation is performed on the second training data to obtain third training data having the same number as the training data to be analyzed, and a second residual value corresponding to the third training data is determined, and when the difference between the first residual value and the second residual value is less than a preset threshold, the first training data is used as the training data for model training. That is, in the present invention, through the above steps, normal first training data are screened out from the training data to be analyzed, and the first training data are used as training data for model training, thereby ensuring the accuracy of the model training data, thereby improving the accuracy of the neural network model trained based on the training data.

[0109] In addition, the present invention can identify machine learning model security issues caused by various black boxes and white boxes such as developers, internal and external customers, and multi-modal data cheating. During the implementation of the invention, residual analysis and feature replacement are used to discover the attacker's replacement data and abnormal data as much as possible, so as to accurately identify abnormal users, which is conducive to discovering model problems in advance and avoiding unnecessary economic losses.

[0110] Based on the above, refer to Figure 3 Step S12 "perform abnormal analysis on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results and use them as abnormal training data" may include:

[0111] S31, classifying the model prediction results according to the label values ​​to obtain classification results.

[0112] The label value in this embodiment has been introduced accordingly in the above embodiment. At this time, the model prediction result, such as a cat with a value of 90 points, can be classified according to the label into at least one category to obtain a classification result, such as cats and dogs.

[0113] S32: The training data to be analyzed corresponding to the model prediction results whose values ​​satisfy the preset data interval in the classification results are used as the first abnormal training data.

[0114] In this embodiment, the training data to be analyzed corresponding to the model prediction results with larger values ​​(such as greater than 96) and smaller values ​​(less than 10) in the classification results are considered to be abnormal data, which are called first abnormal training data in this embodiment.

[0115] The reasons why the above data with larger or smaller values ​​are abnormal data are:

[0116] For samples composed of normal data sets, their distribution is consistent, and the prediction results are slightly different after upsampling or feature engineering, and the fluctuation of the prediction results is small. Therefore, for data whose values ​​calculated by the above method are greater than the system specific threshold P or less than the specific threshold Q, it can be determined that the data is high-risk abnormal data. For high-risk abnormal data, the data can be confirmed manually, where P>Q.

[0117] S32: Obtain user behavior data corresponding to the model prediction result, and perform cluster analysis on the user behavior data to obtain a clustering result.

[0118] In this embodiment, the user behavior data mainly includes usage time, usage times and usage cycle, etc.

[0119] The number of times used refers to the number of times the device is used continuously. For example, if the device is used at 8 a.m. for 3 times, the number of times used is 3 and the time of use is 8 a.m. If the device is used every two weeks, the usage cycle is two weeks.

[0120] External users may use data cheating to deceive the model in order to achieve some illegal purposes, and maliciously guide the artificial intelligence model to make wrong predictions. A typical financial application scenario is the bank's automatic credit model. If external customers are participants in the model construction, at this time, they can construct a specific data set to add a normal sample set, or reconstruct the normal sample set by adding forged features, thereby taking advantage of the model's defects to make illegal predictions and achieve the purpose of illegal credit. Therefore, in this embodiment, the user behavior data will be analyzed to find users with abnormal operation frequencies. The data of this user is very likely to be abnormal training data.

[0121] After obtaining the behavior data of each user, cluster analysis will be performed on the user behavior data to obtain cluster results. For example, most users will use the model between 9:00 and 13:00 in the morning, and the number of times of use is 1.

[0122] S33: Determine abnormal users based on the clustering result, and use the to-be-analyzed training data corresponding to the abnormal users as second abnormal training data.

[0123] In this embodiment, after the clustering result is obtained, the users corresponding to the data that are greatly different from the clustering result are abnormal users.

[0124] Specifically, refer to Figure 4 Based on the clustering result, determining abnormal users may include:

[0125] S41. Based on the clustering result, screen out users whose user behavior data satisfy preset behavior rules and use them as initial abnormal users.

[0126] In this embodiment, the preset behavior rule may be that at least one of the usage time, usage frequency and usage cycle exceeds the set value, that is, the user has different usage habits from most users. The usage habits of most users can be determined based on clustering results. For example, most users use the model from 9:00 to 13:00 in the morning, while some users use it from 21:00 to 3:00 the next day, which is different from most users, and they are considered to be initial abnormal users.

[0127] S42: Use the training data to be analyzed of the initial abnormal user as fourth training data, and use the training data to be analyzed of the non-initial abnormal user as fifth training data.

[0128] At this time, the data of the initial abnormal users and the data of the non-initial abnormal users are separated and used as the fourth training data and the fifth training data, respectively.

[0129] S43: Calculate a third residual value of the eigenvalue of the fifth training data.

[0130] For the specific residual calculation process, please refer to the corresponding part above.

[0131] S44: Perform feature replacement on the fourth training data to obtain sixth training data, and calculate a fourth residual value of the feature value of the sixth training data.

[0132] The process of feature replacement and residual calculation refers to the corresponding parts above. Feature replacement can also be feature filtering.

[0133] S45: When the third residual value and the fourth residual value satisfy a preset residual value rule, treating the initial abnormal user as an abnormal user.

[0134] Specifically, when the difference between the third residual value of the normal user and the fourth residual value of the initial abnormal user is too large and greater than a set threshold, it is considered that the initial abnormal user is indeed different from the normal user, and the initial abnormal user is regarded as an abnormal user.

[0135] In addition, the higher the fourth residual value, the shorter the usage period, and the higher the number of services, the higher the possibility that the user is not an abnormal user.

[0136] After the abnormal user is determined, the to-be-analyzed training data corresponding to the abnormal user is used as second abnormal training data.

[0137] S34: Combine the first abnormal training data and the second abnormal training data to obtain abnormal training data.

[0138] In this embodiment, the first abnormal training data and the second abnormal training data are combined to obtain abnormal training data.

[0139] On the basis of this embodiment, when the difference between the first residual value and the second residual value is not less than the preset threshold, the above-mentioned preset behavior rules can be adjusted to redetermine the initial abnormal user and the abnormal user, and then re-execute the step of "performing an abnormal analysis on the model prediction result to determine the training data to be analyzed corresponding to the abnormal model prediction result, and use it as abnormal training data", and then execute it sequentially until the difference between the first residual value and the second residual value is less than the preset threshold, and stop when the first training data is used as the training data for model training.

[0140] In this embodiment, a process for determining abnormal training data is provided, and then the abnormal training data can be determined through the above process, and the abnormal data can be eliminated to obtain normal training data, thereby ensuring the accuracy of the trained model.

[0141] Optionally, based on the above-mentioned embodiment of the method for determining training data, another embodiment of the present invention provides a device for determining training data, referring to Figure 5 , which may include:

[0142] The data acquisition module 11 is used to acquire the training data to be analyzed and the model prediction results of the training data to be analyzed;

[0143] An abnormality analysis module 12 is used to perform abnormality analysis on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results and use them as abnormal training data;

[0144] A data deletion module 13 is used to delete the abnormal training data from the training data to be analyzed to obtain first training data;

[0145] A relationship determination module 14, configured to determine a first association relationship between a feature value of the training data to be analyzed and a corresponding label value, and to determine a second association relationship between a feature value of the first training data and a corresponding label value;

[0146] A feature determination module 15, configured to determine a replacement feature based on the first association relationship and the second association relationship;

[0147] A replacement module 16, configured to use the replacement feature to replace the feature of the first training data to obtain second training data, and determine a first residual value corresponding to the first training data and the second training data;

[0148] A data adding module 17, configured to perform a data adding operation on the first training data to obtain third training data having the same number as the training data to be analyzed;

[0149] The data determination module 18 is used to determine the second residual value corresponding to the third training data, and when the difference between the first residual value and the second residual value is less than a preset threshold, use the first training data as training data for model training.

[0150] Furthermore, the abnormality analysis module includes:

[0151] A classification submodule is used to classify the model prediction results according to the label values ​​to obtain classification results;

[0152] A first data determination submodule is used to use the training data to be analyzed corresponding to the model prediction results whose values ​​satisfy the preset data interval range in the classification results as the first abnormal training data;

[0153] The clustering submodule is used to obtain user behavior data corresponding to the model prediction results, and perform cluster analysis on the user behavior data to obtain clustering results;

[0154] A second data determination submodule is used to determine abnormal users based on the clustering result, and use the to-be-analyzed training data corresponding to the abnormal users as second abnormal training data;

[0155] The third data determination submodule is used to combine the first abnormal training data and the second abnormal training data to obtain abnormal training data.

[0156] Further, the second data determination submodule includes:

[0157] A user screening unit, configured to screen out users whose user behavior data meets preset behavior rules based on the clustering result and use them as initial abnormal users;

[0158] a data determination unit, configured to use the to-be-analyzed training data of the initial abnormal user as fourth training data, and use the to-be-analyzed training data of the non-initial abnormal user as fifth training data;

[0159] A first residual calculation unit, used for calculating a third residual value of the eigenvalue of the fifth training data;

[0160] a second residual calculation unit, configured to perform feature replacement on the fourth training data to obtain sixth training data, and calculate a fourth residual value of a feature value of the sixth training data;

[0161] A user determination unit is configured to determine the initial abnormal user as an abnormal user if the third residual value and the fourth residual value satisfy a preset residual value rule.

[0162] Furthermore, the feature determination module includes:

[0163] A first feature determination unit, configured to use a feature value with the largest change in feature sequence value between the first association relationship and the second association relationship as a first replacement feature;

[0164] A feature screening unit, configured to randomly screen a preset number of features from the feature values ​​with the largest non-sequential value changes in the first association relationship and the second association relationship, and use the features as second replacement features;

[0165] The second feature determination unit is used to use the first replacement feature and the second replacement feature as replacement features.

[0166] Furthermore, the abnormality analysis module 12 is also used for:

[0167] When the difference between the first residual value and the second residual value is not less than a preset threshold, an abnormal analysis is performed on the model prediction result to determine the training data to be analyzed corresponding to the abnormal model prediction result and use it as abnormal training data.

[0168] In this embodiment, training data to be analyzed and model prediction results of the training data to be analyzed are obtained, and an abnormal analysis is performed on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results, and the abnormal training data is deleted from the training data to be analyzed to obtain first training data, and a first residual value corresponding to the first training data is determined, a first association relationship between a feature value of the training data to be analyzed and a corresponding label value is determined, and a second association relationship between the feature value of the first training data and the corresponding label value is determined, and a replacement feature is determined based on the first association relationship and the second association relationship, and the feature of the first training data is replaced using the replacement feature to obtain second training data, and a data addition operation is performed on the second training data to obtain third training data having the same number as the training data to be analyzed, and a second residual value corresponding to the third training data is determined, and when the difference between the first residual value and the second residual value is less than a preset threshold, the first training data is used as the training data for model training. That is, in the present invention, through the above steps, normal first training data are screened out from the training data to be analyzed, and the first training data are used as training data for model training, thereby ensuring the accuracy of the model training data, thereby improving the accuracy of the neural network model trained based on the training data.

[0169] It should be noted that, for the working process of each module, sub-module and unit in this embodiment, please refer to the corresponding description in the above embodiment, which will not be repeated here.

[0170] Optionally, based on the above embodiments of the method and device for determining training data, another embodiment of the present invention provides an electronic device, including: a memory and a processor;

[0171] Wherein, the memory is used to store programs;

[0172] The processor calls the program and is used to execute the above-mentioned method for determining training data.

[0173] In this embodiment, training data to be analyzed and model prediction results of the training data to be analyzed are obtained, and an abnormal analysis is performed on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results, and the abnormal training data is deleted from the training data to be analyzed to obtain first training data, and a first residual value corresponding to the first training data is determined, a first association relationship between a feature value of the training data to be analyzed and a corresponding label value is determined, and a second association relationship between the feature value of the first training data and the corresponding label value is determined, and a replacement feature is determined based on the first association relationship and the second association relationship, and the feature of the first training data is replaced using the replacement feature to obtain second training data, and a data addition operation is performed on the second training data to obtain third training data having the same number as the training data to be analyzed, and a second residual value corresponding to the third training data is determined, and when the difference between the first residual value and the second residual value is less than a preset threshold, the first training data is used as the training data for model training. That is, in the present invention, through the above steps, normal first training data are screened out from the training data to be analyzed, and the first training data are used as training data for model training, thereby ensuring the accuracy of the model training data, thereby improving the accuracy of the neural network model trained based on the training data.

[0174] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining training data, characterized in that: Applied to user behavior data analysis, including: Obtaining training data to be analyzed and model prediction results of the training data to be analyzed; Performing an abnormal analysis on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results and use them as abnormal training data; Deleting the abnormal training data from the training data to be analyzed to obtain first training data; Determine a first association relationship between the feature value of the training data to be analyzed and the corresponding label value, and determine a second association relationship between the feature value of the first training data and the corresponding label value; Determine a replacement feature based on the first association relationship and the second association relationship; Using the replacement feature, replace the feature of the first training data to obtain second training data, and determine a first residual value corresponding to the first training data and the second training data; Performing a data addition operation on the first training data to obtain third training data having the same number as the training data to be analyzed; Determine a second residual value corresponding to the third training data, and when a difference between the first residual value and the second residual value is less than a preset threshold, use the first training data as training data for model training; An abnormal analysis is performed on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results and use them as abnormal training data, including: classifying the model prediction results according to the label values ​​to obtain classification results; using the training data to be analyzed corresponding to the model prediction results whose values ​​meet the preset data interval range in the classification results as the first abnormal training data; obtaining user behavior data corresponding to the model prediction results, and performing cluster analysis on the user behavior data to obtain clustering results; based on the clustering results, determining abnormal users, and using the training data to be analyzed corresponding to the abnormal users as the second abnormal training data; combining the first abnormal training data and the second abnormal training data to obtain abnormal training data; The user behavior data includes usage time, usage times and usage cycles, where the usage times refers to the number of consecutive times the user uses the device.

2. The method for determining training data according to claim 1, characterized in that: Based on the clustering result, determining abnormal users includes: Based on the clustering results, users whose user behavior data meets preset behavior rules are screened out and used as initial abnormal users; Using the training data to be analyzed of the initial abnormal user as fourth training data, and using the training data to be analyzed of the non-initial abnormal user as fifth training data; Calculate a third residual value of the eigenvalue of the fifth training data; Performing feature replacement on the fourth training data to obtain sixth training data, and calculating a fourth residual value of the feature value of the sixth training data; In a case where the third residual value and the fourth residual value satisfy a preset residual value rule, the initial abnormal user is regarded as an abnormal user.

3. The method for determining training data according to claim 1, characterized in that: Determining a replacement feature based on the first association relationship and the second association relationship includes: The feature value with the largest change in feature sequence value in the first association relationship and the second association relationship is used as a first replacement feature; Randomly select a preset number of features from the feature values ​​with the largest non-sequential value changes in the first association relationship and the second association relationship, and use them as second replacement features; The first replacement feature and the second replacement feature are used as replacement features.

4. The method for determining training data according to claim 1, characterized in that: Also includes: When the difference between the first residual value and the second residual value is not less than a preset threshold, return to perform abnormal analysis on the model prediction result to determine the training data to be analyzed corresponding to the abnormal model prediction result and use it as abnormal training data. The step is performed sequentially until the difference between the first residual value and the second residual value is less than a preset threshold, and the first training data is used as the training data for model training and stops.

5. A device for determining training data, characterized in that: Applied to user behavior data analysis, including: A data acquisition module, used to acquire the training data to be analyzed and the model prediction results of the training data to be analyzed; An abnormality analysis module is used to perform abnormality analysis on the model prediction results to determine the training data to be analyzed corresponding to the abnormal model prediction results and use them as abnormal training data; A data deletion module, used for deleting the abnormal training data from the training data to be analyzed to obtain first training data; a relationship determination module, configured to determine a first association relationship between a feature value of the training data to be analyzed and a corresponding label value, and to determine a second association relationship between a feature value of the first training data and a corresponding label value; A feature determination module, configured to determine a replacement feature based on the first association relationship and the second association relationship; A replacement module, configured to use the replacement feature to replace the feature of the first training data to obtain second training data, and determine a first residual value corresponding to the first training data and the second training data; A data adding module, used for performing a data adding operation on the first training data to obtain third training data having the same number as the training data to be analyzed; a data determination module, configured to determine a second residual value corresponding to the third training data, and, when a difference between the first residual value and the second residual value is less than a preset threshold, use the first training data as training data for model training; The abnormality analysis module comprises: A classification submodule is used to classify the model prediction results according to the label values ​​to obtain classification results; A first data determination submodule is used to use the training data to be analyzed corresponding to the model prediction result whose value satisfies the preset data interval in the classification result as the first abnormal training data; The clustering submodule is used to obtain user behavior data corresponding to the model prediction results, and perform cluster analysis on the user behavior data to obtain clustering results; A second data determination submodule is used to determine abnormal users based on the clustering results, and use the to-be-analyzed training data corresponding to the abnormal users as second abnormal training data; A third data determination submodule, configured to combine the first abnormal training data and the second abnormal training data to obtain abnormal training data; The user behavior data includes usage time, usage times and usage cycles, where the usage times refers to the number of consecutive times the user uses the device.

6. The apparatus for determining training data according to claim 5, characterized in that: The second data determination submodule includes: A user screening unit, configured to screen out users whose user behavior data meets preset behavior rules based on the clustering result and use them as initial abnormal users; a data determination unit, configured to use the to-be-analyzed training data of the initial abnormal user as fourth training data, and use the to-be-analyzed training data of the non-initial abnormal user as fifth training data; A first residual calculation unit, used for calculating a third residual value of the eigenvalue of the fifth training data; a second residual calculation unit, configured to perform feature replacement on the fourth training data to obtain sixth training data, and calculate a fourth residual value of a feature value of the sixth training data; A user determination unit is configured to determine the initial abnormal user as an abnormal user if the third residual value and the fourth residual value satisfy a preset residual value rule.

7. The apparatus for determining training data according to claim 5, characterized in that: The feature determination module comprises: A first feature determination unit, configured to use a feature value with the largest change in feature sequence value between the first association relationship and the second association relationship as a first replacement feature; A feature screening unit, configured to randomly screen a preset number of features from the feature values ​​with the largest non-sequential value changes in the first association relationship and the second association relationship, and use the features as second replacement features; The second feature determination unit is used to use the first replacement feature and the second replacement feature as replacement features.

8. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor calls the program and is used to execute a method for determining training data as described in any one of claims 1-4.

Images