Processing method and device for optimizing data set

Abstract

The embodiment of the invention relates to a processing method and device for optimizing a data set. The method comprises the steps that the maximum feature range of input data of a target model is confirmed; a classification model and a dimension reduction model are constructed; carrying out big data acquisition; preprocessing the collected data set; performing classification model data set construction based on the preprocessed data set to obtain a corresponding training / evaluation set training classification model; after training is finished, similar data de-duplication is carried out on the preprocessed data set based on a classification model; performing sample balance and data enhancement processing on the deduplicated data set; performing dimension reduction model data set construction based on the enhanced first data set to obtain a corresponding training / evaluation set, and training a dimension reduction model based on the training / evaluation set and the target model; and after the training is finished, performing feature dimension reduction on the first data set based on the dimension reduction model to obtain a corresponding optimized data set. Through the method, the model training efficiency can be improved, the model generalization is enhanced, and the model training complexity is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a processing method and device for optimizing a data set. Background Art

[0002] With the maturity and development of artificial intelligence (AI) technology, a large number of AI models have been applied in multiple technical fields. Before each AI model is put into use, it will collect corresponding big data based on the model functions (such as regression calculation, classification, etc.), and build a data set for model training based on the collected data. The data quality of the data set is closely related to the model training effect. The better the data quality, the better the training effect, and vice versa. It is difficult to maintain a good quality state for the massive data collected through public channels, so a round of optimization must be performed on the collected data set before model training. At present, most conventional data set optimization schemes are still based on simple data cleaning and standardization. There are no mature solutions for classification deduplication, classification balance, and feature dimensionality reduction, which can further improve the efficiency and effect of model training. And this is the technical problem that the present invention needs to solve. Summary of the invention

[0003] The purpose of the present invention is to provide a processing method, device, electronic device and computer-readable storage medium for optimizing data sets in view of the defects of the prior art. The present invention confirms the maximum feature range of the model input data based on the output data features of the target model; and constructs a classification model for classifying and identifying the input data, and constructs a dimension reduction model for reducing the feature dimension of the input high-dimensional data; and collects big data based on the input / output data feature range of the target model as the data collection requirement; and preprocesses the collected data set; and constructs a classification model data set based on the preprocessed data set to obtain a corresponding training / evaluation set, and trains the classification model based on the training / evaluation set; and after the classification model training is completed, the preprocessed data set is deduplicated based on the classification model; and the deduplicated data set is sample balanced and data enhanced to obtain a first data set; and the dimension reduction model data set is constructed based on the first data set to obtain a corresponding training / evaluation set, and the dimension reduction model is trained based on the training / evaluation set and the target model; and after the dimension reduction model training is completed, the first data set is subjected to feature dimension reduction processing based on the dimension reduction model to obtain an optimized data set. The present invention adds classification deduplication, classification balancing, and feature dimensionality reduction steps on the basis of conventional optimization schemes. The present invention can not only further improve the training efficiency of the model and enhance the generalization performance of the model, but also reduce the complexity of model training through dimensionality reduction.

[0004] To achieve the above object, a first aspect of an embodiment of the present invention provides a method for processing an optimized data set, the method comprising:

[0005] The target training model of the data set is used as the corresponding first target model; and based on the output data features of the first target model, the maximum feature range of the model input data is confirmed to obtain the corresponding first feature item sequence; and a first classification model for classifying and identifying the input data is constructed according to the preset first data classification label set and the first feature item sequence; and a first dimension reduction model for performing feature dimension reduction processing on the input high-dimensional data is constructed according to the first feature item sequence; the first feature item sequence includes N first feature items, N is a positive integer greater than 1; the first data classification label set includes S first classification labels, S is a positive integer greater than 1;

[0006] Taking the input / output data feature range of the first target model as the data collection requirement, big data collection is performed to obtain a corresponding first collection data set;

[0007] Preprocessing the first acquired data set to obtain a corresponding first preprocessed data set;

[0008] Based on the first preprocessed data set, a classification model data set is constructed to obtain a corresponding first training set and a first evaluation set; and based on the first training set and the first evaluation set, the first classification model is trained; and after the classification model training is completed, the first preprocessed data set is deduplicated based on the first classification model to obtain a corresponding first deduplicated data set;

[0009] Performing sample balancing and data enhancement processing on the first deduplication data set to obtain a corresponding first data set;

[0010] A dimensionality reduction model data set is constructed based on the first data set to obtain a corresponding second training set and a second evaluation set; and the first dimensionality reduction model is trained based on the second training set, the second evaluation set and the first target model; and after the dimensionality reduction model training is completed, feature dimensionality reduction processing is performed on the first data set based on the first dimensionality reduction model to obtain a corresponding first optimized data set.

[0011] Preferably, the model input vector of the first target model is denoted as X, and the model output vector is denoted as Y; the feature dimension of the model input vector X is adjustable and the maximum is N, that is, the model input vector X is composed of at most N input features x i , 1≤i≤N; the feature dimension of the model output vector Y remains unchanged, that is, the model output vector Y consists of M output features y j Composition, 1≤j≤M, M is a positive integer greater than 1; each of the output features yj Corresponding to a target model prediction feature item;

[0012] The first classification model is implemented based on a machine learning model or a deep learning model; the first classification model is used to input the vector d according to the model. ain Perform classification recognition processing and output the corresponding classification prediction vector d aout ; The model input vector d ain The feature dimension is fixed to N; the classification prediction vector d aout The feature dimension of is fixed to S, and is composed of S classification prediction probabilities; the classification prediction probabilities correspond one-to-one to the first classification labels;

[0013] The first dimensionality reduction model is implemented based on a type of machine learning model or a deep learning model; the first dimensionality reduction model is used to bin Perform feature dimensionality reduction processing and output the corresponding dimensionality reduction vector d bout ; The model input vector d bin The feature dimension is fixed to N; the dimension reduction vector d bout The feature dimension K is adjustable, 1≤K <N;

[0014] The first acquisition data set includes a plurality of first acquisition data records; the first acquisition data record includes N first feature data fields and M second feature data fields; the first feature data fields correspond one-to-one to the first feature items; the second feature data fields correspond one-to-one to the target model prediction feature items;

[0015] The first preprocessed data set includes a plurality of first preprocessed data records; the first preprocessed data record includes N third feature data fields and M fourth feature data fields; the third feature data fields correspond one-to-one to the first feature items; the fourth feature data fields correspond one-to-one to the target model prediction feature items;

[0016] The first deduplication data set includes a plurality of first deduplication data records; the first deduplication data record includes a first data type field, N fifth characteristic data fields and M sixth characteristic data fields; the first data type field corresponds to the first classification label in a one-to-one manner; the fifth characteristic data field corresponds to the first characteristic item in a one-to-one manner; the sixth characteristic data field corresponds to the target model prediction characteristic item in a one-to-one manner;

[0017] The first data set includes a plurality of first data records; the first data record includes a second data type field, N seventh feature data fields and M eighth feature data fields; the second data type field corresponds one-to-one to the first classification label; the seventh feature data field corresponds one-to-one to the first feature item; the eighth feature data field corresponds one-to-one to the target model prediction feature item;

[0018] The first optimized data set includes multiple second data records; the second data record includes a third data type field, K ninth feature data fields and M tenth feature data fields; the second data type field corresponds one-to-one to the first classification label; the tenth feature data field corresponds one-to-one to the target model prediction feature item.

[0019] Preferably, the step of performing big data collection based on the input / output data feature range of the first target model as a data collection requirement to obtain a corresponding first collection data set specifically includes:

[0020] The N first feature items and the M target model prediction feature items form a corresponding collection object attribute set;

[0021] And through multiple big data collection channels, big data collection is performed with the collection object attribute set as the attribute requirements of a single data object to obtain multiple collected data objects; the multiple big data collection channels at least include government / institutional data channels, commercial databases, social media Internet and third-party data service companies; each of the collected data objects includes N first attribute data and M second attribute data, the first attribute data corresponds to the first feature item one by one; the second attribute data corresponds to the target model prediction feature item one by one;

[0022] and taking each of the collected data objects as a corresponding current data object; and taking each of the first attribute data of the current data object as a corresponding first characteristic data field, and taking each of the second attribute data of the current data object as a corresponding second characteristic data field, and forming a corresponding first collected data record by the N third characteristic data fields and the M fourth characteristic data fields corresponding to the current data object;

[0023] All the first collected data records obtained form a corresponding first collected data set.

[0024] Preferably, the preprocessing of the first acquired data set to obtain a corresponding first preprocessed data set specifically includes:

[0025] Pre-setting a corresponding feature item data rationality inspection standard and a corresponding feature item data standardization conversion rule for the N first feature items and the M target model prediction feature items respectively;

[0026] A round of traversal is performed on all the first collected data records of the first collected data set; and in this round of traversal, the first collected data record currently traversed is used as the corresponding current data record; a round of polling is performed on all characteristic data fields of the current data record; and in this round of polling, the first or second characteristic data field currently polled is recorded as the corresponding current field; and whether the current field is empty is identified; if the current field is empty, the current data record is recorded as an abnormal record; if the current field is not empty, the current field is rationally checked based on the corresponding characteristic item data rationality check standard to obtain the corresponding The first inspection result is obtained, and when the first inspection result is reasonable, the field data of the current field is subjected to standardized conversion processing based on the corresponding standardized conversion rule of the characteristic item data and the conversion result is used as a corresponding third or fourth characteristic data field, and when the first inspection result is unreasonable, the current data record is recorded as an abnormal record; and at the end of polling all characteristic data fields of the current data record, whether the current data record is recorded as an abnormal record is identified, and if not, a corresponding first pre-processed data record is formed based on the N third characteristic data fields and the M fourth characteristic data fields corresponding to the current data record;

[0027] And when the traversal of all the first collected data records of the first collected data set is completed, all the obtained first pre-processed data records form a corresponding first pre-processed data set.

[0028] Preferably, the construction of the classification model dataset based on the first preprocessed dataset to obtain the corresponding first training set and first evaluation set specifically includes:

[0029] Pre-setting a corresponding data classification rule for each of the first classification labels of the first data classification label set; and based on a manual or machine data sorting method, randomly sorting a group of a specified number of the first pre-processed data records from the first pre-processed data set according to each of the data classification rules to form a corresponding first category record set;

[0030] Each obtained set of the first category records is recorded as the corresponding current set; the first classification label corresponding to the current set is used as the corresponding current classification label; all the first pre-processed data records of the current set are traversed in one round; and in this round of traversal, the first pre-processed data record currently traversed is used as the corresponding current record; and based on the model input vector d ain The vector format of the model input vector is composed of the N third feature data fields of the current record and recorded as the corresponding first training vector; and based on the classification prediction vector d aout A classification prediction vector with a length of S and all the classification prediction probabilities being zero is set as the corresponding first label vector, and the classification prediction probability corresponding to the current classification label in the first label vector is reset to 1; and the first training vector corresponding to the current record and the first label vector form a corresponding first record; and at the end of this round of traversal, all the first records corresponding to the current set form a corresponding first record set; and the first record set is divided into two subsets according to a preset first segmentation ratio and recorded as the corresponding first subset and second subset; wherein the first and second subsets are both composed of multiple first records; the ratio of the total number of records in the first subset to the total number of records in the second subset satisfies the first segmentation ratio;

[0031] The S obtained first subsets constitute the corresponding first training set; and the S obtained second subsets constitute the corresponding first evaluation set; wherein, the first training set and the first evaluation set are both composed of multiple first records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; the first record includes the first training vector and the first label vector.

[0032] Furthermore, the training of the first classification model based on the first training set and the first evaluation set specifically includes:

[0033] Step 61, taking the first of the first records in the first training set as the corresponding current training record;

[0034] Step 62, inputting the first training vector of the current training record into the first classification model for classification recognition processing to obtain a corresponding first classification prediction vector;

[0035] Step 63, inputting the first classification prediction vector and the first label vector of the current training record into a preset first model loss function to calculate and obtain a corresponding first loss value;

[0036] Wherein, the first model loss function at least includes an L1 loss function, an L2 loss function and a cross entropy loss function;

[0037] Step 64, identifying whether the first loss value satisfies a preset first loss value range; if the first loss value satisfies the first loss value range, identifying whether the current training record is the last first record of the first training set, if so, turning to step 65, if not, extracting the next first record of the first training set as the new current training record and returning to step 62; if the first loss value does not satisfy the first loss value range, performing a round of parameter optimization processing on the model parameters of the first classification model in the direction of minimizing the first model loss function based on a preset first model optimizer, and returning to step 62 at the end of this round of parameter optimization processing;

[0038] Wherein, the first model optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0039] Step 65, perform a round of traversal on all the first records of the first evaluation set; and in this round of traversal, use the first record currently traversed as the corresponding current evaluation record; and input the first training vector of the current evaluation record into the first classification model for classification and recognition processing to obtain the corresponding second classification prediction vector; and form a corresponding first prediction-label pair by the second classification prediction vector and the first label vector of the current training record; and at the end of this round of traversal, construct a confusion matrix based on all the first prediction-label pairs obtained and record it as the corresponding first confusion matrix; and perform corresponding precision and recall calculations based on the first confusion matrix to obtain the corresponding first precision and first recall; and perform F1 score calculation based on the first precision and the first recall to obtain the corresponding first F1 score;

[0040] Step 66, identifying whether the first precision, the first recall and the first F1 score all satisfy the corresponding first precision range, the first recall range and the first F1 score range; if the first precision does not satisfy the corresponding first precision range, or the first recall does not satisfy the corresponding first recall range, or the first F1 score does not satisfy the corresponding first F1 score range, returning to step 61 to continue training; if the first precision, the first recall and the first F1 score all satisfy the corresponding first precision range, the first recall range and the first F1 score range, stopping training and confirming that the classification model training is completed.

[0041] Preferably, performing similar data deduplication processing on the first preprocessed data set based on the first classification model to obtain a corresponding first deduplication data set specifically includes:

[0042] Perform a round of traversal on all the first preprocessed data records of the first preprocessed data set; and in this round of traversal, record the first preprocessed data record currently traversed as the corresponding current record; and based on the model input vector d ain The vector format is composed of N third feature data fields of the current record to form a model input vector as the corresponding first input vector; the first input vector is input into the first classification model for classification recognition processing to obtain the corresponding third classification prediction vector; the first classification label corresponding to the classification prediction probability with the largest probability value in the third classification prediction vector is taken as the corresponding first data type; and at the end of this round of traversal, multiple first pre-processed data records with the same first data type are extracted and grouped into one category to form a corresponding second category record set;

[0043] After the traversal of all the first preprocessed data records of the first preprocessed data set is completed, a round of traversal is performed on all the obtained second-category record sets; and in this round of traversal, the currently traversed second-category record set is recorded as the corresponding current set; and based on the model input vector d ain The vector format is composed of N third characteristic data fields of each of the first preprocessed data records in the current set to form a corresponding first data vector; and the vector similarity of each two of the first data vectors in the current set is calculated to obtain a corresponding first similarity; and the first similarities whose similarity exceeds a preset first similarity threshold are recorded as corresponding second similarities; and one of the two first preprocessed data records corresponding to each of the second similarities is selected and recorded as a corresponding duplicate record; and all the duplicate records in the current set are deleted;

[0044] After traversing all the second-category record sets, all the second-category record sets are merged to form a corresponding first merged set;

[0045] And perform a round of traversal on all the first pre-processed data records of the first merged set; and during this round of traversal, record the first pre-processed data record currently traversed as the corresponding current record; and use the first data type corresponding to the current record, N third characteristic data fields and M fourth characteristic data fields as the corresponding first data type fields, N fifth characteristic data fields and M sixth characteristic data fields to form a corresponding first deduplication data record; and at the end of this round of traversal, all the first deduplication data records obtained are used to form the corresponding first deduplication data set.

[0046] Preferably, performing sample balancing and data enhancement processing on the first deduplication data set to obtain the corresponding first data set specifically includes:

[0047] Step 81, extracting multiple first deduplication data records with the same first data type field from the first deduplication data set and grouping them into one category to form a corresponding third category record set;

[0048] Step 82, counting the total number of records of each of the third-category record sets to obtain a corresponding first total number of records; and calculating the mean of all the obtained first total number of records to obtain a corresponding first average total number; and calculating a corresponding first ratio = first total number of records / first average total number based on each of the first total number of records and the first average total number; and recording the third-category record set whose first ratio is less than a preset first ratio threshold as a corresponding under-sampled set, and recording the third-category record set whose first ratio is greater than a preset second ratio threshold as a corresponding over-sampled set;

[0049] Wherein, 0<first ratio threshold<1<second ratio threshold;

[0050] Step 83, identifying the total number of the under-sampled sets and the over-sampled sets; if the total number of the under-sampled sets is not 0 or the total number of the over-sampled sets is not 0, then when the total number of the under-sampled sets is not 0, using the over-sampling method to perform record addition processing on each of the under-sampled sets, and when the total number of the over-sampled sets is not 0, using the under-sampling method to perform record deletion processing on each of the over-sampled sets, and after the record addition / deletion processing is completed, return to step 82; if the total number of the under-sampled sets and the total number of the over-sampled sets are both 0, go to step 84;

[0051] Step 84, merge all the third category record sets to obtain the corresponding second merged set; and perform a round of traversal on all the first deduplicated data records of the second merged set; and in this round of traversal, record the first deduplicated data record currently traversed as the corresponding current record; and use the first data type field, N fifth characteristic data fields and M sixth characteristic data fields corresponding to the current record as the corresponding second data type field, N seventh characteristic data fields and M eighth characteristic data fields to form a corresponding first data record; and at the end of this round of traversal, all the first data records obtained are used to form the corresponding first data set.

[0052] Preferably, the step of constructing a dimension reduction model dataset based on the first dataset to obtain a corresponding second training set and a second evaluation set specifically includes:

[0053] Extract multiple first data records with the same second data type field in the first data set and group them into one category to form a corresponding fourth category record set; randomly select a specified number of first data records from each fourth category record set to form a corresponding second record set; and merge all the obtained second record sets to obtain a corresponding third merged set;

[0054] and perform a round of traversal on all the first data records of the third merged set; and in this round of traversal, record the first data record currently traversed as the corresponding current record; and based on the model input vector d of the first dimensionality reduction model bin The vector format of the model input vector is composed of the N seventh characteristic data fields of the current record and recorded as the corresponding second training vector; and the vector format of the model output vector based on the first target model is composed of the M eighth characteristic data fields of the current record and recorded as the corresponding second label vector; and the second training vector and the second label vector corresponding to the current record form a corresponding second record; and at the end of this round of traversal, all the obtained second records form a corresponding second record sequence;

[0055] and dividing the second record sequence into two subsequences according to a preset second division ratio, recorded as a corresponding first subsequence and a second subsequence; wherein both the first and second subsequences are composed of a plurality of the second records; and a ratio of the total number of records in the first subsequence to the total number of records in the second subsequence satisfies the second division ratio;

[0056] The first subsequence is used as the corresponding second training set, and the second subsequence is used as the corresponding second evaluation set; wherein the second training set and the second evaluation set are both composed of multiple second records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio; and the second record includes the second training vector and the second label vector.

[0057] Furthermore, the training of the first dimensionality reduction model based on the second training set, the second evaluation set and the first target model specifically includes:

[0058] Step 101, identifying a preset feature dimensionality reduction processing mode; if the feature dimensionality reduction processing mode is the first mode, go to step 102; if the feature dimensionality reduction processing mode is the second mode, go to step 105;

[0059] Wherein, the feature dimension reduction processing mode includes a first mode and a second mode;

[0060] Step 102, according to the preset k value selection rule, select multiple integer values ​​from the value range of the feature dimension K of the first dimensionality reduction model to form a corresponding k value range; and based on each k value in the k value range, set the feature dimension K of the first dimensionality reduction model to obtain a corresponding second dimensionality reduction model; and based on the feature dimension K of each second dimensionality reduction model, set the feature dimension of the model input vector X of the first target model to obtain a corresponding second target model; and each second dimensionality reduction model is connected with the corresponding second target model to form a corresponding first training model;

[0061] Wherein, the k value range includes multiple k values;

[0062] Step 103, performing model training and evaluation on each of the first training models based on the second training set and the second evaluation set to obtain a corresponding first evaluation score;

[0063] Step 104, taking the second dimensionality reduction model and the second target model corresponding to the first training model having the highest first evaluation score as the latest first dimensionality reduction model and the first target model; and going to step 110;

[0064] Step 105, identifying key feature items in the first feature item sequence according to the second training set and the second evaluation set based on a principal component analysis method to obtain a corresponding first key feature item sequence;

[0065] Wherein, the first key feature item sequence includes a plurality of first key feature items;

[0066] Step 106: Count the total number of feature items in the first key feature item sequence to obtain a corresponding first feature total number; and set the feature dimension K of the first dimensionality reduction model based on the first feature total number to obtain a corresponding third dimensionality reduction model;

[0067] Step 107, reconstructing the second training set and the second evaluation set according to the first key feature item sequence based on a principal component analysis method to obtain a corresponding third training set and a third evaluation set;

[0068] Wherein, the third training set and the third evaluation set are both composed of a plurality of third records; the ratio of the total number of records of the third training set to the third evaluation set satisfies the second segmentation ratio; the third record includes a third training vector, a third label vector and a first target vector;

[0069] Step 108, performing model training and evaluation on the third dimensionality reduction model based on the third training set and the third evaluation set to obtain a corresponding first feature evaluation score sequence;

[0070] wherein the first feature evaluation score sequence consists of the first feature evaluation scores of the total number of the first features;

[0071] Step 109, and identify whether each of the first feature evaluation scores in the first feature evaluation score sequence exceeds a preset feature score threshold; if so, use the current third dimensionality reduction model as the latest first dimensionality reduction model, and go to step 110; if not, delete the first key feature item in the first key feature item sequence whose first feature evaluation score does not exceed the feature score threshold, and return to step 106;

[0072] Step 110, solidifying the model parameters of the latest first dimensionality reduction model; and after the model parameters are solidified, confirming that the dimensionality reduction model training is completed.

[0073] Further preferably, performing model training and evaluation on each of the first training models based on the second training set and the second evaluation set to obtain a corresponding first evaluation score specifically includes:

[0074] Step 111, constructing a loss function of the first training model based on the specified loss function of the first target model and recording it as the corresponding second model loss function;

[0075] Step 112, taking the first second record of the second training set as the corresponding current training record;

[0076] Step 113, inputting the second training vector of the current training record into the second dimensionality reduction model of the first training model to perform feature dimensionality reduction processing and outputting a corresponding first dimensionality reduction vector; and inputting the first dimensionality reduction vector into the second target model of the first training model to perform model processing and using the obtained model output vector as the corresponding first prediction vector;

[0077] Step 114, inputting the first prediction vector and the second label vector of the current training record into the second model loss function to calculate and obtain a corresponding second loss value;

[0078] Step 115, identifying whether the second loss value satisfies a preset second loss value range; if the second loss value satisfies the second loss value range, identifying whether the current training record is the last second record of the second training set, if so, turning to step 116, if not, extracting the next second record of the second training set as the new current training record and returning to step 113; if the second loss value does not satisfy the second loss value range, performing a round of parameter optimization processing on the model parameters of the first training model in the direction of minimizing the second model loss function based on a preset second model optimizer, and returning to step 113 at the end of this round of parameter optimization processing;

[0079] Wherein, the second model optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0080] Step 116, perform a round of traversal on all the second records of the second evaluation set; and in this round of traversal, use the second record currently traversed as the corresponding current evaluation record; and input the second training vector of the current evaluation record into the second dimensionality reduction model of the first training model for feature dimensionality reduction processing and output the corresponding second dimensionality reduction vector; and input the second dimensionality reduction vector into the second target model of the first training model for model processing and use the obtained model output vector as the corresponding second prediction vector; and form a corresponding second prediction-label pair with the second prediction vector and the second label vector of the current training record; and end this round of traversal , identify the model function of the second target model of the first training model; if the current second target model is a regression calculation model, bring all the second prediction-label pairs obtained into the preset first model evaluation function to calculate and obtain the corresponding first evaluation value; if the current second target model is a classification model, construct a confusion matrix based on all the second prediction-label pairs obtained and record it as the corresponding second confusion matrix, and perform corresponding precision and recall calculations based on the second confusion matrix to obtain corresponding second precision and second recall, and perform F1 score calculation based on the second precision and the second recall to obtain the corresponding second F1 score;

[0081] Wherein, the first model evaluation function at least includes a standard deviation function and a RMSE function;

[0082] Step 117, identifying the model function of the second target model of the first training model; if the current second target model is a regression calculation model, then identifying whether the first evaluation value satisfies the preset first evaluation value range; if the first evaluation value does not satisfy the first evaluation value range, returning to step 112 to continue training; if the first evaluation value satisfies the first evaluation value range, the reciprocal of the first evaluation value is used as the corresponding first evaluation score and output; if the current second target model is a classification model, then identifying whether the second precision, the second recall rate and the second F1 score all satisfy the corresponding second precision range, the second recall rate range and the second F1 score range; if the second precision does not satisfy the corresponding second precision range or the second recall rate does not satisfy the corresponding second recall rate range or the second F1 score does not satisfy the corresponding second F1 score range, returning to step 112 to continue training; if the second precision, the second recall rate and the second F1 score all satisfy the corresponding second precision range, the second recall rate range and the second F1 score range, the second F1 score is used as the corresponding first evaluation score and output.

[0083] Further preferably, the principal component analysis method is used to identify the key feature items in the first feature item sequence according to the second training set and the second evaluation set to obtain the corresponding first key feature item sequence, which specifically includes:

[0084] Step 121, counting the total number of the second records in the second training set and the second evaluation set to obtain a corresponding first total number and a second total number; and summing the first and second total numbers to obtain a corresponding third total number Q;

[0085] Step 122, constructing a feature matrix A with a shape of Q×N according to all the second training vectors of the second training set and the second evaluation set;

[0086] The feature matrix A is composed of Q×N first matrix units, each row of the feature matrix A corresponds to one first feature item, and each column corresponds to one second training vector;

[0087] Step 123, based on the principal component analysis method, the covariance matrix of the characteristic matrix A is calculated to obtain a corresponding first covariance matrix; and the first covariance matrix is ​​subjected to eigendecomposition to obtain first principal component eigenvalues ​​and first principal component eigenvectors corresponding to each of the first characteristic items;

[0088] Step 124, extract Q first matrix units from each row of the feature matrix A to form a corresponding first feature vector; and form a corresponding first target tensor from all the second label vectors of the second training set and the second evaluation set; and calculate the Pearson correlation coefficient between each first feature vector and the first target tensor to obtain a corresponding first Pearson correlation coefficient;

[0089] Step 125, based on a preset robust standard deviation calculation method, the robust standard deviation of each of the first feature vectors is calculated to obtain a corresponding first robust standard deviation;

[0090] Wherein, the robust standard deviation calculation method includes MAD method, Qn method and Sn method;

[0091] Step 126, calculating the ratio coefficient of every two first feature items according to the first principal component eigenvalues, the first Pearson correlation coefficients and the first robust standard deviations corresponding to the first feature items to obtain the corresponding first ratio coefficients;

[0092] The calculation formula of the first ratio coefficient is:

[0093]

[0094] 1≤g、h≤Q,Rg,h is the ratio coefficient of the g-th first characteristic item to the h-th first characteristic item; λ g 、r g , S g is the first principal component eigenvalue, the first Pearson correlation coefficient and the first robust standard deviation corresponding to the g-th first feature item; h 、r h , S h is the first principal component eigenvalue, the first Pearson correlation coefficient and the first robust standard deviation corresponding to the hth first feature item; the first ratio coefficient R g,h If it is greater than a preset ratio coefficient threshold, it means that the g-th first feature item contributes more to the prediction of the first target model than the h-th first feature item, and the first ratio coefficient R g,h If it is less than or equal to the ratio coefficient threshold, it means that the contribution of the g-th first feature item to the prediction of the first target model is smaller than the contribution of the h-th first feature item;

[0095] Step 127, record each of the first feature items whose first principal component eigenvalues ​​exceed a preset principal component feature threshold as a corresponding first key feature item; and according to the first ratio coefficients of all the first key feature items, sort all the first key feature items obtained in order of their prediction contribution to the first target model from large to small to obtain a corresponding first key feature item sequence.

[0096] Further preferably, the principal component analysis method is used to reconstruct the second training set and the second evaluation set according to the first key feature item sequence to obtain the corresponding third training set and third evaluation set, which specifically includes:

[0097] The second training set or the third evaluation set is used as the corresponding current data set; and each second record of the current data set is recorded as the corresponding current record; and based on the principal component analysis method, the covariance matrix of the second training vector of the current record is calculated to obtain the corresponding second covariance matrix; and the second covariance matrix is ​​subjected to eigendecomposition to obtain the second principal component eigenvalues ​​and second principal component eigenvectors corresponding to each first feature item; and the second principal component eigenvectors corresponding to each first key feature item are sorted in the sorting order of the first key feature item sequence to obtain a corresponding third label vector; and the second label vector of the current record is used as a corresponding first target vector; and the third training vector corresponding to the current record, the third label vector and the first target vector form a corresponding third record; and all the third records corresponding to the current data set form the corresponding third training set or the third evaluation set.

[0098] Further preferably, the performing model training and evaluation on the third dimensionality reduction model based on the third training set and the third evaluation set to obtain a corresponding first feature evaluation score sequence specifically includes:

[0099] Step 141, taking the first third record of the third training set as the corresponding current training record;

[0100] Step 142, inputting the third training vector of the current training record into the third dimensionality reduction model to perform feature dimensionality reduction processing and outputting a corresponding third dimensionality reduction vector;

[0101] Step 143, inputting the third dimension reduction vector and the third label vector of the current training record into a preset third model loss function to calculate and obtain a corresponding third loss value;

[0102] Wherein, the third model loss function at least includes a mean square error loss function and a variance loss function;

[0103] Step 144, identifying whether the third loss value satisfies a preset third loss value range; if the third loss value satisfies the third loss value range, identifying whether the current training record is the last third record of the third training set, if so, turning to step 145, if not, extracting the next third record of the third training set as the new current training record and returning to step 142; if the third loss value does not satisfy the third loss value range, performing a round of parameter optimization processing on the model parameters of the third dimensionality reduction model in the direction of minimizing the third model loss function based on a preset third model optimizer, and returning to step 142 at the end of this round of parameter optimization processing;

[0104] Wherein, the third model optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0105] Step 145, perform a round of traversal on all the third records of the third evaluation set; and in this round of traversal, use the third record currently traversed as the corresponding current evaluation record; and input the third training vector of the current evaluation record into the third dimensionality reduction model for feature dimensionality reduction processing and output the corresponding fourth dimensionality reduction vector; and form a corresponding third prediction-label pair by the fourth dimensionality reduction vector and the third label vector of the current training record; and at the end of this round of traversal, bring all the obtained third prediction-label pairs into the preset second model evaluation function to calculate and obtain the corresponding second evaluation value;

[0106] Wherein, the second model evaluation function at least includes a standard deviation function and an RMSE function;

[0107] Step 146, identifying whether the second evaluation value meets a preset second evaluation value range, if the second evaluation value does not meet the second evaluation value range, returning to step 141 to continue training, if the second evaluation value meets the second evaluation value range, turning to step 147;

[0108] Step 147, counting the total number of the third records in the third evaluation set and taking the obtained total number as the corresponding sample total number U; recording each of the third records in the third evaluation set as a corresponding sample record; and constructing a feature matrix C with a shape of U×N based on the third training vectors of the U sample records; and constructing a feature matrix D with a shape of U×K based on the fourth dimensionality reduction vectors corresponding to the U sample records; and constructing a feature matrix E with a shape of U×M based on the first target vectors of the U sample records;

[0109] Wherein, the feature matrix C is composed of U×N second matrix units, each row of the feature matrix C corresponds to one of the first key feature items, and each column corresponds to one of the third training vectors of the sample record; the feature matrix D is composed of U×K third matrix units, each row of the feature matrix D corresponds to one of the first key feature items, and each column corresponds to one of the fourth reduced dimensionality vectors corresponding to the sample record; the feature matrix E is composed of U×M fourth matrix units, each row of the feature matrix E corresponds to one of the target model prediction feature items, and each column corresponds to one of the first target vectors of the sample record;

[0110] Step 148, extract U of the second matrix units in each row of the feature matrix C to form a corresponding first key feature vector; extract M of the fourth matrix units in each column of the feature matrix E to form a corresponding first sample target vector; and calculate the correlation measure between each of the first key feature vectors and each of the first sample target vectors to obtain a corresponding first correlation measure;

[0111] Step 149, confirming the principal component eigenvalues ​​corresponding to each of the first key feature items according to the feature matrix D to obtain the corresponding third principal component eigenvalues;

[0112] Step 150, extract the K third matrix units in each column of the feature matrix D to form a corresponding first sample dimensionality reduction vector; and extract the third matrix units corresponding to each of the first key feature items on each of the first sample dimensionality reduction vectors as the corresponding first dimensionality reduction features;

[0113] Step 151, performing key feature evaluation processing based on the first correlation measure, the third principal component eigenvalue and the first dimensionality reduction feature corresponding to each of the first key feature items to obtain the corresponding first feature evaluation score;

[0114] The calculation formula of the first feature evaluation score F is:

[0115]

[0116] 1≤sample index u≤U,w u is the weight coefficient corresponding to the u-th target vector of the first sample; r u is the first correlation measure between the first key feature vector corresponding to the current key feature item and the uth first sample target vector; α is a preset regularization coefficient; d u is the first dimensionality reduction feature corresponding to the current key feature item on the u-th dimensionality reduction vector of the first sample; λ is the third principal component eigenvalue corresponding to the current key feature item;

[0117] Step 152: sort all the obtained first feature evaluation scores according to the sorting order of the key feature items in the first key feature item sequence to form a corresponding first feature evaluation score sequence.

[0118] Preferably, performing feature dimensionality reduction processing on the first data set based on the first dimensionality reduction model to obtain a corresponding first optimized data set specifically includes:

[0119] Perform a round of traversal on all the first data records of the first data set; and in this round of traversal, use the first data record currently traversed as the corresponding current record; and based on the model input vector d of the first dimensionality reduction model bin The vector format is composed of N seventh feature data fields of the current record to form a model input vector recorded as the corresponding current input vector; and the current input vector is input into the first dimensionality reduction model for feature dimensionality reduction processing to obtain the corresponding current dimensionality reduction vector; and the K vector features of the current dimensionality reduction vector are extracted as the corresponding K ninth feature data fields; and the second data type field and M eighth feature data fields of the current record are used as the corresponding third data type field and M tenth feature data fields; and the third data type field corresponding to the current record, K ninth feature data fields and M tenth feature data fields form a corresponding second data record; and at the end of this round of traversal, all the obtained second data records form the corresponding first optimized data set.

[0120] A second aspect of an embodiment of the present invention provides a device for implementing the processing method of optimizing a data set as described in the first aspect above, the device comprising: a model preparation module, a data acquisition module, a data preprocessing module, a classification and duplication removal module, a balance enhancement module and a feature dimension reduction module;

[0121] The model preparation module is used to use the target training model of the data set as the corresponding first target model; and based on the output data features of the first target model, the maximum feature range of the model input data is confirmed to obtain the corresponding first feature item sequence; and a first classification model for classifying and identifying the input data is constructed according to the preset first data classification label set and the first feature item sequence; and a first dimension reduction model for performing feature dimension reduction processing on the input high-dimensional data is constructed according to the first feature item sequence; the first feature item sequence includes N first feature items, N is a positive integer greater than 1; the first data classification label set includes S first classification labels, S is a positive integer greater than 1;

[0122] The data acquisition module is used to perform big data acquisition based on the input / output data feature range of the first target model as the data acquisition requirement to obtain a corresponding first acquisition data set;

[0123] The data preprocessing module is used to preprocess the first acquired data set to obtain a corresponding first preprocessed data set;

[0124] The classification deduplication module is used to construct a classification model data set based on the first preprocessed data set to obtain a corresponding first training set and a first evaluation set; and to train the first classification model based on the first training set and the first evaluation set; and after the classification model training is completed, the first preprocessed data set is deduplicated based on the first classification model to obtain a corresponding first deduplicated data set;

[0125] The balancing and enhancing module is used to perform sample balancing and data enhancement processing on the first deduplication data set to obtain a corresponding first data set;

[0126] The feature dimension reduction module is used to construct a dimension reduction model data set based on the first data set to obtain a corresponding second training set and a second evaluation set; and to train the first dimension reduction model based on the second training set, the second evaluation set and the first target model; and after the dimension reduction model training is completed, the first data set is subjected to feature dimension reduction processing based on the first dimension reduction model to obtain a corresponding first optimized data set.

[0127] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0128] The processor is used to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0129] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0130] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0131] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for optimizing a data set. It can be seen from the above invention content that the embodiment of the present invention confirms the maximum feature range of the model input data based on the output data feature of the target model; and constructs a classification model for classifying and identifying the input data, and constructs a dimension reduction model for feature dimension reduction of the input high-dimensional data; and collects big data based on the input / output data feature range of the target model as the data collection requirement; and preprocesses the collected data set; and constructs a classification model data set based on the preprocessed data set to obtain a corresponding training / evaluation set, and trains the classification model based on the training / evaluation set; and after the classification model training is completed, the preprocessed data set is deduplicated based on the classification model; and the deduplicated data set is sample balanced and data enhanced to obtain a first data set; and the dimension reduction model data set is constructed based on the first data set to obtain a corresponding training / evaluation set, and the dimension reduction model is trained based on the training / evaluation set and the target model; and after the dimension reduction model training is completed, the first data set is subjected to feature dimension reduction processing based on the dimension reduction model to obtain an optimized data set. The embodiment of the present invention adds classification deduplication, classification balancing, and feature dimensionality reduction and other operation steps on the basis of the conventional optimization scheme. The embodiment of the present invention not only further improves the training efficiency of the model and enhances the generalization performance of the model, but also reduces the complexity of model training through dimensionality reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0132] Figure 1 A schematic diagram of a processing method for optimizing a data set provided in Embodiment 1 of the present invention;

[0133] Figure 2 A module structure diagram of a processing device for optimizing a data set provided in Embodiment 2 of the present invention;

[0134] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0135] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0136] A processing method for optimizing a data set is provided in a first embodiment of the present invention, such as Figure 1 A schematic diagram of a method for processing an optimized data set provided in the first embodiment of the present invention is shown, and includes the following main steps:

[0137] Step 1: Use the target training model of the data set as the corresponding first target model; confirm the maximum feature range of the model input data based on the output data features of the first target model to obtain the corresponding first feature item sequence; and construct a first classification model for classifying and identifying the input data according to the preset first data classification label set and the first feature item sequence; and construct a first dimensionality reduction model for performing feature dimensionality reduction processing on the input high-dimensional data according to the first feature item sequence;

[0138] The first feature item sequence includes N first feature items, where N is a positive integer greater than 1; the first data classification label set includes S first classification labels, where S is a positive integer greater than 1.

[0139] Here, the model input vector of the first target model of the embodiment of the present invention is recorded as X, and the model output vector is recorded as Y; the feature dimension of the model input vector X is adjustable and the maximum is N, that is, the model input vector X is composed of at most N input features x i , 1≤i≤N; the feature dimension of the model output vector Y remains unchanged, that is, the model output vector Y consists of M output features y j Composition, 1≤j≤M, M is a positive integer greater than 1; each output feature y j Corresponding to a target model prediction feature item.

[0140] The first classification model of the embodiment of the present invention is implemented based on a type of machine learning model or a deep learning model; the first classification model is used to ain Perform classification recognition processing and output the corresponding classification prediction vector d aout ; where the model input vector d ain The feature dimension is fixed to N; the classification prediction vector d aout The feature dimension of is fixed to S, which consists of S classification prediction probabilities; the classification prediction probabilities correspond one-to-one to the first classification labels.

[0141] The first dimensionality reduction model of the embodiment of the present invention is implemented based on a type of machine learning model or a deep learning model; the first dimensionality reduction model is used to transform the model input vector d bin Perform feature dimensionality reduction processing and output the corresponding dimensionality reduction vector d bout ; Model input vector d bin The feature dimension is fixed to N; the dimension reduction vector d bout The feature dimension K is adjustable, 1≤K <N。

[0142] Step 2, performing big data collection based on the input / output data feature range of the first target model as the data collection requirement to obtain a corresponding first collection data set;

[0143] The first acquisition data set includes a plurality of first acquisition data records; the first acquisition data record includes N first feature data fields and M second feature data fields; the first feature data fields correspond to the first feature items one by one; the second feature data fields correspond to the target model prediction feature items one by one;

[0144] Specifically comprising: step 21, forming a corresponding collection object attribute set from N first feature items and M target model prediction feature items;

[0145] Step 22, through multiple big data collection channels, big data collection is performed with the collection object attribute set as the attribute requirement of a single data object to obtain multiple collection data objects;

[0146] The multiple big data collection channels include at least government / institutional data channels, commercial databases, social media Internet and third-party data service companies; each collected data object includes N first attribute data and M second attribute data, the first attribute data corresponds to the first feature item one by one; the second attribute data corresponds to the target model prediction feature item one by one;

[0147] Step 23, and use each collected data object as the corresponding current data object; and use each first attribute data of the current data object as a corresponding first characteristic data field, and use each second attribute data of the current data object as a corresponding second characteristic data field, and use the N third characteristic data fields and M fourth characteristic data fields corresponding to the current data object to form a corresponding first collected data record;

[0148] Step 24: All the first collected data records are used to form a corresponding first collected data set.

[0149] Step 3, preprocessing the first acquired data set to obtain a corresponding first preprocessed data set;

[0150] The first preprocessed data set includes a plurality of first preprocessed data records; the first preprocessed data record includes N third characteristic data fields and M fourth characteristic data fields; the third characteristic data fields correspond one-to-one to the first characteristic items; the fourth characteristic data fields correspond one-to-one to the target model prediction characteristic items;

[0151] Specifically, it includes: step 31, pre-setting a corresponding feature item data rationality inspection standard and a corresponding feature item data standardization conversion rule for the N first feature items and the M target model prediction feature items respectively;

[0152] Step 32, and perform a round of traversal of all first collected data records of the first collected data set; and in this round of traversal, take the first collected data record currently traversed as the corresponding current data record; and perform a round of polling of all characteristic data fields of the current data record; and in this round of polling, record the first or second characteristic data field currently polled as the corresponding current field; and identify whether the current field is empty; if the current field is empty, the current data record is recorded as an abnormal record; if the current field is not empty, then the rationality test of the current field is performed based on the corresponding characteristic item data rationality test standard to obtain the corresponding first test result, and when the first test result is reasonable, the field data of the current field is standardized and converted based on the corresponding characteristic item data standardization conversion rule and the conversion result is used as a corresponding third or fourth characteristic data field, and when the first test result is unreasonable, the current data record is recorded as an abnormal record; and at the end of polling of all characteristic data fields of the current data record, it is identified whether the current data record is recorded as an abnormal record, if not, a corresponding first pre-processed data record is formed based on the N third characteristic data fields and M fourth characteristic data fields corresponding to the current data record;

[0153] Among them, the first test results include reasonable and unreasonable;

[0154] Step 33: When the traversal of all first collected data records of the first collected data set is completed, all the obtained first pre-processed data records are used to form a corresponding first pre-processed data set.

[0155] Step 4: construct a classification model data set based on the first preprocessed data set to obtain a corresponding first training set and a first evaluation set; and train the first classification model based on the first training set and the first evaluation set; and after the classification model training is completed, perform similar data deduplication processing on the first preprocessed data set based on the first classification model to obtain a corresponding first deduplication data set;

[0156] Specifically, it includes: step 41, constructing a classification model data set based on the first preprocessed data set to obtain a corresponding first training set and a first evaluation set;

[0157] Specifically, it includes: step 411, pre-setting a corresponding data classification rule for each first classification label of the first data classification label set; and based on a manual or machine data sorting method, randomly sorting a group of a specified number of first pre-processed data records from the first pre-processed data set according to each data classification rule to form a corresponding first category record set;

[0158] Step 412, each obtained first-category record set is recorded as the corresponding current set; the first classification label corresponding to the current set is used as the corresponding current classification label; all first pre-processed data records of the current set are traversed once; and during this traversal, the first pre-processed data record currently traversed is used as the corresponding current record; and based on the model input vector d ain The vector format of the model is composed of the N third feature data fields of the current record, forming a model input vector recorded as the corresponding first training vector; and based on the classification prediction vector d aout A classification prediction vector with a length of S and all classification prediction probabilities being zero is set as the corresponding first label vector, and the classification prediction probability corresponding to the current classification label in the first label vector is reset to 1; and a corresponding first record is formed by the first training vector and the first label vector corresponding to the current record; and at the end of this round of traversal, all first records corresponding to the current set form a corresponding first record set; and the first record set is divided into two subsets according to a preset first division ratio and recorded as the corresponding first subset and second subset;

[0159] The first split ratio is a preset ratio value, such as 8:2; both the first and second subsets are composed of a plurality of first records; the ratio of the total number of records in the first subset to the total number of records in the second subset satisfies the first split ratio;

[0160] Step 413, the obtained S first subsets form a corresponding first training set; and the obtained S second subsets form a corresponding first evaluation set;

[0161] Wherein, the first training set and the first evaluation set are both composed of a plurality of first records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies a first segmentation ratio; the first record includes a first training vector and a first label vector;

[0162] Step 42, training a first classification model based on the first training set and the first evaluation set;

[0163] Specifically, it includes: step 421, taking the first first record of the first training set as the corresponding current training record;

[0164] Step 422, inputting the first training vector of the current training record into the first classification model for classification recognition processing to obtain a corresponding first classification prediction vector;

[0165] Step 423, inputting the first classification prediction vector and the first label vector of the current training record into a preset first model loss function to calculate and obtain a corresponding first loss value;

[0166] Wherein, the first model loss function at least includes an L1 loss function, an L2 loss function and a cross entropy loss function;

[0167] Step 424, identifying whether the first loss value satisfies a preset first loss value range; if the first loss value satisfies the first loss value range, identifying whether the current training record is the last first record of the first training set, if so, turning to step 425, if not, extracting the next first record of the first training set as a new current training record and returning to step 422; if the first loss value does not satisfy the first loss value range, performing a round of parameter optimization processing on the model parameters of the first classification model in the direction of minimizing the first model loss function based on a preset first model optimizer, and returning to step 422 at the end of this round of parameter optimization processing;

[0168] The first loss value range is a preset loss value range; the first model optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0169] Step 425, perform a round of traversal on all first records of the first evaluation set; and in this round of traversal, use the first record currently traversed as the corresponding current evaluation record; and input the first training vector of the current evaluation record into the first classification model for classification and recognition processing to obtain the corresponding second classification prediction vector; and form a corresponding first prediction-label pair by the second classification prediction vector and the first label vector of the current training record; and at the end of this round of traversal, construct a confusion matrix based on all the first prediction-label pairs obtained and record it as the corresponding first confusion matrix; and perform corresponding precision and recall calculations based on the first confusion matrix to obtain the corresponding first precision and first recall; and perform F1 score calculation based on the first precision and first recall to obtain the corresponding first F1 score;

[0170] Step 426, identifying whether the first precision, the first recall and the first F1 score all satisfy the corresponding first precision range, the first recall range and the first F1 score range; if the first precision does not satisfy the corresponding first precision range, or the first recall does not satisfy the corresponding first recall range, or the first F1 score does not satisfy the corresponding first F1 score range, returning to step 421 to continue training; if the first precision, the first recall and the first F1 score all satisfy the corresponding first precision range, the first recall range and the first F1 score range, stopping training and confirming that the classification model training is finished;

[0171] Here, the first precision range, the first recall range, and the first F1 score range are three preset value ranges;

[0172] Step 43: After the classification model training is completed, the first preprocessed data set is subjected to similar data deduplication processing based on the first classification model to obtain a corresponding first deduplication data set;

[0173] The first deduplication data set includes a plurality of first deduplication data records; the first deduplication data record includes a first data type field, N fifth characteristic data fields, and M sixth characteristic data fields; the first data type field corresponds to the first classification label one by one; the fifth characteristic data field corresponds to the first characteristic item one by one; the sixth characteristic data field corresponds to the target model prediction characteristic item one by one;

[0174] Specifically, step 431 includes: performing a round of traversal on all first preprocessed data records of the first preprocessed data set; and in this round of traversal, recording the first preprocessed data record currently traversed as the corresponding current record; and based on the model input vector d ain The vector format is composed of N third characteristic data fields of the current record to form a model input vector and record it as the corresponding first input vector; the first input vector is input into the first classification model for classification recognition processing to obtain the corresponding third classification prediction vector; and the first classification label corresponding to the classification prediction probability with the largest probability value in the third classification prediction vector is taken as the corresponding first data type; and at the end of this round of traversal, multiple first pre-processed data records with the same first data type are extracted and clustered into one category to form a corresponding second category record set;

[0175] Step 432, after the traversal of all the first preprocessed data records of the first preprocessed data set is completed, a round of traversal is performed on all the second-category record sets obtained; and in this round of traversal, the currently traversed second-category record set is recorded as the corresponding current set; and based on the model input vector d ain The vector format is composed of N third characteristic data fields of each first preprocessed data record of the current set to form a corresponding first data vector; and the vector similarity of each two first data vectors in the current set is calculated to obtain the corresponding first similarity; and the first similarity whose similarity exceeds the preset first similarity threshold is recorded as the corresponding second similarity; and one of the two first preprocessed data records corresponding to each second similarity is selected and recorded as the corresponding duplicate record; and all duplicate records in the current set are deleted;

[0176] Here, the embodiment of the present invention adopts a conventional similarity algorithm to calculate the vector similarity, such as a cosine similarity algorithm, etc.; the first similarity threshold is a preset similarity threshold;

[0177] Step 433: After the traversal of all the second-category record sets is completed, all the second-category record sets are merged to form a corresponding first merged set;

[0178] Step 434, and perform a round of traversal on all the first preprocessed data records of the first merged set; and in this round of traversal, record the first preprocessed data record currently traversed as the corresponding current record; and use the first data type, N third characteristic data fields and M fourth characteristic data fields corresponding to the current record as the corresponding first data type fields, N fifth characteristic data fields and M sixth characteristic data fields to form a corresponding first deduplicated data record; and at the end of this round of traversal, all the first deduplicated data records obtained form a corresponding first deduplicated data set.

[0179] Step 5, performing sample balancing and data enhancement processing on the first deduplication data set to obtain a corresponding first data set;

[0180] The first data set includes a plurality of first data records; the first data record includes a second data type field, N seventh feature data fields and M eighth feature data fields; the second data type field corresponds one-to-one to the first classification label; the seventh feature data field corresponds one-to-one to the first feature item; the eighth feature data field corresponds one-to-one to the target model prediction feature item;

[0181] Specifically, it includes: step 51, extracting a plurality of first deduplication data records with the same first data type field from the first deduplication data set and grouping them into one category to form a corresponding third category record set;

[0182] Step 52, counting the total number of records of each third-category record set to obtain the corresponding first total number of records; and calculating the mean of all the obtained first total number of records to obtain the corresponding first average total number; and calculating the corresponding first ratio = first total number of records / first average total number based on each first total number of records and the first average total number; and recording the third-category record set whose first ratio is less than a preset first ratio threshold as the corresponding under-sampled set, and recording the third-category record set whose first ratio is greater than a preset second ratio threshold as the corresponding over-sampled set;

[0183] The first and second ratio thresholds are two preset ratio values, and 0<first ratio threshold<1<second ratio threshold;

[0184] Step 53, identifying the total number of under-sampled sets and over-sampled sets; if the total number of under-sampled sets is not 0 or the total number of over-sampled sets is not 0, then when the total number of under-sampled sets is not 0, using the over-sampling method to perform record addition processing on each under-sampled set, and when the total number of over-sampled sets is not 0, using the under-sampling method to perform record deletion processing on each over-sampled set, and after the record addition / deletion processing is completed, return to step 52; if the total number of under-sampled sets and the total number of over-sampled sets are both 0, go to step 54;

[0185] Step 54, merge all the third-category record sets to obtain the corresponding second merged set; and perform a round of traversal on all the first deduplicated data records of the second merged set; and in this round of traversal, record the first deduplicated data record currently traversed as the corresponding current record; and use the first data type field, N fifth characteristic data fields and M sixth characteristic data fields corresponding to the current record as the corresponding second data type field, N seventh characteristic data fields and M eighth characteristic data fields to form a corresponding first data record; and at the end of this round of traversal, all the first data records obtained form the corresponding first data set.

[0186] Step 6: construct a dimension reduction model data set based on the first data set to obtain a corresponding second training set and a second evaluation set; and train the first dimension reduction model based on the second training set, the second evaluation set and the first target model; and after the dimension reduction model training is completed, perform feature dimension reduction processing on the first data set based on the first dimension reduction model to obtain a corresponding first optimized data set;

[0187] Specifically, it includes: step 61, constructing a dimension reduction model data set based on the first data set to obtain a corresponding second training set and a second evaluation set;

[0188] Specifically, it includes: step 611, extracting multiple first data records with the same second data type field in the first data set and grouping them into one category to form a corresponding fourth category record set; randomly selecting a specified number of first data records from each fourth category record set to form a corresponding second record set; and merging all the obtained second record sets to obtain a corresponding third merged set;

[0189] Step 612, perform a round of traversal on all first data records of the third merged set; and in this round of traversal, record the first data record currently traversed as the corresponding current record; and based on the model input vector d of the first dimensionality reduction model bin The vector format of the model output vector is composed of the N seventh characteristic data fields of the current record, which is recorded as the corresponding second training vector; and the vector format of the model output vector based on the first target model is composed of the M eighth characteristic data fields of the current record, which is recorded as the corresponding second label vector; and the second training vector and the second label vector corresponding to the current record are composed of a corresponding second record; and at the end of this round of traversal, all the obtained second records are composed of a corresponding second record sequence;

[0190] Step 613: dividing the second record sequence into two subsequences according to a preset second division ratio and recording them as a corresponding first subsequence and a second subsequence;

[0191] The second split ratio is a preset ratio value, such as 8:2; both the first and second subsequences are composed of a plurality of second records; the ratio of the total number of records in the first subsequence to the total number of records in the second subsequence satisfies the second split ratio;

[0192] Step 614, and use the first subsequence as the corresponding second training set, and use the second subsequence as the corresponding second evaluation set;

[0193] Wherein, the second training set and the second evaluation set are both composed of a plurality of second records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio; the second record includes a second training vector and a second label vector;

[0194] Step 62, training the first dimensionality reduction model based on the second training set, the second evaluation set and the first target model;

[0195] Specifically, the method includes: step 621, identifying a preset feature dimension reduction processing mode; if the feature dimension reduction processing mode is the first mode, then going to step 622; if the feature dimension reduction processing mode is the second mode, then going to step 625;

[0196] The feature dimension reduction processing mode includes a first mode and a second mode;

[0197] Step 622, according to the preset k value selection rule, multiple integer values ​​are selected from the value range of the feature dimension K of the first dimensionality reduction model to form a corresponding k value range; and based on each k value in the k value range, the feature dimension K of the first dimensionality reduction model is set to obtain a corresponding second dimensionality reduction model; and based on the feature dimension K of each second dimensionality reduction model, the feature dimension of the model input vector X of the first target model is set to obtain a corresponding second target model; and each second dimensionality reduction model is connected with the corresponding second target model to form a corresponding first training model;

[0198] Wherein, the k value range includes multiple k values;

[0199] Here, the k value selection rule of the embodiment of the present invention can be customized based on application requirements; for example, within the value range of the feature dimension K, an odd number greater than 3 is selected as a k value to form a corresponding k value range; for another example, within the value range of the feature dimension K, an even number greater than 2 is selected as a k value to form a corresponding k value range;

[0200] Step 623, performing model training and evaluation on each first training model based on the second training set and the second evaluation set to obtain a corresponding first evaluation score;

[0201] Specifically comprising: step 6231, constructing a loss function of the first training model based on the specified loss function of the first target model and recording it as the corresponding second model loss function;

[0202] Step 6232, taking the first second record of the second training set as the corresponding current training record;

[0203] Step 6233, input the second training vector of the current training record into the second dimensionality reduction model of the first training model for feature dimensionality reduction processing and output the corresponding first dimensionality reduction vector; and input the first dimensionality reduction vector into the second target model of the first training model for model processing and use the obtained model output vector as the corresponding first prediction vector;

[0204] Step 6234, input the first prediction vector and the second label vector of the current training record into the second model loss function to calculate and obtain a corresponding second loss value;

[0205] Step 6235, identifying whether the second loss value satisfies a preset second loss value range; if the second loss value satisfies the second loss value range, identifying whether the current training record is the last second record of the second training set, if so, turning to step 6236, if not, extracting the next second record of the second training set as the new current training record and returning to step 6233; if the second loss value does not satisfy the second loss value range, performing a round of parameter optimization processing on the model parameters of the first training model in the direction of minimizing the second model loss function based on the preset second model optimizer, and returning to step 6233 at the end of this round of parameter optimization processing;

[0206] The second loss value range is a preset loss value range; the second model optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0207] Step 6236, perform a round of traversal on all second records of the second evaluation set; and in this round of traversal, use the second record currently traversed as the corresponding current evaluation record; and input the second training vector of the current evaluation record into the second dimensionality reduction model of the first training model for feature dimensionality reduction processing and output the corresponding second dimensionality reduction vector; and input the second dimensionality reduction vector into the second target model of the first training model for model processing and use the obtained model output vector as the corresponding second prediction vector; and form a corresponding second prediction-label pair by the second prediction vector and the second label vector of the current training record; and at the end of this round of traversal, identify the model function of the second target model of the first training model; if the current second target model is a regression calculation model, then bring all the obtained second prediction-label pairs into the preset first model evaluation function to calculate and obtain the corresponding first evaluation value; if the current second target model is a classification model, then construct a confusion matrix based on all the obtained second prediction-label pairs and record it as the corresponding second confusion matrix, and perform corresponding precision and recall calculations based on the second confusion matrix to obtain the corresponding second precision and second recall, and perform F1 score calculations based on the second precision and second recall to obtain the corresponding second F1 score;

[0208] Wherein, the first model evaluation function includes at least a standard deviation function and a RMSE function;

[0209] Step 6237, identifying the model function of the second target model of the first training model; if the current second target model is a regression calculation model, then identifying whether the first evaluation value satisfies the preset first evaluation value range, if the first evaluation value does not satisfy the first evaluation value range, returning to step 6232 to continue training, if the first evaluation value satisfies the first evaluation value range, taking the inverse of the first evaluation value as the corresponding first evaluation score and outputting it; if the current second target model is a classification model, then identifying whether the second precision, the second recall rate and the second F1 score all satisfy the corresponding second precision range, the second recall rate range and the second F1 score range, if the second precision does not satisfy the corresponding second precision range or the second recall rate does not satisfy the corresponding second recall rate range or the second F1 score does not satisfy the corresponding second F1 score range, returning to step 6232 to continue training, if the second precision, the second recall rate and the second F1 score all satisfy the corresponding second precision range, the second recall rate range and the second F1 score range, taking the second F1 score as the corresponding first evaluation score and outputting it;

[0210] Here, the first evaluation value range is a preset evaluation value range;

[0211] Step 624, and use the second dimensionality reduction model and the second target model corresponding to the first training model with the highest first evaluation score as the latest first dimensionality reduction model and the first target model; and go to step 630;

[0212] Step 625, identifying key feature items in the first feature item sequence based on the second training set and the second evaluation set based on a principal component analysis (PCA) method to obtain a corresponding first key feature item sequence;

[0213] Wherein, the first key feature item sequence includes a plurality of first key feature items;

[0214] Specifically, it includes: step 6251, counting the total number of the second records of the second training set and the second evaluation set to obtain a corresponding first total number and a second total number; and summing the first and second total numbers to obtain a corresponding third total number Q;

[0215] Step 6252, and construct a feature matrix A with a shape of Q×N based on all second training vectors of the second training set and the second evaluation set;

[0216] The feature matrix A is composed of Q×N first matrix units, each row of the feature matrix A corresponds to a first feature item, and each column corresponds to a second training vector;

[0217] Step 6253, based on the principal component analysis method, the covariance matrix of the characteristic matrix A is calculated to obtain the corresponding first covariance matrix; and the first covariance matrix is ​​subjected to eigendecomposition to obtain the first principal component eigenvalues ​​and the first principal component eigenvectors corresponding to the first characteristic items;

[0218] Step 6254, extract Q first matrix units from each row of the feature matrix A to form a corresponding first eigenvector; and form a corresponding first target tensor from all second label vectors of the second training set and the second evaluation set; and calculate the Pearson correlation coefficient between each first eigenvector and the first target tensor to obtain the corresponding first Pearson correlation coefficient;

[0219] Step 6255, based on a preset robust standard deviation calculation method, the robust standard deviation of each first eigenvector is calculated to obtain a corresponding first robust standard deviation;

[0220] Among them, the robust standard deviation calculation methods include MAD method, Qn method, and Sn method;

[0221] Step 6256, and according to the first principal component eigenvalue, the first Pearson correlation coefficient and the first robust standard deviation corresponding to each first characteristic item, the ratio coefficient of every two first characteristic items is calculated to obtain the corresponding first ratio coefficient;

[0222] Among them, the calculation formula of the first ratio coefficient is:

[0223]

[0224] 1≤g、h≤Q,R g,h is the ratio coefficient of the g-th first characteristic item to the h-th first characteristic item; g 、r g , S g is the first principal component eigenvalue, first Pearson correlation coefficient and first robust standard deviation corresponding to the g-th first characteristic item; h 、r h , S h is the first principal component eigenvalue, first Pearson correlation coefficient and first robust standard deviation corresponding to the hth first characteristic item;

[0225] Here, the first ratio coefficient R g,h If it is greater than the preset ratio coefficient threshold, it means that the contribution of the g-th first feature item to the prediction of the first target model is greater than that of the h-th first feature item. The first ratio coefficient R g,h If it is less than or equal to the ratio coefficient threshold, it means that the contribution of the g-th first feature item to the prediction of the first target model is smaller than that of the h-th first feature item; the ratio coefficient threshold is a preset threshold parameter;

[0226] Step 6257, each first feature item whose first principal component eigenvalue exceeds a preset principal component feature threshold is recorded as a corresponding first key feature item; and according to the first ratio coefficients of all first key feature items, all the first key feature items are sorted in descending order of their contribution to the prediction of the first target model to obtain a corresponding first key feature item sequence;

[0227] Here, the principal component feature threshold is a pre-set threshold parameter;

[0228] Step 626, the total number of feature items of the first key feature item sequence is counted to obtain a corresponding first feature total number; and based on the first feature total number, the feature dimension K of the first dimensionality reduction model is set to obtain a corresponding third dimensionality reduction model;

[0229] Step 627, reconstructing the second training set and the second evaluation set according to the first key feature item sequence based on the principal component analysis method to obtain a corresponding third training set and a third evaluation set;

[0230] Wherein, the third training set and the third evaluation set are both composed of a plurality of third records; the ratio of the total number of records of the third training set to the third evaluation set satisfies the second segmentation ratio; the third record includes a third training vector, a third label vector and a first target vector;

[0231] Specifically, it includes: taking the second training set or the third evaluation set as the corresponding current data set; and recording each second record of the current data set as the corresponding current record; and based on the principal component analysis method, calculating the covariance matrix of the second training vector of the current record to obtain the corresponding second covariance matrix; and performing eigendecomposition on the second covariance matrix to obtain the second principal component eigenvalues ​​and second principal component eigenvectors corresponding to each first characteristic item; and sorting the second principal component eigenvectors corresponding to each first key characteristic item according to the sorting order of the first key characteristic item sequence to obtain a corresponding third label vector; and taking the second label vector of the current record as a corresponding first target vector; and forming a corresponding third record by the third training vector, the third label vector and the first target vector corresponding to the current record; and forming a corresponding third training set or a third evaluation set by all the third records corresponding to the current data set;

[0232] Step 628, performing model training and evaluation on the third dimensionality reduction model based on the third training set and the third evaluation set to obtain a corresponding first feature evaluation score sequence;

[0233] Wherein, the first feature evaluation score sequence consists of the first feature evaluation scores of the total number of first features;

[0234] Specifically, it includes: step 628-1, taking the first third record of the third training set as the corresponding current training record;

[0235] Step 628-2, inputting the third training vector of the current training record into the third dimensionality reduction model to perform feature dimensionality reduction processing and outputting the corresponding third dimensionality reduction vector;

[0236] Step 628-3, inputting the third dimension reduction vector and the third label vector of the current training record into a preset third model loss function to calculate and obtain a corresponding third loss value;

[0237] Wherein, the third model loss function at least includes a mean square error loss function and a variance loss function;

[0238] Step 628-4, identifying whether the third loss value satisfies a preset third loss value range; if the third loss value satisfies the third loss value range, identifying whether the current training record is the last third record of the third training set, if so, turning to step 628-5, if not, extracting the next third record of the third training set as the new current training record and returning to step 628-2; if the third loss value does not satisfy the third loss value range, performing a round of parameter optimization processing on the model parameters of the third dimensionality reduction model in the direction of minimizing the third model loss function based on a preset third model optimizer, and returning to step 628-2 at the end of this round of parameter optimization processing;

[0239] The third loss value range is a preset loss value range; the third model optimizer includes at least an SGD optimizer and an ADAM optimizer;

[0240] Step 628-5, perform a round of traversal on all third records of the third evaluation set; and in this round of traversal, use the third record currently traversed as the corresponding current evaluation record; and input the third training vector of the current evaluation record into the third dimensionality reduction model for feature dimensionality reduction processing and output the corresponding fourth dimensionality reduction vector; and form a corresponding third prediction-label pair by the fourth dimensionality reduction vector and the third label vector of the current training record; and at the end of this round of traversal, bring all the obtained third prediction-label pairs into the preset second model evaluation function to calculate and obtain the corresponding second evaluation value;

[0241] Wherein, the second model evaluation function includes at least a standard deviation function and a RMSE function;

[0242] Step 628-6, identifying whether the second evaluation value meets the preset second evaluation value range, if the second evaluation value does not meet the second evaluation value range, returning to step 628-1 to continue training, if the second evaluation value meets the second evaluation value range, turning to step 628-7;

[0243] Here, the second evaluation value range is a preset evaluation value range;

[0244] Step 628-7, count the total number of the third records in the third evaluation set and use the obtained total number as the corresponding sample total number U; record each third record in the third evaluation set as a corresponding sample record; and construct a feature matrix C with a shape of U×N based on the third training vectors of the U sample records; and construct a feature matrix D with a shape of U×K based on the fourth dimensionality reduction vectors corresponding to the U sample records; and construct a feature matrix E with a shape of U×M based on the first target vectors of the U sample records;

[0245] Among them, the feature matrix C is composed of U×N second matrix units, each row of the feature matrix C corresponds to a first key feature item, and each column corresponds to a third training vector of a sample record; the feature matrix D is composed of U×K third matrix units, each row of the feature matrix D corresponds to a first key feature item, and each column corresponds to a fourth dimensionality reduction vector corresponding to a sample record; the feature matrix E is composed of U×M fourth matrix units, each row of the feature matrix E corresponds to a target model prediction feature item, and each column corresponds to a first target vector of a sample record;

[0246] Step 628-8, extract U second matrix units from each row of the feature matrix C to form a corresponding first key feature vector; extract M fourth matrix units from each column of the feature matrix E to form a corresponding first sample target vector; and calculate the correlation measure between each first key feature vector and each first sample target vector to obtain a corresponding first correlation measure;

[0247] Step 628-9, confirming the principal component eigenvalues ​​corresponding to each first key feature item according to the feature matrix D to obtain the corresponding third principal component eigenvalues;

[0248] Step 628-10, extract K third matrix units from each column of the feature matrix D to form a corresponding first sample dimensionality reduction vector; and extract the third matrix units corresponding to each first key feature item on each first sample dimensionality reduction vector as the corresponding first dimensionality reduction feature;

[0249] Step 628-11, performing key feature evaluation processing based on the first correlation measure, the third principal component eigenvalue and the first dimensionality reduction feature corresponding to each first key feature item to obtain a corresponding first feature evaluation score;

[0250] Among them, the calculation formula of the first feature evaluation score F is:

[0251]

[0252] 1≤sample index u≤U,w u is the weight coefficient corresponding to the u-th first sample target vector; r u is the first correlation measure between the first key feature vector corresponding to the current key feature item and the uth first sample target vector; α is a preset regularization coefficient; d u is the first dimension reduction feature corresponding to the current key feature item on the u-th first sample dimension reduction vector; λ is the third principal component eigenvalue corresponding to the current key feature item;

[0253] Step 628-12, sorting all obtained first feature evaluation scores according to the sorting order of the key feature items in the first key feature item sequence to form a corresponding first feature evaluation score sequence;

[0254] Step 629, and identify whether each first feature evaluation score of the first feature evaluation score sequence exceeds the preset feature score threshold; if so, the current third dimensionality reduction model is used as the latest first dimensionality reduction model, and the process goes to step 630; if not, the first key feature item in the first key feature item sequence whose first feature evaluation score does not exceed the feature score threshold is deleted, and the process returns to step 626;

[0255] Here, the feature scoring threshold is a pre-set score parameter;

[0256] Step 630, solidifying the model parameters of the latest first dimensionality reduction model; and after the model parameters are solidified, confirming that the dimensionality reduction model training is completed;

[0257] Step 63: After the dimensionality reduction model training is completed, feature dimensionality reduction processing is performed on the first data set based on the first dimensionality reduction model to obtain a corresponding first optimized data set;

[0258] The first optimized data set includes a plurality of second data records; the second data record includes a third data type field, K ninth feature data fields, and M tenth feature data fields; the second data type field corresponds one-to-one to the first classification label; the tenth feature data field corresponds one-to-one to the target model prediction feature item;

[0259] Specifically, it includes: performing a round of traversal on all first data records of the first data set; and in this round of traversal, taking the first data record currently traversed as the corresponding current record; and based on the model input vector d of the first dimensionality reduction model bin The vector format is composed of N seventh feature data fields of the current record to form a model input vector recorded as the corresponding current input vector; and the current input vector is input into the first dimensionality reduction model for feature dimensionality reduction processing to obtain the corresponding current dimensionality reduction vector; and the K vector features of the current dimensionality reduction vector are extracted as the corresponding K ninth feature data fields; and the second data type field and M eighth feature data fields of the current record are used as the corresponding third data type field and M tenth feature data fields; and the third data type field, K ninth feature data fields and M tenth feature data fields corresponding to the current record form a corresponding second data record; and at the end of this round of traversal, all the obtained second data records form the corresponding first optimized data set.

[0260] Figure 2This is a module structure diagram of a processing device for optimizing a data set provided in Embodiment 2 of the present invention. The device is a terminal device or server that implements the aforementioned method embodiment, or can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the processing device for optimizing the data set provided in the second embodiment of the present invention includes: a model preparation module 201, a data acquisition module 202, a data preprocessing module 203, a classification and deduplication module 204, a balance enhancement module 205 and a feature dimension reduction module 206.

[0261] The model preparation module 201 is used to use the target training model of the data set as the corresponding first target model; and based on the output data features of the first target model, the maximum feature range of the model input data is confirmed to obtain the corresponding first feature item sequence; and according to the preset first data classification label set and the first feature item sequence, a first classification model for classifying and identifying the input data is constructed; and according to the first feature item sequence, a first dimension reduction model for performing feature dimension reduction processing on the input high-dimensional data is constructed; the first feature item sequence includes N first feature items, N is a positive integer greater than 1; the first data classification label set includes S first classification labels, S is a positive integer greater than 1.

[0262] The data collection module 202 is used to collect big data based on the input / output data feature range of the first target model as the data collection requirement to obtain a corresponding first collection data set.

[0263] The data preprocessing module 203 is used to preprocess the first acquired data set to obtain a corresponding first preprocessed data set.

[0264] The classification deduplication module 204 is used to construct a classification model data set based on the first preprocessed data set to obtain a corresponding first training set and a first evaluation set; and to train the first classification model based on the first training set and the first evaluation set; and after the classification model training is completed, the first preprocessed data set is deduplicated based on the first classification model to obtain a corresponding first deduplicated data set.

[0265] The balancing and enhancing module 205 is used to perform sample balancing and data enhancement processing on the first deduplication data set to obtain a corresponding first data set.

[0266] The feature dimensionality reduction module 206 is used to construct a dimensionality reduction model data set based on the first data set to obtain a corresponding second training set and a second evaluation set; and to train the first dimensionality reduction model based on the second training set, the second evaluation set and the first target model; and after the dimensionality reduction model training is completed, perform feature dimensionality reduction processing on the first data set based on the first dimensionality reduction model to obtain a corresponding first optimized data set.

[0267] An embodiment of the present invention provides a processing device for optimizing a data set, which can execute the method steps in the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0268] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the model preparation module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0269] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a certain module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0270] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state drive (SSD)), etc.

[0271] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server for implementing the method of the aforementioned embodiment, or may be a terminal device or a server for implementing the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 3 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303. Various instructions may be stored in the memory 302 to complete various processing functions and implement the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention also includes: a power supply 304, a system bus 305 and a communication port 306. The system bus 305 is used to realize the communication connection between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0272] exist Figure 3The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as a client, a read-write library, and a read-only library). The memory may include a random access memory (RAM), and may also include a non-volatile memory (Non-Vol at i le Memory), such as at least one disk storage.

[0273] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (Network Processor, NP), a graphics processing unit (Graphics Processing Unit, GPU), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0274] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method and processing process provided in the above embodiments.

[0275] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for optimizing a data set. It can be seen from the above invention content that the embodiment of the present invention confirms the maximum feature range of the model input data based on the output data feature of the target model; and constructs a classification model for classifying and identifying the input data, and constructs a dimension reduction model for feature dimension reduction of the input high-dimensional data; and collects big data based on the input / output data feature range of the target model as the data collection requirement; and preprocesses the collected data set; and constructs a classification model data set based on the preprocessed data set to obtain a corresponding training / evaluation set, and trains the classification model based on the training / evaluation set; and after the classification model training is completed, the preprocessed data set is deduplicated based on the classification model; and the deduplicated data set is sample balanced and data enhanced to obtain a first data set; and the dimension reduction model data set is constructed based on the first data set to obtain a corresponding training / evaluation set, and the dimension reduction model is trained based on the training / evaluation set and the target model; and after the dimension reduction model training is completed, the first data set is subjected to feature dimension reduction processing based on the dimension reduction model to obtain an optimized data set. The embodiment of the present invention adds classification deduplication, classification balancing, and feature dimensionality reduction and other operation steps on the basis of the conventional optimization scheme. The embodiment of the present invention not only further improves the training efficiency of the model and enhances the generalization performance of the model, but also reduces the complexity of model training through dimensionality reduction.

[0276] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0277] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0278] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing an optimized data set, characterized in that: The method comprises: The target training model of the data set is used as the corresponding first target model; and based on the output data features of the first target model, the maximum feature range of the model input data is confirmed to obtain the corresponding first feature item sequence; and a first classification model for classifying and identifying the input data is constructed according to the preset first data classification label set and the first feature item sequence; and a first dimension reduction model for performing feature dimension reduction processing on the input high-dimensional data is constructed according to the first feature item sequence; the first feature item sequence includes N first feature items, N is a positive integer greater than 1; the first data classification label set includes S first classification labels, S is a positive integer greater than 1; Taking the input / output data feature range of the first target model as the data collection requirement, big data collection is performed to obtain a corresponding first collection data set; Preprocessing the first acquired data set to obtain a corresponding first preprocessed data set; Based on the first preprocessed data set, a classification model data set is constructed to obtain a corresponding first training set and a first evaluation set; and based on the first training set and the first evaluation set, the first classification model is trained; and after the classification model training is completed, the first preprocessed data set is deduplicated based on the first classification model to obtain a corresponding first deduplicated data set; Performing sample balancing and data enhancement processing on the first deduplication data set to obtain a corresponding first data set; A dimensionality reduction model data set is constructed based on the first data set to obtain a corresponding second training set and a second evaluation set; and the first dimensionality reduction model is trained based on the second training set, the second evaluation set and the first target model; and after the dimensionality reduction model training is completed, feature dimensionality reduction processing is performed on the first data set based on the first dimensionality reduction model to obtain a corresponding first optimized data set.

2. The method for processing an optimized data set according to claim 1, characterized in that: The model input vector of the first target model is denoted as X, and the model output vector is denoted as Y; the feature dimension of the model input vector X is adjustable and the maximum is N, that is, the model input vector X is composed of at most N input features x i , 1≤i≤N; the feature dimension of the model output vector Y remains unchanged, that is, the model output vector Y consists of M output features y j Composition, 1≤j≤M, M is a positive integer greater than 1; each of the output features y j Corresponding to a target model prediction feature item; The first classification model is implemented based on a machine learning model or a deep learning model; the first classification model is used to input the vector d according to the model. ain Perform classification recognition processing and output the corresponding classification prediction vector d aout ; The model input vector d ain The feature dimension is fixed to N; the classification prediction vector d aout The feature dimension of is fixed to S, and is composed of S classification prediction probabilities; the classification prediction probabilities correspond one-to-one to the first classification labels; The first dimensionality reduction model is implemented based on a type of machine learning model or a deep learning model; the first dimensionality reduction model is used to bin Perform feature dimensionality reduction processing and output the corresponding dimensionality reduction vector d bout ; The model input vector d bin The feature dimension is fixed to N; the dimension reduction vector d bout The feature dimension K is adjustable, 1≤K <N; The first acquisition data set includes a plurality of first acquisition data records; the first acquisition data record includes N first feature data fields and M second feature data fields; the first feature data fields correspond one-to-one to the first feature items; the second feature data fields correspond one-to-one to the target model prediction feature items; The first preprocessed data set includes a plurality of first preprocessed data records; the first preprocessed data records include N third characteristic data fields and M fourth characteristic data fields; The third feature data field corresponds one-to-one to the first feature item; the fourth feature data field corresponds one-to-one to the target model prediction feature item; The first deduplication data set includes a plurality of first deduplication data records; the first deduplication data record includes a first data type field, N fifth characteristic data fields and M sixth characteristic data fields; the first data type field corresponds to the first classification label in a one-to-one manner; the fifth characteristic data field corresponds to the first characteristic item in a one-to-one manner; the sixth characteristic data field corresponds to the target model prediction characteristic item in a one-to-one manner; The first data set includes a plurality of first data records; the first data record includes a second data type field, N seventh feature data fields and M eighth feature data fields; the second data type field corresponds one-to-one to the first classification label; the seventh feature data field corresponds one-to-one to the first feature item; the eighth feature data field corresponds one-to-one to the target model prediction feature item; The first optimized data set includes multiple second data records; the second data record includes a third data type field, K ninth feature data fields and M tenth feature data fields; the second data type field corresponds one-to-one to the first classification label; the tenth feature data field corresponds one-to-one to the target model prediction feature item.

3. The method for processing an optimized data set according to claim 2, characterized in that: The step of performing big data collection based on the input / output data feature range of the first target model as a data collection requirement to obtain a corresponding first collection data set specifically includes: The N first feature items and the M target model prediction feature items form a corresponding collection object attribute set; And through multiple big data collection channels, big data collection is performed with the collection object attribute set as the attribute requirements of a single data object to obtain multiple collected data objects; the multiple big data collection channels at least include government / institutional data channels, commercial databases, social media Internet and third-party data service companies; each of the collected data objects includes N first attribute data and M second attribute data, the first attribute data corresponds to the first feature item one by one; the second attribute data corresponds to the target model prediction feature item one by one; and taking each of the collected data objects as a corresponding current data object; and taking each of the first attribute data of the current data object as a corresponding first characteristic data field, and taking each of the second attribute data of the current data object as a corresponding second characteristic data field, and forming a corresponding first collected data record by the N third characteristic data fields and the M fourth characteristic data fields corresponding to the current data object; All the first collected data records obtained form a corresponding first collected data set.

4. The method for processing an optimized data set according to claim 2, characterized in that: The preprocessing of the first acquired data set to obtain a corresponding first preprocessed data set specifically includes: Pre-setting a corresponding feature item data rationality inspection standard and a corresponding feature item data standardization conversion rule for the N first feature items and the M target model prediction feature items respectively; A round of traversal is performed on all the first collected data records of the first collected data set; and in this round of traversal, the first collected data record currently traversed is used as the corresponding current data record; a round of polling is performed on all characteristic data fields of the current data record; and in this round of polling, the first or second characteristic data field currently polled is recorded as the corresponding current field; and whether the current field is empty is identified; if the current field is empty, the current data record is recorded as an abnormal record; if the current field is not empty, the current field is rationally checked based on the corresponding characteristic item data rationality check standard to obtain the corresponding The first inspection result is obtained, and when the first inspection result is reasonable, the field data of the current field is subjected to standardized conversion processing based on the corresponding standardized conversion rule of the characteristic item data and the conversion result is used as a corresponding third or fourth characteristic data field, and when the first inspection result is unreasonable, the current data record is recorded as an abnormal record; and at the end of polling all characteristic data fields of the current data record, whether the current data record is recorded as an abnormal record is identified, and if not, a corresponding first pre-processed data record is formed based on the N third characteristic data fields and the M fourth characteristic data fields corresponding to the current data record; And when the traversal of all the first collected data records of the first collected data set is completed, all the obtained first pre-processed data records form a corresponding first pre-processed data set.

5. The method for processing an optimized data set according to claim 2, characterized in that: The step of constructing a classification model dataset based on the first preprocessed dataset to obtain a corresponding first training set and a first evaluation set specifically includes: Pre-setting a corresponding data classification rule for each of the first classification labels of the first data classification label set; and based on a manual or machine data sorting method, randomly sorting a group of a specified number of the first pre-processed data records from the first pre-processed data set according to each of the data classification rules to form a corresponding first category record set; Each obtained set of the first category records is recorded as the corresponding current set; the first classification label corresponding to the current set is used as the corresponding current classification label; all the first pre-processed data records of the current set are traversed in one round; and in this round of traversal, the first pre-processed data record currently traversed is used as the corresponding current record; and based on the model input vector d ain The vector format of the model input vector is composed of the N third feature data fields of the current record and recorded as the corresponding first training vector; and based on the classification prediction vector d aout A classification prediction vector with a length of S and all the classification prediction probabilities being zero is set as the corresponding first label vector, and the classification prediction probability corresponding to the current classification label in the first label vector is reset to 1; and the first training vector corresponding to the current record and the first label vector form a corresponding first record; and at the end of this round of traversal, all the first records corresponding to the current set form a corresponding first record set; and the first record set is divided into two subsets according to a preset first segmentation ratio and recorded as the corresponding first subset and second subset; wherein the first and second subsets are both composed of multiple first records; the ratio of the total number of records in the first subset to the total number of records in the second subset satisfies the first segmentation ratio; The S obtained first subsets constitute the corresponding first training set; and the S obtained second subsets constitute the corresponding first evaluation set; wherein, the first training set and the first evaluation set are both composed of multiple first records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; the first record includes the first training vector and the first label vector.

6. The method for processing an optimized data set according to claim 5, characterized in that: The training of the first classification model based on the first training set and the first evaluation set specifically includes: Step 61, taking the first of the first records in the first training set as the corresponding current training record; Step 62, inputting the first training vector of the current training record into the first classification model for classification recognition processing to obtain a corresponding first classification prediction vector; Step 63, inputting the first classification prediction vector and the first label vector of the current training record into a preset first model loss function to calculate and obtain a corresponding first loss value; Wherein, the first model loss function at least includes an L1 loss function, an L2 loss function and a cross entropy loss function; Step 64, identifying whether the first loss value satisfies a preset first loss value range; if the first loss value satisfies the first loss value range, identifying whether the current training record is the last first record of the first training set, if so, turning to step 65, if not, extracting the next first record of the first training set as the new current training record and returning to step 62; if the first loss value does not satisfy the first loss value range, performing a round of parameter optimization processing on the model parameters of the first classification model in the direction of minimizing the first model loss function based on a preset first model optimizer, and returning to step 62 at the end of this round of parameter optimization processing; Wherein, the first model optimizer includes at least an SGD optimizer and an ADAM optimizer; Step 65, perform a round of traversal on all the first records of the first evaluation set; and in this round of traversal, use the first record currently traversed as the corresponding current evaluation record; and input the first training vector of the current evaluation record into the first classification model for classification and recognition processing to obtain the corresponding second classification prediction vector; and form a corresponding first prediction-label pair by the second classification prediction vector and the first label vector of the current training record; and at the end of this round of traversal, construct a confusion matrix based on all the first prediction-label pairs obtained and record it as the corresponding first confusion matrix; and perform corresponding precision and recall calculations based on the first confusion matrix to obtain the corresponding first precision and first recall; and perform F1 score calculation based on the first precision and the first recall to obtain the corresponding first F1 score; Step 66, identifying whether the first precision, the first recall and the first F1 score all satisfy the corresponding first precision range, the first recall range and the first F1 score range; if the first precision does not satisfy the corresponding first precision range, or the first recall does not satisfy the corresponding first recall range, or the first F1 score does not satisfy the corresponding first F1 score range, returning to step 61 to continue training; if the first precision, the first recall and the first F1 score all satisfy the corresponding first precision range, the first recall range and the first F1 score range, stopping training and confirming that the classification model training is completed.

7. The method for processing an optimized data set according to claim 2, characterized in that: The performing similar data deduplication processing on the first preprocessed data set based on the first classification model to obtain a corresponding first deduplication data set specifically includes: Perform a round of traversal on all the first preprocessed data records of the first preprocessed data set; and in this round of traversal, record the first preprocessed data record currently traversed as the corresponding current record; and based on the model input vector d ain The vector format is composed of N third feature data fields of the current record to form a model input vector as the corresponding first input vector; the first input vector is input into the first classification model for classification recognition processing to obtain the corresponding third classification prediction vector; the first classification label corresponding to the classification prediction probability with the largest probability value in the third classification prediction vector is taken as the corresponding first data type; and at the end of this round of traversal, multiple first pre-processed data records with the same first data type are extracted and grouped into one category to form a corresponding second category record set; After the traversal of all the first preprocessed data records of the first preprocessed data set is completed, a round of traversal is performed on all the obtained second-category record sets; and in this round of traversal, the currently traversed second-category record set is recorded as the corresponding current set; and based on the model input vector d ain The vector format is composed of N third characteristic data fields of each of the first preprocessed data records in the current set to form a corresponding first data vector; and the vector similarity of each two of the first data vectors in the current set is calculated to obtain a corresponding first similarity; and the first similarities whose similarity exceeds a preset first similarity threshold are recorded as corresponding second similarities; and one of the two first preprocessed data records corresponding to each of the second similarities is selected and recorded as a corresponding duplicate record; and all the duplicate records in the current set are deleted; After traversing all the second-category record sets, all the second-category record sets are merged to form a corresponding first merged set; And perform a round of traversal on all the first pre-processed data records of the first merged set; and during this round of traversal, record the first pre-processed data record currently traversed as the corresponding current record; and use the first data type corresponding to the current record, N third characteristic data fields and M fourth characteristic data fields as the corresponding first data type fields, N fifth characteristic data fields and M sixth characteristic data fields to form a corresponding first deduplication data record; and at the end of this round of traversal, all the first deduplication data records obtained are used to form the corresponding first deduplication data set.

8. The method for processing an optimized data set according to claim 2, characterized in that: The performing sample balancing and data enhancement processing on the first deduplication data set to obtain a corresponding first data set specifically includes: Step 81, extracting multiple first deduplication data records with the same first data type field from the first deduplication data set and grouping them into one category to form a corresponding third category record set; Step 82, counting the total number of records of each of the third-category record sets to obtain a corresponding first total number of records; and calculating the mean of all the obtained first total number of records to obtain a corresponding first average total number; and calculating a corresponding first ratio = first total number of records / first average total number based on each of the first total number of records and the first average total number; and recording the third-category record set whose first ratio is less than a preset first ratio threshold as a corresponding under-sampled set, and recording the third-category record set whose first ratio is greater than a preset second ratio threshold as a corresponding over-sampled set; Wherein, 0<first ratio threshold<1<second ratio threshold; Step 83, identifying the total number of the under-sampled sets and the over-sampled sets; if the total number of the under-sampled sets is not 0 or the total number of the over-sampled sets is not 0, then when the total number of the under-sampled sets is not 0, using the over-sampling method to perform record addition processing on each of the under-sampled sets, and when the total number of the over-sampled sets is not 0, using the under-sampling method to perform record deletion processing on each of the over-sampled sets, and after the record addition / deletion processing is completed, return to step 82; if the total number of the under-sampled sets and the total number of the over-sampled sets are both 0, go to step 84; Step 84, merge all the third category record sets to obtain the corresponding second merged set; and perform a round of traversal on all the first deduplicated data records of the second merged set; and in this round of traversal, record the first deduplicated data record currently traversed as the corresponding current record; and use the first data type field, N fifth characteristic data fields and M sixth characteristic data fields corresponding to the current record as the corresponding second data type field, N seventh characteristic data fields and M eighth characteristic data fields to form a corresponding first data record; and at the end of this round of traversal, all the first data records obtained are used to form the corresponding first data set.

9. The method for processing an optimized data set according to claim 2, characterized in that: The step of constructing a dimensionality reduction model dataset based on the first dataset to obtain a corresponding second training set and a second evaluation set specifically includes: Extract multiple first data records with the same second data type field in the first data set and group them into one category to form a corresponding fourth category record set; randomly select a specified number of first data records from each fourth category record set to form a corresponding second record set; and merge all the obtained second record sets to obtain a corresponding third merged set; and perform a round of traversal on all the first data records of the third merged set; and in this round of traversal, record the first data record currently traversed as the corresponding current record; and based on the model input vector d of the first dimensionality reduction model bin The vector format of the model input vector is composed of the N seventh characteristic data fields of the current record and recorded as the corresponding second training vector; and the vector format of the model output vector based on the first target model is composed of the M eighth characteristic data fields of the current record and recorded as the corresponding second label vector; and the second training vector and the second label vector corresponding to the current record form a corresponding second record; and at the end of this round of traversal, all the obtained second records form a corresponding second record sequence; and dividing the second record sequence into two subsequences according to a preset second division ratio, recorded as a corresponding first subsequence and a second subsequence; wherein both the first and second subsequences are composed of a plurality of the second records; and a ratio of the total number of records in the first subsequence to the total number of records in the second subsequence satisfies the second division ratio; The first subsequence is used as the corresponding second training set, and the second subsequence is used as the corresponding second evaluation set; wherein the second training set and the second evaluation set are both composed of multiple second records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio; and the second record includes the second training vector and the second label vector.

10. The method for processing an optimized data set according to claim 9, characterized in that: The training of the first dimensionality reduction model based on the second training set, the second evaluation set and the first target model specifically includes: Step 101, identifying a preset feature dimensionality reduction processing mode; if the feature dimensionality reduction processing mode is the first mode, go to step 102; if the feature dimensionality reduction processing mode is the second mode, go to step 105; Wherein, the feature dimension reduction processing mode includes a first mode and a second mode; Step 102, according to the preset k value selection rule, select multiple integer values ​​from the value range of the feature dimension K of the first dimensionality reduction model to form a corresponding k value range; and based on each k value in the k value range, set the feature dimension K of the first dimensionality reduction model to obtain a corresponding second dimensionality reduction model; and based on the feature dimension K of each second dimensionality reduction model, set the feature dimension of the model input vector X of the first target model to obtain a corresponding second target model; and each second dimensionality reduction model is connected with the corresponding second target model to form a corresponding first training model; Wherein, the k value range includes multiple k values; Step 103, performing model training and evaluation on each of the first training models based on the second training set and the second evaluation set to obtain a corresponding first evaluation score; Step 104, taking the second dimensionality reduction model and the second target model corresponding to the first training model having the highest first evaluation score as the latest first dimensionality reduction model and the first target model; and going to step 110; Step 105, identifying key feature items in the first feature item sequence according to the second training set and the second evaluation set based on a principal component analysis method to obtain a corresponding first key feature item sequence; Wherein, the first key feature item sequence includes a plurality of first key feature items; Step 106: Count the total number of feature items in the first key feature item sequence to obtain a corresponding first feature total number; and set the feature dimension K of the first dimensionality reduction model based on the first feature total number to obtain a corresponding third dimensionality reduction model; Step 107, reconstructing the second training set and the second evaluation set according to the first key feature item sequence based on a principal component analysis method to obtain a corresponding third training set and a third evaluation set; Wherein, the third training set and the third evaluation set are both composed of a plurality of third records; the ratio of the total number of records of the third training set to the third evaluation set satisfies the second segmentation ratio; the third record includes a third training vector, a third label vector and a first target vector; Step 108, performing model training and evaluation on the third dimensionality reduction model based on the third training set and the third evaluation set to obtain a corresponding first feature evaluation score sequence; wherein the first feature evaluation score sequence consists of the first feature evaluation scores of the total number of the first features; Step 109, and identify whether each of the first feature evaluation scores in the first feature evaluation score sequence exceeds a preset feature score threshold; if so, use the current third dimensionality reduction model as the latest first dimensionality reduction model, and go to step 110; if not, delete the first key feature item in the first key feature item sequence whose first feature evaluation score does not exceed the feature score threshold, and return to step 106; Step 110, solidifying the model parameters of the latest first dimensionality reduction model; and after the model parameters are solidified, confirming that the dimensionality reduction model training is completed.

11. The method for processing an optimized data set according to claim 10, characterized in that: The performing model training and evaluation on each of the first training models based on the second training set and the second evaluation set to obtain a corresponding first evaluation score specifically includes: Step 111, constructing a loss function of the first training model based on the specified loss function of the first target model and recording it as the corresponding second model loss function; Step 112, taking the first second record of the second training set as the corresponding current training record; Step 113, inputting the second training vector of the current training record into the second dimensionality reduction model of the first training model to perform feature dimensionality reduction processing and outputting a corresponding first dimensionality reduction vector; and inputting the first dimensionality reduction vector into the second target model of the first training model to perform model processing and using the obtained model output vector as the corresponding first prediction vector; Step 114, inputting the first prediction vector and the second label vector of the current training record into the second model loss function to calculate and obtain a corresponding second loss value; Step 115, identifying whether the second loss value satisfies a preset second loss value range; if the second loss value satisfies the second loss value range, identifying whether the current training record is the last second record of the second training set, if so, turning to step 116, if not, extracting the next second record of the second training set as the new current training record and returning to step 113; if the second loss value does not satisfy the second loss value range, performing a round of parameter optimization processing on the model parameters of the first training model in the direction of minimizing the second model loss function based on a preset second model optimizer, and returning to step 113 at the end of this round of parameter optimization processing; Wherein, the second model optimizer includes at least an SGD optimizer and an ADAM optimizer; Step 116, perform a round of traversal on all the second records of the second evaluation set; and in this round of traversal, use the second record currently traversed as the corresponding current evaluation record; and input the second training vector of the current evaluation record into the second dimensionality reduction model of the first training model for feature dimensionality reduction processing and output the corresponding second dimensionality reduction vector; and input the second dimensionality reduction vector into the second target model of the first training model for model processing and use the obtained model output vector as the corresponding second prediction vector; and form a corresponding second prediction-label pair with the second prediction vector and the second label vector of the current training record; and end this round of traversal , identify the model function of the second target model of the first training model; if the current second target model is a regression calculation model, bring all the second prediction-label pairs obtained into the preset first model evaluation function to calculate and obtain the corresponding first evaluation value; if the current second target model is a classification model, construct a confusion matrix based on all the second prediction-label pairs obtained and record it as the corresponding second confusion matrix, and perform corresponding precision and recall calculations based on the second confusion matrix to obtain corresponding second precision and second recall, and perform F1 score calculation based on the second precision and the second recall to obtain the corresponding second F1 score; Wherein, the first model evaluation function at least includes a standard deviation function and a RMSE function; Step 117, identifying the model function of the second target model of the first training model; if the current second target model is a regression calculation model, then identifying whether the first evaluation value satisfies the preset first evaluation value range; if the first evaluation value does not satisfy the first evaluation value range, returning to step 112 to continue training; if the first evaluation value satisfies the first evaluation value range, the reciprocal of the first evaluation value is used as the corresponding first evaluation score and output; if the current second target model is a classification model, then identifying whether the second precision, the second recall rate and the second F1 score all satisfy the corresponding second precision range, the second recall rate range and the second F1 score range; if the second precision does not satisfy the corresponding second precision range or the second recall rate does not satisfy the corresponding second recall rate range or the second F1 score does not satisfy the corresponding second F1 score range, returning to step 112 to continue training; if the second precision, the second recall rate and the second F1 score all satisfy the corresponding second precision range, the second recall rate range and the second F1 score range, the second F1 score is used as the corresponding first evaluation score and output.

12. The method for processing an optimized data set according to claim 10, characterized in that: The method of identifying the key feature items in the first feature item sequence according to the second training set and the second evaluation set based on the principal component analysis method to obtain the corresponding first key feature item sequence specifically includes: Step 121, counting the total number of the second records in the second training set and the second evaluation set to obtain a corresponding first total number and a second total number; and summing the first and second total numbers to obtain a corresponding third total number Q; Step 122, constructing a feature matrix A with a shape of Q×N according to all the second training vectors of the second training set and the second evaluation set; The feature matrix A is composed of Q×N first matrix units, each row of the feature matrix A corresponds to one first feature item, and each column corresponds to one second training vector; Step 123, based on the principal component analysis method, the covariance matrix of the characteristic matrix A is calculated to obtain a corresponding first covariance matrix; and the first covariance matrix is ​​subjected to eigendecomposition to obtain first principal component eigenvalues ​​and first principal component eigenvectors corresponding to each of the first characteristic items; Step 124, extract Q first matrix units from each row of the feature matrix A to form a corresponding first feature vector; and form a corresponding first target tensor from all the second label vectors of the second training set and the second evaluation set; and calculate the Pearson correlation coefficient between each first feature vector and the first target tensor to obtain a corresponding first Pearson correlation coefficient; Step 125, based on a preset robust standard deviation calculation method, the robust standard deviation of each of the first feature vectors is calculated to obtain a corresponding first robust standard deviation; Wherein, the robust standard deviation calculation method includes MAD method, Qn method and Sn method; Step 126, calculating the ratio coefficient of every two first feature items according to the first principal component eigenvalues, the first Pearson correlation coefficients and the first robust standard deviations corresponding to the first feature items to obtain the corresponding first ratio coefficients; The calculation formula of the first ratio coefficient is: 1≤g、h≤Q,R g,h is the ratio coefficient of the g-th first characteristic item to the h-th first characteristic item; λ g 、r g , S g is the first principal component eigenvalue, the first Pearson correlation coefficient and the first robust standard deviation corresponding to the g-th first feature item; h 、r h , S h is the first principal component eigenvalue, the first Pearson correlation coefficient and the first robust standard deviation corresponding to the hth first feature item; the first ratio coefficient R g,h If it is greater than a preset ratio coefficient threshold, it means that the g-th first feature item contributes more to the prediction of the first target model than the h-th first feature item, and the first ratio coefficient R g,h If it is less than or equal to the ratio coefficient threshold, it means that the contribution of the g-th first feature item to the prediction of the first target model is smaller than the contribution of the h-th first feature item; Step 127, record each of the first feature items whose first principal component eigenvalues ​​exceed a preset principal component feature threshold as a corresponding first key feature item; and according to the first ratio coefficients of all the first key feature items, sort all the first key feature items obtained in order of their prediction contribution to the first target model from large to small to obtain a corresponding first key feature item sequence.

13. The method for processing an optimized data set according to claim 10, characterized in that: The method of reconstructing the second training set and the second evaluation set according to the first key feature item sequence based on the principal component analysis method to obtain the corresponding third training set and third evaluation set specifically includes: The second training set or the third evaluation set is used as the corresponding current data set; and each second record of the current data set is recorded as the corresponding current record; and based on the principal component analysis method, the covariance matrix of the second training vector of the current record is calculated to obtain the corresponding second covariance matrix; and the second covariance matrix is ​​subjected to eigendecomposition to obtain the second principal component eigenvalues ​​and second principal component eigenvectors corresponding to each first feature item; and the second principal component eigenvectors corresponding to each first key feature item are sorted in the sorting order of the first key feature item sequence to obtain a corresponding third label vector; and the second label vector of the current record is used as a corresponding first target vector; and the third training vector corresponding to the current record, the third label vector and the first target vector form a corresponding third record; and all the third records corresponding to the current data set form the corresponding third training set or the third evaluation set.

14. The method for processing an optimized data set according to claim 10, characterized in that: The performing model training and evaluation on the third dimensionality reduction model based on the third training set and the third evaluation set to obtain a corresponding first feature evaluation score sequence specifically includes: Step 141, taking the first third record of the third training set as the corresponding current training record; Step 142, inputting the third training vector of the current training record into the third dimensionality reduction model to perform feature dimensionality reduction processing and outputting a corresponding third dimensionality reduction vector; Step 143, inputting the third dimension reduction vector and the third label vector of the current training record into a preset third model loss function to calculate and obtain a corresponding third loss value; Wherein, the third model loss function at least includes a mean square error loss function and a variance loss function; Step 144, identifying whether the third loss value satisfies a preset third loss value range; if the third loss value satisfies the third loss value range, identifying whether the current training record is the last third record of the third training set, if so, turning to step 145, if not, extracting the next third record of the third training set as the new current training record and returning to step 142; if the third loss value does not satisfy the third loss value range, performing a round of parameter optimization processing on the model parameters of the third dimensionality reduction model in the direction of minimizing the third model loss function based on a preset third model optimizer, and returning to step 142 at the end of this round of parameter optimization processing; Wherein, the third model optimizer includes at least an SGD optimizer and an ADAM optimizer; Step 145, perform a round of traversal on all the third records of the third evaluation set; and in this round of traversal, use the third record currently traversed as the corresponding current evaluation record; and input the third training vector of the current evaluation record into the third dimensionality reduction model for feature dimensionality reduction processing and output the corresponding fourth dimensionality reduction vector; and form a corresponding third prediction-label pair by the fourth dimensionality reduction vector and the third label vector of the current training record; and at the end of this round of traversal, bring all the obtained third prediction-label pairs into the preset second model evaluation function to calculate and obtain the corresponding second evaluation value; Wherein, the second model evaluation function at least includes a standard deviation function and an RMSE function; Step 146, identifying whether the second evaluation value meets a preset second evaluation value range, if the second evaluation value does not meet the second evaluation value range, returning to step 141 to continue training, if the second evaluation value meets the second evaluation value range, turning to step 147; Step 147, counting the total number of the third records in the third evaluation set and taking the obtained total number as the corresponding sample total number U; recording each of the third records in the third evaluation set as a corresponding sample record; and constructing a feature matrix C with a shape of U×N based on the third training vectors of the U sample records; and constructing a feature matrix D with a shape of U×K based on the fourth dimensionality reduction vectors corresponding to the U sample records; and constructing a feature matrix E with a shape of U×M based on the first target vectors of the U sample records; Wherein, the feature matrix C is composed of U×N second matrix units, each row of the feature matrix C corresponds to one of the first key feature items, and each column corresponds to one of the third training vectors of the sample record; the feature matrix D is composed of U×K third matrix units, each row of the feature matrix D corresponds to one of the first key feature items, and each column corresponds to one of the fourth reduced dimensionality vectors corresponding to the sample record; the feature matrix E is composed of U×M fourth matrix units, each row of the feature matrix E corresponds to one of the target model prediction feature items, and each column corresponds to one of the first target vectors of the sample record; Step 148, extract U of the second matrix units in each row of the feature matrix C to form a corresponding first key feature vector; extract M of the fourth matrix units in each column of the feature matrix E to form a corresponding first sample target vector; and calculate the correlation measure between each of the first key feature vectors and each of the first sample target vectors to obtain a corresponding first correlation measure; Step 149, confirming the principal component eigenvalues ​​corresponding to each of the first key feature items according to the feature matrix D to obtain the corresponding third principal component eigenvalues; Step 150, extract the K third matrix units in each column of the feature matrix D to form a corresponding first sample dimensionality reduction vector; and extract the third matrix units corresponding to each of the first key feature items on each of the first sample dimensionality reduction vectors as the corresponding first dimensionality reduction features; Step 151, performing key feature evaluation processing based on the first correlation measure, the third principal component eigenvalue and the first dimensionality reduction feature corresponding to each of the first key feature items to obtain the corresponding first feature evaluation score; The calculation formula of the first feature evaluation score F is: 1≤sample index u≤U,w u is the weight coefficient corresponding to the u-th target vector of the first sample; r u is the first correlation measure between the first key feature vector corresponding to the current key feature item and the uth first sample target vector; α is a preset regularization coefficient; d u is the first dimensionality reduction feature corresponding to the current key feature item on the u-th dimensionality reduction vector of the first sample; λ is the third principal component eigenvalue corresponding to the current key feature item; Step 152: sort all the obtained first feature evaluation scores according to the sorting order of the key feature items in the first key feature item sequence to form a corresponding first feature evaluation score sequence.

15. The method for processing an optimized data set according to claim 2, characterized in that: The performing feature dimensionality reduction processing on the first data set based on the first dimensionality reduction model to obtain a corresponding first optimized data set specifically includes: Perform a round of traversal on all the first data records of the first data set; and in this round of traversal, use the first data record currently traversed as the corresponding current record; and based on the model input vector d of the first dimensionality reduction model bin The vector format is composed of N seventh feature data fields of the current record to form a model input vector recorded as the corresponding current input vector; and the current input vector is input into the first dimensionality reduction model for feature dimensionality reduction processing to obtain the corresponding current dimensionality reduction vector; and the K vector features of the current dimensionality reduction vector are extracted as the corresponding K ninth feature data fields; and the second data type field and M eighth feature data fields of the current record are used as the corresponding third data type field and M tenth feature data fields; and the third data type field corresponding to the current record, K ninth feature data fields and M tenth feature data fields form a corresponding second data record; and at the end of this round of traversal, all the obtained second data records form the corresponding first optimized data set.

16. A device for executing the method for processing an optimized data set according to any one of claims 1 to 15, characterized in that: The device comprises: a model preparation module, a data acquisition module, a data preprocessing module, a classification and duplication removal module, a balance enhancement module and a feature dimension reduction module; The model preparation module is used to use the target training model of the data set as the corresponding first target model; and based on the output data features of the first target model, the maximum feature range of the model input data is confirmed to obtain the corresponding first feature item sequence; and a first classification model for classifying and identifying the input data is constructed according to the preset first data classification label set and the first feature item sequence; and a first dimension reduction model for performing feature dimension reduction processing on the input high-dimensional data is constructed according to the first feature item sequence; the first feature item sequence includes N first feature items, N is a positive integer greater than 1; the first data classification label set includes S first classification labels, S is a positive integer greater than 1; The data acquisition module is used to perform big data acquisition based on the input / output data feature range of the first target model as the data acquisition requirement to obtain a corresponding first acquisition data set; The data preprocessing module is used to preprocess the first acquired data set to obtain a corresponding first preprocessed data set; The classification deduplication module is used to construct a classification model data set based on the first preprocessed data set to obtain a corresponding first training set and a first evaluation set; and to train the first classification model based on the first training set and the first evaluation set; and after the classification model training is completed, the first preprocessed data set is deduplicated based on the first classification model to obtain a corresponding first deduplicated data set; The balancing and enhancing module is used to perform sample balancing and data enhancement processing on the first deduplication data set to obtain a corresponding first data set; The feature dimension reduction module is used to construct a dimension reduction model data set based on the first data set to obtain a corresponding second training set and a second evaluation set; and to train the first dimension reduction model based on the second training set, the second evaluation set and the first target model; and after the dimension reduction model training is completed, the first data set is subjected to feature dimension reduction processing based on the first dimension reduction model to obtain a corresponding first optimized data set.

17. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 15; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 15.

Images