Model training method based on dynamic classification and big data prediction method

Abstract

The invention discloses a dynamic classification-based model training method and a big data prediction method. The method comprises the steps of obtaining a first training subset, a second training subset and a prediction set; a plurality of classification models are trained according to a first training subset and a second training subset obtained through division according to the initial division value, and an optimal classification model is obtained according to the score value of each classification model; and using the optimal classification model to classify the training set and the prediction set, and further performing redundancy training on the prediction model to obtain a data prediction result. According to the method, the problems of feature dependence and high and low numerical value mixing existing in a traditional single classification model are solved by dynamically adjusting division values and selecting an optimal classification model; a data set, used for training, of the prediction model is expanded through a redundancy training method, the utilization rate of edge data is increased, and therefore a good prediction result can still be obtained under the condition that the data classification condition is poor; the method is more flexible and reliable.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a model training method and a big data prediction method based on dynamic classification. Background Art

[0002] Big data prediction technology relies on the ability to analyze and process massive amounts of data, and aims to extract valuable information from complex data sets, and to perform trend prediction and decision support. For example, in the field of batteries, big data prediction technology can effectively predict the capacity of batteries and their related states, estimate the battery health status, and predict the remaining service life. However, due to the diversity and complexity of the data itself, big data prediction still faces the problem of poor accuracy. In order to improve the accuracy of prediction, existing methods usually use feature engineering techniques to extract effective features from the data, and improve the prediction effect by superimposing multiple prediction models. However, when classifying data, the classification effect of a single classification model is poor, and traditional classification models mainly rely on features to classify targets, which easily leads to a mixture of high and low values ​​in the classification results, thereby affecting the accuracy of classification.

[0003] Therefore, it is very important to design a method that can dynamically classify data to improve classification accuracy. Summary of the invention

[0004] The main technical problem solved by the present invention is how to improve the classification accuracy and thus improve the accuracy of the prediction results.

[0005] According to the first aspect, an embodiment provides a model training method based on dynamic classification, including:

[0006] Acquire sample data, the sample data including a training set and a prediction set, the training set including a first training subset and a second training subset;

[0007] Obtaining an initial partition value, partitioning the first training subset according to the initial partition value to obtain a first training subset after partitioning, and training multiple classification models according to the first training subset and the second training subset after partitioning, wherein training any one of the classification models according to the first training subset and the second training subset after partitioning includes:

[0008] A current training round obtaining step, obtaining the current training round of the current classification model;

[0009] A training and prediction step, using the divided first training subset to train the current classification model, and predicting the trained classification model in the second training subset to obtain a prediction result;

[0010] The parameter calculation step is to obtain the confusion matrix corresponding to the current classification model in the current training round according to the obtained prediction results and the actual classification results; obtain the loss value corresponding to the current classification model in the current training round according to the confusion matrix; set the tolerance value corresponding to the current classification model in the current training round according to the current training round and the preset total number of training; obtain the best loss value and loss value record table of the current classification model according to the loss value and tolerance value corresponding to the current classification model in the current training round and the loss value corresponding to the current classification model in the historical training rounds;

[0011] A judgment step, judging whether the classification model after the current training round meets the convergence condition or the termination condition;

[0012] A first continuing training step, if the classification model after the current training round training in the judging step does not meet the convergence condition and the termination condition, then retraining the current analysis model from the current training round obtaining step;

[0013] A second continuing training step, if the classification model after the current training round training in the judgment step does not meet the convergence condition and meets the termination condition, then obtaining a penalty matrix, and obtaining a penalty result matrix according to the penalty matrix and the confusion matrix corresponding to the current classification model in the current training round; obtaining a learning rate, correcting the division value according to the penalty result matrix and the learning rate, using the corrected division value to re-divide the first training subset to obtain a re-divided first training subset, and re-training the current classification model according to the re-divided first training subset and the second training subset;

[0014] A scoring calculation step, if the classification model after the current training round meets the convergence condition in the judgment step, then the scoring value of the current classification model is calculated according to the best loss value of the current classification model and the loss value record table;

[0015] A classification model is selected as the best classification model according to the score values ​​of the various classification models.

[0016] According to the second aspect, an embodiment provides a big data prediction method based on redundant training, including:

[0017] Obtaining the best classification model through the model training method;

[0018] The training set and the prediction set are classified using the best classification model to obtain multiple training subsets and prediction subsets. The prediction model is redundantly trained based on the obtained multiple training subsets and prediction subsets to obtain data prediction results.

[0019] According to a model training method and a big data prediction method based on dynamic classification of the above-mentioned embodiment, first, data cleaning and feature extraction are performed on the sample data, and the first training subset, the second training subset and the prediction set after division are obtained according to the initial division value, and then the second training subset and the first training subset after division are used to train multiple classification models. When the classification model cannot converge, a penalty is performed in combination with the classification result, so as to dynamically adjust the division value, thereby solving the problem of feature dependence and mixed high and low values ​​in the traditional classification model. Finally, the best classification model is selected according to the score values ​​of different classification models, thereby avoiding the situation where the classification effect of a single classification model is poor; the prediction model is trained by adjacent training subsets, and then the trained prediction model is used to predict a single prediction subset. On the one hand, the data set used for training the prediction model can be expanded, and on the other hand, the utilization rate of edge data can be improved. Compared with the traditional method, this method has higher flexibility and reliability, and can still obtain better prediction results when the data classification is not good. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of a model training method based on dynamic classification;

[0021] Figure 2 It is a flowchart of a big data prediction method based on redundant training;

[0022] Figure 3 Training flowchart for classification model training;

[0023] Figure 4 This is the prediction result diagram without dynamic classification and redundant training;

[0024] Figure 5 This is the prediction result diagram after dynamic classification and redundant training. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0026] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0027] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0028] In an embodiment of the present invention, the inventive concept is to first divide the first training subset according to the initial division value, and train multiple classification models according to the divided first training subset. In this process, the division value is dynamically adjusted by combining factors such as data characteristics, classification results and penalties, which can solve the problems of feature dependence, mixed high and low values ​​and insufficient accuracy of classification results in traditional classification models; then, based on the dynamic adjustment of the division value, a scoring mechanism for multiple classification models is set to select the best classification model. Compared with traditional classification models, this is conducive to improving the adaptability and generalization ability of the classification model, obtaining better classification effects, and thus obtaining more reliable prediction results.

[0029] Step S100: Acquire sample data, process the sample data, and obtain a first training subset, a second training subset, and a prediction set.

[0030] First, the acquired sample data is cleaned to eliminate missing values ​​and outliers. The sample data collected in this embodiment is the cell capacity data in the production process. Then, feature engineering methods are used to extract features from the sample data to obtain a training set and a prediction set with features. Generally, 70% of all the data with features are used as the training set, and the remaining part is used as the prediction set. Then, the training set is divided into a first training subset and a second training subset in a ratio of 3:7.

[0031] Step S110: obtaining an initial partition value, training multiple classification models according to the initial partition value, the first training subset and the second training subset, and obtaining a score value of each classification model.

[0032] First, set the initial partition value, and record the number of initial partition values ​​as N. The value of N can be set according to the actual situation. Sort the initial partition values ​​and the data in the training set in ascending or descending order; use all the partition values ​​to partition the first training subset, thereby obtaining the first training subset after partitioning, where N partition values ​​will divide the first training subset into N+1 categories, and the category numbers start from 0;

[0033] Assume that the data in the training set is battery cell capacity data. Since the battery cell capacity of the actually produced battery is not a fixed value, for example, a battery with a battery cell capacity of 108Ah needs to be produced, but the actual battery cell capacity is in the range of 106Ah to 110Ah and satisfies the normal distribution. In order to adapt to different data ranges, this embodiment obtains the partition value by setting the proportion of data in different categories, wherein the proportion of different categories can be set by oneself; then taking N=2 as an example, at this time, two initial partition values ​​are required to divide the first training subset into three categories, and the proportions of these three categories are set to 0.3, 0.4 and 0.3 respectively, that is, the first 30% of the data in the first training subset is taken as the 0th category, the last 30% of the data is taken as the 2nd category, and the middle 40% of the data is taken as the 1st category, and then the largest value in the 0th category is taken as the first partition value, and the largest value in the 1st category is taken as the second partition value; different categories correspond to different capacity levels.

[0034] Then, prepare several classification models, such as SVM (support vector machine), random forest model and multi-layer perceptron model, etc. The number of classification models can be set according to the actual situation. The multiple classification models are trained according to the second training subset and the first training subset after division. The training process of any classification model is as follows Figure 3 As shown, including:

[0035] The current training round acquisition step S111 is to acquire the current training round of the current classification model.

[0036] In the training and prediction step S112, the current classification model is trained using the first training subset after division, and the trained classification model is predicted in the second training subset to obtain a prediction result.

[0037] Parameter calculation step S113, according to the obtained prediction results and the actual classification results, the confusion matrix corresponding to the current classification model in the current training round is obtained; wherein the obtained confusion matrix is ​​a (N+1)*(N+1) matrix, each row in the confusion matrix corresponds to an actual category, each element in the confusion matrix corresponds to a classification result of the current classification model, the elements on the off-diagonal line of the confusion matrix correspond to the misclassification situation, that is, the element value on the off-diagonal line is the proportion of the amount of each misclassified data to all the data, and the elements on the diagonal line of the confusion matrix correspond to the correct situation, that is, the element value on the diagonal line is the proportion of the amount of each correctly classified data to all the data;

[0038] Taking N=2 as an example, if the confusion matrix corresponding to the current classification model in the current training round is:

[0039]

[0040] The first row in the confusion matrix represents the proportion of data that should belong to class 0 to all data, the second row represents the proportion of data that should belong to class 1 to all data, and the third row represents the proportion of data that should belong to class 2 to all data;

[0041] Taking the second row as an example, a total of 40% of the data belongs to the first category, but only 20% of the data is correctly classified, 5% of the data is classified into the 0th category, 15% of the data is classified into the second category, and so on; the row numbers and column numbers of all matrices in this embodiment are numbered from 0, for example, the coordinates of the elements corresponding to the first row and the first column in the matrix are (0, 0), and the coordinates of the elements corresponding to the third row and the third column are (1, 2); that is, the element value with coordinates (1, 2) in the confusion matrix represents the proportion of data that should belong to the first category but is classified as the second category by the current classification model to all data; it can be seen that, for the confusion matrix, the misclassification corresponding to the elements above the diagonal of the confusion matrix is ​​relatively large relative to the actual classification result, and the misclassification corresponding to the elements below the diagonal of the confusion matrix is ​​relatively small relative to the actual classification result;

[0042] It should be noted that, taking the battery cell capacity data as an example, the actual classification results are manually labeled, and the real capacity level is obtained through professional knowledge and experimental measurement. This method relies on the knowledge and experimental equipment of experts. It conducts detailed tests on the battery and combines multiple factors for comprehensive judgment to obtain accurate capacity data and classify it accordingly.

[0043] Calculate the cumulative sum of all elements on the off-diagonal line of the confusion matrix, and use the obtained cumulative sum as the loss value corresponding to the current classification model in the current training round, that is, the cumulative sum of all elements corresponding to the misclassification in the confusion matrix is ​​used as the loss value corresponding to the current classification model in the current training round;

[0044] The tolerance value corresponding to the current classification model in the current training round is set according to the current training round and the preset total number of training: if the current training round is less than 1 / 2 of the preset total number of training, the tolerance value is set to 1.04; if the current training round is greater than or equal to 1 / 2 of the preset total number of training and less than 3 / 4 of the preset total number of training, the tolerance value is set to 1.1; if the current training round is greater than or equal to 3 / 4 of the preset total number of training, the tolerance value is set to 1.16; wherein the preset total number of training in this embodiment can be set according to actual conditions;

[0045] According to the loss value and tolerance value corresponding to the current classification model in the current training round and the loss value corresponding to the current classification model in the historical training round, the relevant parameters of the current classification model are obtained, namely, the best loss value, the best model file, the iteration record table and the loss value record table of the current classification model; wherein, the best loss value refers to the minimum value of all loss values ​​obtained by the current classification model in different training rounds, the best model file refers to the model file corresponding to the best loss value, the iteration record table is used to judge the training status of the current classification model, and the loss value record table is used to record the loss values ​​obtained by the current classification model in different training rounds. The specific process of obtaining the best loss value, the best model file, the iteration record table and the loss value record table of the current classification model is as follows:

[0046] Set the best loss value for the first training to 1. If the loss value of the current classification model in the current training round is less than the best loss value, set the loss value corresponding to the current training round as the best loss value. The model file obtained after the current training round is completed with the best loss value is used as the best model file, and the iteration mark value is recorded as True. Delete all data in the iteration record table and the loss value record table, and then record the current iteration mark value in the iteration record table. At this time, the number of data in the iteration record table is 1, and the data in the loss value record table is 0.

[0047] If the loss value obtained in the current training round is greater than or equal to the optimal loss value and less than the product of the optimal loss value and the tolerance value corresponding to the current training round, the iteration mark value is recorded as True, and the obtained iteration mark value is recorded in the iteration record table; if the loss value obtained in the current training round is greater than the product of the optimal loss value and the tolerance value corresponding to the current training round, the iteration mark value is recorded as False, and the obtained iteration mark value is recorded in the iteration record table; when the number of data in the iteration record table is greater than or equal to 5 and the last 5 are [False, True, False, True, False] or the number of data recorded in the iteration record table is equal to 10% of the total number of training and all are False, the entire iteration record table is reset to 5 False, that is, the iteration record table at this time only includes 5 False se, and adjust the learning rate to 0.9 times the initial value. In this embodiment, the initial value of the learning rate is set to 0.2, and the learning rate is used to subsequently correct the division value; if the number of data recorded in the loss value record table is greater than 10 data and the last 10 consecutive data are the same loss value, the learning rate is increased to 1.1 times the original value; wherein, resetting the iteration record table is used to prevent repeated triggering operations.

[0048] Determine step S114, determine whether the classification model after training in the current training round meets the convergence condition or the termination condition, wherein the convergence condition refers to: if the number of data recorded in the iteration record table of the classification model after training in the current training round is greater than the preset first reference value, at the end of the iteration record table, the number of data values ​​that are continuously True is equal to the preset first reference value and the variance between the loss values ​​corresponding to each data is less than the product between the preset reference coefficient and the tolerance value, the classification model is considered to be converged; in this embodiment, the preset first reference value is 10, and the preset reference coefficient is 0.0009, that is, if the number of data recorded in the iteration record table is greater than 10, the last 10 iteration mark values ​​are continuously True, and the variance between the loss values ​​obtained by these 10 iteration mark values ​​in the corresponding training rounds is less than the product between 0.0009 and the tolerance value corresponding to the current training round, the classification model is considered to be converged at this time;

[0049] The termination condition refers to the condition that the current classification model has not converged but training needs to be stopped. The termination condition includes two situations. The first situation is that if the current training round is equal to the total training number, the training is stopped; the second situation is that when the current training round is not equal to the total training number, the number of data in the loss value recording table is greater than the preset second reference value and the number of loss values ​​that are the same value in succession is equal to the preset second reference value. The preset second reference value set in this embodiment is 15, that is, if the number of data recorded in the loss value recording table is greater than 15 and the last 15 consecutive data are the same loss value, the training is stopped. At this time, it means that the current classification model is stuck in the local minimum; in addition, when the number of data recorded in the loss value recording table is greater than 10 data and the last 10 consecutive data are the same loss value, the learning rate needs to be increased to 1.1 times the original value. Therefore, in the second case, the learning rate will also be increased five times in succession accordingly.

[0050] The first continuing training step S115, if the classification model after the current training round training does not meet the convergence condition and the termination condition in the judging step, then start from the current training round obtaining step and retrain the current analysis model.

[0051] The second continuing training step S116, if the classification model after the current training round training does not meet the convergence condition and meets the termination condition in the judgment step, that is, the current classification model has stopped training, but has not converged, it is necessary to adjust the partition value according to the confusion matrix and learning rate after the current training round. The specific adjustment process is:

[0052] First, a penalty matrix is ​​obtained. The penalty matrix in this embodiment is related to the number of partition values. When the number of partition values ​​is N, the penalty matrix is ​​set to:

[0053]

[0054] Then when N=2, the corresponding penalty matrix is:

[0055]

[0056] The first penalty operation is performed according to the obtained penalty matrix. This penalty is intended to set a corresponding penalty for the degree of misclassification, that is, the greater the classification result, the greater the penalty. Since the elements on the diagonal of the confusion matrix correspond to the correct classification of the current classification model according to the partition value, the farther away from the diagonal, the greater the deviation of the current classification model after the partition value is divided. For example, for data that should belong to class 0, data classified as class 2 will have a greater deviation. Therefore, the penalty for data classified into class 1 is less than that for data classified into class 2. Taking the data with the element value X in the penalty matrix as an example, the first penalty operation is performed on the data:

[0057]

[0058] Among them, Y is the result of the element value X in the penalty matrix after the first penalty operation, and N is the number of partition values.

[0059] Process each element value in the penalty matrix to obtain the classification result matrix. When N=2, the corresponding classification result matrix is:

[0060]

[0061] Then multiply the elements at the same position of the obtained classification result matrix and the confusion matrix to obtain the classification result correction matrix. When N=2, the corresponding classification result correction matrix is:

[0062]

[0063] Taking into account that different amounts of misclassified data should also have different degrees of punishment, that is, if a classification result is misclassified and the corresponding proportion of the classification result is higher, the classification result represented is more deviated from the actual situation, and the required degree of punishment is also greater. Therefore, it is necessary to perform a second penalty operation on the classification result correction matrix. The higher the corresponding element value for the misclassification situation, the higher the degree of punishment that needs to be set. In this embodiment, the square value of each element in the classification result correction matrix is ​​used as the element value at the corresponding position of the final penalty matrix. When N=2, the final penalty matrix obtained can be expressed as:

[0064]

[0065] Multiply the elements at the same position of the confusion matrix and the final penalty matrix. Since each element in the confusion matrix is ​​in the form of the proportion of different classification situations to all data, it is also necessary to compensate each multiplication result by percentage, that is, multiply the elements at the same position of the confusion matrix and the final penalty matrix, and then multiply the obtained multiplication result by 100 to obtain the penalty result matrix. When N = 2, the penalty result matrix is:

[0066]

[0067] Finally, the partition value is corrected according to the penalty result matrix and the learning rate. The specific correction process is: for the Mth partition value, the positive value and the negative value of the partition value are obtained, wherein the positive value of the partition value is the cumulative sum of all elements from (M-1, M) to (M-1, N) in the penalty result matrix; the negative value of the partition value is the opposite of the cumulative sum of all elements from (M, 0) to (M, M-1) in the penalty result matrix; the sum of the positive value and the negative value is multiplied by the learning rate, the product is added to the Mth partition value, and the sum is used as the corrected Mth partition value;

[0068] For example, when N=2, for the initial first division value A, the positive value of the division value is the cumulative sum of all elements with coordinates from (0, 1) to (0, 2) in the confusion matrix, that is, the positive value of the division value is 0.15129+0.43681=0.5881, and the negative value of the division value is the opposite of the cumulative sum of all elements with coordinates from (1, 0) to (1, 0) in the confusion matrix, that is, (-0.01922) is the negative value of the division value. Therefore, the correction result for the first division value is A+(0.5881+(-0.01922))*learning rate, and so on.

[0069] Since the elements above the diagonal of the confusion matrix correspond to the situation that the actual classification results are larger, and the elements below the diagonal of the confusion matrix correspond to the situation that the actual classification results are smaller, take the first partition value when N=2 as an example. According to this partition value, the classification model will divide the data into Class 0 and Class 1. For Class 0, only 10% of the data in the classification results are correctly classified as Class 0, and the rest of the data are more classified into Class 1 and Class 2. In other words, the first partition value is smaller than Class 0. If the first partition value is increased, more data that should belong to Class 0 can be correctly classified into Class 0. Therefore, the situation in the confusion matrix that should belong to Class 0 but is classified into other categories indicates that the first partition value needs to be increased. The value increases in a positive direction, that is, the positive value when adjusting the first partition value is obtained according to the cumulative sum of all elements from (0, 1) to (0, 2); for the first category, 5% of the data that should belong to the first category is divided into the 0th category, that is, for the first category, if the first partition value is reduced, the amount of data that should belong to the first category but is divided into the 0th category can be reduced. Therefore, in the case where the confusion matrix should belong to the first category but is divided into the 0th category, the first partition value needs to be negatively reduced, that is, the opposite number of the cumulative sum of all elements from (1, 0) to (1, 0) is used as the negative value when adjusting the first partition value, and so on; this embodiment adjusts the partition value accordingly through positive and negative values, so as to achieve the best partition balance;

[0070] The first training subset is re-divided using the revised partition value to obtain a re-divided first training subset, and the current classification model is re-trained according to the second training subset and the re-divided first training subset.

[0071] In the scoring calculation step S117, if the classification model after the current training round meets the convergence condition in the judgment step, the scoring value of the current classification model is calculated according to the best loss value of the current classification model and the loss value record table; the method for obtaining the scoring value of the current classification model is as follows: the average value of the last preset third reference value loss values ​​in the loss value record table of the current classification model is calculated, and the obtained average value is weightedly summed with the best loss value corresponding to the current classification model, and the obtained weighted summation result is used as the current scoring value of the classification model; wherein, the preset third reference value set in this embodiment is 10, and the reference weights of the obtained average value and the best loss value are both 0.5, that is, the average value of the last 10 loss values ​​in the loss value record table of the current classification model is calculated, and the obtained average value is weightedly summed with the best loss value corresponding to the current classification model, and the obtained weighted summation result is used as the scoring value of the current classification model.

[0072] Step S120: Obtain the best classification model according to the scoring values ​​of each classification model.

[0073] A classification model is selected as the best classification model according to the scoring values ​​of each classification model. In this embodiment, the best model file saved by the classification model with the smallest scoring value is used as the best classification model.

[0074] Please refer to Figure 1 In this embodiment, the sample data is first cleaned and features are extracted to obtain a first training subset, a second training subset and a prediction set, and the first training subset after division is obtained according to the initial division value, and then the second training subset and the first training subset after division are used to train multiple classification models. When the classification model cannot converge, a penalty is performed in combination with the classification result, so as to dynamically adjust the division value, thereby solving the problem of feature dependence and mixed high and low values ​​in the traditional classification model. Finally, the best classification model is selected according to the scoring values ​​of different classification models, thereby avoiding the situation where the classification effect of a single classification model is poor. Compared with the traditional method, this embodiment has higher flexibility and reliability.

[0075] A big data prediction method based on redundant training, that is, training the prediction model according to the best classification model and the redundant training method to obtain the data prediction result, please refer to Figure 2 , the process includes:

[0076] Step S200: Use the best classification model to divide the entire training set and prediction set to obtain various training subsets and prediction subsets.

[0077] In general, by using the best classification model to divide the entire training set and prediction set, N+1 training subsets and N+1 prediction subsets can be obtained, and the training subsets and the prediction subsets are one-to-one corresponding, where the i-th training subset and the i-th prediction subset belong to the same category.

[0078] Step S210: performing redundant training according to each training subset and prediction subset to obtain data prediction results.

[0079] Since the data samples of each training subset and prediction subset obtained after the actual division may be unbalanced, for example, the first training subset contains only 10 sample data, while the first prediction subset contains 30 data, which will lead to the phenomenon of unbalanced training samples; in addition, it is not ruled out that the training subset cannot correspond to the prediction subset one by one due to the lack of classification in the training set. Therefore, this embodiment first selects the training subset according to the prediction subset, that is:

[0080] For any prediction subset, a prediction model is trained using the training subset corresponding to the prediction subset and the training subset adjacent to the training subset, and then the trained prediction model is used to predict the prediction subset. For example, for the i-th prediction subset, when i>1, the prediction model is trained using the i-1-th training subset, the i-th training subset, and the i+1-th training subset, and the trained prediction model is used to predict the i-th prediction subset. By expanding the amount of data in the training subset, the prediction model can be used even when the data classification is poor. Get better prediction results; for example, for the second prediction subset, use the first, second and third training subsets to train the prediction model, and then use the trained prediction model to predict the second prediction subset; when i = 1, the only one adjacent to the first training subset is the second training subset, so for the first prediction subset, use the first and second training subsets to train the prediction model, and then use the good prediction model to predict the first prediction subset, and so on; finally, use the prediction results output by all prediction models as data prediction results.

[0081] It should be noted that, since different training subsets correspond to different categories, for example, for the i-th training subset, the data in the i-1-th training subset is smaller than the data in the i-th training subset, and the data in the i+1-th training subset is larger than the data in the i-th training subset. Therefore, when using adjacent training subsets to train the prediction model, if all the data in the adjacent training subsets are directly used as the data required for the prediction model training, it will cause a certain deviation in the output result of the prediction model. Therefore, this embodiment only selects part of the data in the adjacent training subsets for training, wherein the total number of "partial data" selected from all adjacent training subsets shall not exceed the number of data in the current training subset, and must be "as close to its own classification range as possible"; that is, for the i-th training subset, if there are m data in this training subset, the data selected from the i-1-th training subset is the largest one in the i-1-th training subset. data, among which Indicates the rounding down symbol. The data selected from the i+1th training subset is the smallest in the i+1th training subset. data, if the number of data contained in the i-1th training subset or the i+1th training subset is insufficient At this time, all the data in the i-1th training subset or the i+1th training subset are selected, and so on.

[0082] The prediction results using the traditional method, that is, without dynamic classification and redundant training, are as follows Figure 4 As shown, using this method, the prediction results after dynamic classification and redundant training are as follows Figure 5As shown in the figure, "Avg_accuracy" represents the average accuracy, which corresponds to the average value of the accuracy of all predicted values; "With in 1%" represents the proportion of data whose error between the predicted value and the actual value is within the range of ±1%, and "With in 0.5%" represents the proportion of data whose error between the predicted value and the actual value is within the range of ±0.5%. By comparing the prediction results without dynamic classification and redundant training with the prediction results after dynamic classification and redundant training, it can be seen that the prediction results are more accurate after using the dynamic classification and redundant training method, which is manifested in the image as a more concentrated distribution of data points and fewer discrete points.

[0083] This embodiment adds a redundant training module on the basis of the first embodiment. By training the prediction model with adjacent training subsets and then using the trained prediction model to predict a single prediction subset, on the one hand, the data set used for training the prediction model can be expanded, and on the other hand, the utilization rate of edge data can be improved, so that better prediction results can still be obtained even when the data classification is not good.

[0084] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0085] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A model training method based on dynamic classification, characterized in that: include: Acquire sample data, the sample data including a training set and a prediction set, the training set including a first training subset and a second training subset; Obtaining an initial partition value, partitioning the first training subset according to the initial partition value to obtain a first training subset after partitioning, and training multiple classification models according to the first training subset and the second training subset after partitioning, wherein training any one of the classification models according to the first training subset and the second training subset after partitioning includes: A current training round obtaining step, obtaining the current training round of the current classification model; A training and prediction step, using the divided first training subset to train the current classification model, and predicting the trained classification model in the second training subset to obtain a prediction result; The parameter calculation step is to obtain the confusion matrix corresponding to the current classification model in the current training round according to the obtained prediction results and the actual classification results; obtain the loss value corresponding to the current classification model in the current training round according to the confusion matrix; set the tolerance value corresponding to the current classification model in the current training round according to the current training round and the preset total number of training; obtain the best loss value and loss value record table of the current classification model according to the loss value and tolerance value corresponding to the current classification model in the current training round and the loss value corresponding to the current classification model in the historical training rounds; A judgment step, judging whether the classification model after the current training round meets the convergence condition or the termination condition; A first continuing training step, if the classification model after the current training round training in the judging step does not meet the convergence condition and the termination condition, then retraining the current analysis model from the current training round obtaining step; A second continuing training step, if the classification model after the current training round training in the judgment step does not meet the convergence condition and meets the termination condition, then obtaining a penalty matrix, and obtaining a penalty result matrix according to the penalty matrix and the confusion matrix corresponding to the current classification model in the current training round; obtaining a learning rate, correcting the division value according to the penalty result matrix and the learning rate, using the corrected division value to re-divide the first training subset to obtain a re-divided first training subset, and re-training the current classification model according to the re-divided first training subset and the second training subset; A scoring calculation step, if the classification model after the current training round meets the convergence condition in the judgment step, then the scoring value of the current classification model is calculated according to the best loss value of the current classification model and the loss value record table; A classification model is selected as the best classification model according to the score values ​​of the various classification models.

2. The method according to claim 1, characterized in that The convergence conditions include: In the parameter calculation step, an iteration record table of the current classification model is obtained according to the loss value and tolerance value corresponding to the current classification model in the current training round and the loss value corresponding to the current classification model in the historical training round; The classification model is considered to have converged when the number of data recorded in the iteration record table is greater than the preset first reference value, and at the end of the iteration record table, the number of data values ​​that are consecutively True is equal to the preset first reference value, and the variance between the loss values ​​corresponding to each data is less than the product of the preset reference coefficient and the tolerance value.

3. The method according to claim 2, characterized in that The method for obtaining the iteration record table of the current classification model includes: If the loss value corresponding to the current classification model in the current training round is less than the optimal loss value, the iteration mark value is recorded as True, all historical data in the iteration record table is deleted, and the obtained iteration mark value is recorded in the iteration record table; If the loss value corresponding to the current training round is greater than or equal to the optimal loss value and less than the product of the optimal loss value and the tolerance value, the iteration mark value is recorded as True, and the obtained iteration mark value is recorded in the iteration record table; If the loss value obtained in the current training round is greater than the product of the optimal loss value and the tolerance value, the iteration mark value is recorded as False, and the obtained iteration mark value is recorded in the iteration record table.

4. The method according to claim 1, characterized in that The termination conditions include: when the current training round is equal to the total training number or the current training round is not equal to the total training number, the number of data in the loss value record table is greater than the preset second reference value and the last number of consecutive loss values ​​of the same value is equal to the preset second reference value.

5. The method according to claim 1, characterized in that The method for obtaining the penalty result matrix is: For any element in the penalty matrix, calculate the ratio between the element and the total number of partition values ​​plus 1, add the square of the obtained ratio to 1, and use the square value of the summed result as the element value of the corresponding position in the classification result matrix, and set the elements on the diagonal of the classification result matrix to 0; multiply the elements in the same position of the classification result matrix and the confusion matrix to obtain the classification result correction matrix; use the square value of each element in the classification result correction matrix as the element value at the corresponding position of the final penalty matrix; The final penalty matrix is ​​multiplied by the elements at the same position of the confusion matrix, and then multiplied by 100. The resulting product is used as the element value at the corresponding position in the penalty result matrix to obtain the penalty result matrix.

6. The method according to claim 1, characterized in that The modifying of the partition value according to the penalty result matrix and the learning rate comprises: For the Mth division value, the total number of division values ​​is N, and the positive value and the negative value of the division value are obtained, the positive value of the division value is the cumulative sum of all elements from (M-1, M) to (M-1, N) in the penalty result matrix; the negative value of the division value is the opposite of the cumulative sum of all elements from (M, 0) to (M, M-1) in the penalty result matrix; the sum of the positive value and the negative value is multiplied by the learning rate, the product is added to the division value, and the addition result is used as the corrected Mth division value.

7. The method according to claim 1, characterized in that The method for obtaining the score value of the current classification model is: calculate the average value of the last preset third reference value loss values ​​in the loss value record table of the current classification model, weightedly sum the obtained average value and the best loss value corresponding to the current classification model, and use the obtained weighted summation result as the score value of the current classification model.

8. A big data prediction method based on redundant training, characterized in that: include: Obtaining the best classification model by the model training method as described in claims 1 to 7; The training set and the prediction set are classified using the best classification model to obtain multiple training subsets and prediction subsets. The prediction model is redundantly trained based on the obtained multiple training subsets and prediction subsets to obtain data prediction results.

9. The method according to claim 8, characterized in that The redundant training method is: for the ith prediction subset, the prediction model is trained using the i-1th training subset, the ith training subset and the i+1th training subset, and the trained prediction model is used to predict the ith prediction subset.

10. A computer-readable storage medium, characterized in that: A computer program is stored on the medium, and the computer program can be executed by a processor to implement the method according to any one of claims 1 to 9.

Images