Model training method and apparatus, and storage medium

Abstract

The present disclosure provides a model training method and device, and a storage medium, relating to the technical field of machine learning. The model training method comprises: obtaining a test result of testing a classification model by using test sample data, wherein the test sample data comprises at least two types of test samples; determining a sampling ratio of at least one type of sample in the test sample data according to the test result; and sampling in the test sample data according to the sampling ratio, incorporating the sampled samples into a training sample data set, so as to perform the next round of training on the classification model according to the training sample data set. Through such a method, the dynamic change of the sample sampling ratio can be realized, the model training speed is improved, and the accuracy of the trained model is also improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a model training method, device and storage medium. BACKGROUND

[0002] Data identification is the basis of enterprise data asset management and data security control, and is also the focus and difficulty of data security work. Intelligent identification based on machine learning is an important means and technical development direction of data identification. Based on machine learning technology, a machine learning classification model can be trained with a data sample set, and the trained mature model can be deployed to a data control platform to identify and classify static or flowing data.

[0003] Due to the particularity of data, in order to prevent data leakage, some classified sample data is difficult to obtain, resulting in a small amount of these classified sample data, which is referred to as small category sample. Due to the small amount of sample data, the model does not learn enough about this type of data, and the fitting and generalization ability is not enough, the recognition accuracy is low, and classification errors are easy to occur, thereby causing false positives and false negatives. SUMMARY

[0004] One object of the present disclosure is to improve the accuracy of the trained model.

[0005] According to one aspect of some embodiments of the present disclosure, a model training method is provided, comprising: obtaining a test result of testing a classification model with test sample data, wherein the test sample data includes at least two types of test samples; determining a sampling ratio of at least one type of sample in the test sample data according to the test result; and sampling in the test sample data according to the sampling ratio, and incorporating the sampled samples into a training sample data set, so as to perform the next round of training on the classification model according to the training sample data set.

[0006] In some embodiments, determining the sampling ratio of at least one type of sample in the test sample data according to the test result comprises: determining a training result parameter according to the test sample data and the test result, the training result parameter including the total number of test sample data, the number of at least one type of test sample, and the number of samples with correct test results in the corresponding type of test sample; and determining the sampling ratio of at least one type of sample in the test sample data according to the training result parameter.

[0007] In some embodiments, determining the sampling ratio of the at least one type of samples in the test sample data according to the training result parameter comprises: determining a sample balance degree index of the at least one type of samples according to the training result parameter; determining a range to which the sample balance degree belongs according to the sample balance degree and a predetermined balance degree threshold; and adjusting the sampling ratio according to the range to which the sample balance degree belongs, comprising: increasing the sampling ratio in a case where the sample balance degree is located in a first predetermined range, decreasing the sampling ratio in a case where the sample balance degree is located in a second predetermined range, and keeping the sampling ratio unchanged in a case where the sample balance degree is located in a third predetermined range.

[0008] In some embodiments, determining the sampling ratio of the at least one type of samples in the test sample data according to the test result comprises: determining a first sampling ratio of the first type of test samples according to the test result; and sampling in the test sample data according to the sampling ratio and incorporating the sampled samples into the training sample data set, comprising: sampling the first sampling ratio of samples in the first type of test samples; and incorporating the sampled samples into the training sample data set.

[0009] In some embodiments, determining the sampling ratio of the at least one type of samples in the test sample data according to the test result comprises: determining a second sampling ratio of samples with correct test results in the second type of test samples and a third sampling ratio of samples with incorrect test results in the second type of test samples according to the test result; and sampling in the test sample data according to the sampling ratio and incorporating the sampled samples into the training sample data set, comprising: sampling the second sampling ratio of samples with correct test results in the second type of test samples and the third sampling ratio of samples with incorrect test results in the second type of test samples; and incorporating the sampled samples into the training sample data set.

[0010] In some embodiments, determining the training result parameter according to the test result comprises: determining the total number of test sample data and the number of the first type of test samples according to the test result; and determining the sampling ratio of the at least one type of samples in the test sample data according to the training result parameter comprises: determining a first ratio of the number of the first type of test samples to the total number of test samples according to the training result parameter; determining a first sample balance degree index according to the first ratio, a historical sample deviation degree of the first type of test samples and a historical sample fluctuation degree of the first type of test samples; and determining the sampling ratio in the first type of test samples according to the first sample balance degree index.

[0011] In some embodiments, determining the training result parameter according to the test result comprises: determining, according to the test result, a total number of the test sample data and a number of samples in the second type of test sample whose test result is correct; determining the sampling ratio of at least one type of sample in the test sample data according to the training result parameter comprises: determining, according to the training result parameter, a second ratio of the number of samples in the second type of test sample whose test result is correct to the total number of test samples; determining a second sample balance degree index according to the second ratio, a historical sample deviation of the samples in the second type of test sample whose test result is correct, and a historical sample fluctuation of the samples in the second type of test sample whose test result is correct; and determining the sampling ratio in the samples in the second type of test sample whose test result is correct according to the second sample balance degree index.

[0012] In some embodiments, determining the training result parameter according to the test result further comprises: determining, according to the test result, a number of samples in the second type of test sample whose test result is incorrect; and determining the sampling ratio of at least one type of sample in the test sample data according to the training result parameter further comprises: determining, according to the training result parameter, a third ratio of the number of samples in the second type of test sample whose test result is incorrect to the total number of test samples; determining a third sample balance degree index according to the third ratio, a historical sample deviation of the samples in the second type of test sample whose test result is incorrect, and a historical sample fluctuation of the test samples in the second type of test sample whose test result is incorrect; and determining the sampling ratio in the samples in the second type of test sample whose test result is incorrect according to the third sample balance degree index.

[0013] In some embodiments, the training result parameter further comprises a number of samples in the first type of test sample whose test result is correct.

[0014] In some embodiments, the method further comprises: storing the training result parameter as historical data, so as to generate the historical sample deviation and the historical sample fluctuation.

[0015] In some embodiments, the training result parameter further comprises a number of samples in the second type of test sample.

[0016] In some embodiments, the historical sample deviation of the first type of test sample is determined according to a ratio of a number of samples in the first type of test sample whose test result is correct to a number of samples in the first type of test sample in a previous round or rounds of model test, wherein a round of model test is performed after each round of model training is completed; and the historical sample fluctuation of the first type of test sample is determined according to a change ratio of the number of samples in the first type of test sample and a change ratio of the test result correctness in each round of the previous round or rounds of model test.

[0017] In some embodiments, the historical sample deviation of the samples with correct test results in the second type of test samples is determined according to a ratio of the number of the samples with incorrect test results in the second type of test samples to the number of the samples in the second type of test samples in the previous one or more rounds of model testing, wherein one round of model testing is performed after each round of model training is completed; and the historical sample fluctuation of the samples with correct test results in the second type of test samples is determined according to a change ratio of the number of the samples with correct test results in each round of the second type of test samples and a change ratio of the test result correctness in the previous one or more rounds of model testing.

[0018] In some embodiments, the historical sample deviation of the samples with incorrect test results in the second type of test samples is determined according to a ratio of the number of the samples with incorrect test results in the second type of test samples to the number of the samples in the second type of test samples in the previous one or more rounds of model testing, wherein one round of model testing is performed after each round of model training is completed; and the historical sample fluctuation of the samples with incorrect test results in the second type of test samples is determined according to a change ratio of the number of the samples with incorrect test results in each round of the second type of test samples and a change ratio of the test result incorrectness in the previous one or more rounds of model testing.

[0019] In some embodiments, the classification model is a text classification model, and the sample data includes text data and classification result labels.

[0020] According to an aspect of some embodiments of the present disclosure, a model training apparatus is provided, which includes: a test result obtaining unit configured to obtain test results of a classification model tested by using test sample data, wherein the test sample data includes at least two types of test samples; a ratio determining unit configured to determine a sampling ratio of at least one type of sample in the test sample data according to the test results; and a data set determining unit configured to sample from the test sample data according to the sampling ratio, incorporate the sampled samples into a training sample data set, and perform a next round of training of the classification model according to the training sample data set.

[0021] In some embodiments, the ratio determining unit is configured to: determine a training result parameter according to the test sample data and the test results, the training result parameter including a total number of the test sample data, and a number of at least one type of test sample and a number of samples with correct test results in the corresponding type of test sample; and determine the sampling ratio of at least one type of sample in the test sample data according to the training result parameter.

[0022] In some embodiments, the apparatus further includes: a storage unit configured to store the training result parameter as historical data, so as to generate the historical sample deviation and the historical sample fluctuation.

[0023] According to an aspect of some embodiments of the present disclosure, a model training apparatus is provided, including: a memory; and a processor coupled to the memory, the processor configured to perform any of the methods above based on instructions stored in the memory.

[0024] According to an aspect of some embodiments of the present disclosure, a non-transitory computer readable storage medium is provided, having stored thereon computer program instructions, the instructions being executable by a processor to implement the steps of any of the methods above. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this disclosure, illustrate embodiments of the present disclosure and together with the description serve to explain the present disclosure. In the drawings:

[0026] Figure 1 Flowchart of some embodiments of the model training method of the present disclosure.

[0027] Figure 2 Flowchart of some other embodiments of the model training method of the present disclosure.

[0028] Figure 3 Schematic diagram of some embodiments of the model training method of the present disclosure.

[0029] Figure 4 Schematic diagram of some embodiments of the model training apparatus of the present disclosure.

[0030] Figure 5 Schematic diagram of some other embodiments of the model training apparatus of the present disclosure.

[0031] Figure 6 Schematic diagram of some other embodiments of the model training apparatus of the present disclosure. DETAILED DESCRIPTION

[0032] The technical solutions of the present disclosure are described in further detail below with reference to the accompanying drawings and embodiments.

[0033] The flowchart of some embodiments of the model training method of the present disclosure is shown in Figure 1

[0034] ​In step 120, a test result of testing the classification model by test sample data is obtained, wherein the test sample data includes at least two types of test samples. In some embodiments, if the current classification model is trained by the first round of training, the training sample is the initial predetermined training sample data, and the test sample is the initial predetermined test sample data. In some embodiments, the two types of test sample data can include one type of small category sample and another type of non-small category sample, such as common sample data that is relatively easy to obtain.

[0035] In some embodiments, the classification model can be a model for processing text data, which can be enterprise data asset management and data security control related data, and can be static or dynamic data. In some embodiments, the classification model can be a data classification model applied to a data management platform for data identification and classification.

[0036] In some embodiments, the classification label of the sample can be determined by any one of the artificial or related technologies.

[0037] In step 140, the sampling ratio of at least one type of sample in the test sample data is determined according to the test result. In some embodiments, the sampling ratio of one or more types of samples can be determined according to the correct rate distribution of each type of sample. For example, when the test correct rate of the small category sample is low, the sampling ratio of the small category sample can be increased so that more small category samples are included in the subsequent training, thereby improving the classification ability of the classification model for the small category sample.

[0038] In some embodiments, since the test result is the classification result of each test sample itself, the correctness of the test result can be determined according to whether the test result and the classification label of the test sample are the same. Further, the correctness of the test result is counted according to the category to which the test sample belongs. In some embodiments, the training result parameter includes the total number of test sample data, the number of at least one type of test sample, and the number of test samples in the corresponding type of test sample whose test result is correct. Further, the sampling ratio of at least one type of sample in the test sample data is determined according to the training result parameter.

[0039] In some embodiments, the method for determining the sampling ratio according to the test result can be as shown in Figure 2

[0040] ​In step 241, training result parameters are determined according to the test sample data and the test results. In some embodiments, the training result parameters include the total number of the test sample data, and the number of at least one type of test sample and the number of samples of the corresponding type of test sample with correct test results. For example, the training result parameters include the sum of the number of common type test samples and the number of small type test samples, the number of small type test samples, and the number of small type test samples with correct test results. The common type samples are easier to obtain than the small type samples, and the sample space is larger.

[0041] In step 242, a sample balance degree index of at least one type of sample is determined according to the training result parameters. In some embodiments, in order to improve the adjustment ability of the sample balance degree, the balance degree index of each type of sample can be determined, so as to adjust the sampling ratio of each type of sample.

[0042] In some embodiments, in order to further improve the adjustment ability of the sample balance degree, the balance degree index of the test samples with correct and incorrect test results in the same type of test sample can be determined respectively, so as to adjust the sampling ratio of the samples with correct and incorrect test results, and improve the balance degree of the test sample extraction.

[0043] In step 243, the category to which the sample balance degree belongs is determined according to the sample balance degree and the predetermined balance degree threshold. In some embodiments, the predetermined balance degree threshold can be one or more, and the predetermined balance degree threshold divides the balance degree into multiple intervals, each interval corresponds to a sampling ratio value, or corresponds to a sampling ratio processing method (such as increasing a predetermined amount, decreasing a predetermined amount, or keeping unchanged on the basis of the current sampling ratio or the predetermined sampling ratio, etc.).

[0044] In step 244, the sampling ratio is adjusted according to the category to which the sample balance degree belongs.

[0045] In some embodiments, taking two values of the predetermined balance degree threshold as an example, the predetermined balance degree threshold divides the sample balance degree into three intervals. If the sample balance degree is in the first predetermined range, the sampling ratio is increased; if the sample balance degree is in the second predetermined range, the sampling ratio is decreased; and if the sample balance degree is in the third predetermined range, the sampling ratio is kept unchanged. In some embodiments, the above-mentioned increasing amplitude can be a predetermined first adjustment granularity or ratio, and the decreasing amplitude can be a predetermined second adjustment granularity or ratio. The predetermined first adjustment granularity or ratio and the predetermined second adjustment granularity or ratio can be the same or different, and the specific values can be set and adjusted according to practice.

[0046] In some embodiments, the predetermined balance threshold or balance category for different test sample categories can be the same or different. For example, a first predetermined balance threshold and a second predetermined balance threshold can be set for the balance index of the test result correct samples in the small category test samples. If the sample balance is greater than or equal to the first predetermined balance threshold, the evaluation level is excellent, and the sampling ratio of the test result correct samples in the small category test samples can be reduced. If the sample balance is less than the second predetermined balance threshold, the evaluation level is poor, and the sampling ratio of the test result correct samples in the small category test samples can be increased. If the sample balance is less than the first predetermined balance threshold and greater than or equal to the second predetermined balance threshold, the evaluation level is moderate, and the current sampling ratio is kept unchanged.

[0047] In step 160, according to the sampling ratio, the test sample data is sampled and incorporated into the training sample data set, so as to train the model according to the training sample data set in the next round. In some embodiments, the updated training sample data set includes all the training samples used in the last round of training, and the sampled samples of the test sample data.

[0048] In some embodiments, the remaining samples in the test sample data can be used as test samples after the next round of training. In other embodiments, a predetermined proportion of samples can be extracted from the test samples and training samples of the current round as test samples of the next round.

[0049] Based on the method in the above-mentioned embodiments, the sampling ratio of the samples can be adjusted according to the test situation during the training process, so as to realize the dynamic change of the sampling ratio of the samples, improve the degree of balanced distribution of the samples, and help to improve the model training speed and the accuracy of the trained model.

[0050] In some embodiments, any one category of test sample in the test sample data is referred to as the first category of test sample. For the first category of test sample, a sampling ratio can be determined, which is referred to as the first sampling ratio. Then, the first sampling ratio is used to sample the first category of test sample, and the sampled samples are incorporated into the training sample data set, so as to realize the adjustment of the amount or proportion of the first category of test sample in the next round of training sample, and improve the balance of the samples.

[0051] In some embodiments, any one type of sample in the test sample data is referred to as a second type of test sample, and for the second type of test sample, the sampling ratio can be determined respectively for samples with correct test results and samples with incorrect test results, such as determining the sampling ratio of samples with correct test results in the second type of test sample, referred to as a second sampling ratio, and determining the sampling ratio of samples with incorrect test results in the second type of test sample, referred to as a third sampling ratio. Further, a second proportion of samples are extracted from the samples with correct test results in the second type of test sample, and a third proportion of samples are extracted from the samples with incorrect test results in the second type of test sample; the extracted samples are incorporated into the training sample data set, thereby realizing the adjustment of the amount or proportion of samples that can be correctly classified by the classification model in the second type of test sample in the next round of training sample, providing the classification model training with an appropriate proportion of information amount of the second type of sample, and improving the model training efficiency and accuracy.

[0052] A schematic diagram of some embodiments of the model training method of the present disclosure is shown in Figure 3

[0053] The current is the i-th round of training and testing for the classification model, where i is an integer greater than 1. In the i-th round of training, the training sample data generated in the i-1-th round is used to train the classification model, such as the i-th round of training set data shown in the figure. The training sample data includes first type of training sample and second type of training sample, and in the figure, the first type of training sample is taken as an ordinary sample, and the second type of training sample is taken as a small category sample as an example.

[0054] After the i-th round of training 301 is completed, the i-th round of test sample data (such as the i-th round of test set data shown in the figure) is used to test the classification model, and the test result is obtained.

[0055] In 302, the test result is compared with the classification label of the test sample to determine whether the test result is correct, and the training result parameter is determined by statistics.

[0056] In 303, the balance degree index is obtained based on the training result parameter.

[0057] In some embodiments, for the samples with correct test results in the second type of test sample, the balance degree index H a which can be calculated by the following formula.

[0058]

[0059] wherein, Ratio ai is the ratio of the number of samples with correct test results in the second type of test sample to the total number of test samples; ​a historical sample deviation of the samples with correct test results in the second type of test samples; a historical sample fluctuation of the samples with correct test results in the second type of test samples. PA j a ratio of the number of samples with correct test results in the second type of test samples to the number of the second type of test samples in the jth round of training; WA s a number of samples with correct test results in the second type of test samples in the sth round of training NA s a change rate compared with the (s-1)th round, and an accuracy of test results in the second type of test samples MA s a weighted sum of the change rate compared with the (s-1)th round. In some embodiments, WA s may be determined by the following formula.

[0060]

[0061] wherein a and β are weights of the sample number change rate and the accuracy change rate, and in some embodiments, a can be 20% and β can be 80%.

[0062] In some embodiments, to ensure that the historical sample fluctuation and the historical sample deviation of the second type of samples can be obtained, after each round of testing, the number of samples with correct test results in the second type of test samples, the number of the second type of test samples, and the accuracy of test results in the second type of test samples are obtained according to the test results, and the information is stored for use in subsequent rounds of testing.

[0063] In some embodiments, for the samples with incorrect test results in the second type of test samples, the balance degree indicator H b may be obtained by the following formula.

[0064]

[0065] wherein Ratio bi is a ratio of the number of samples with incorrect test results in the second type of test samples to the total number of test samples; a historical sample deviation of the samples with incorrect test results in the second type of test samples; a historical sample fluctuation of the samples with incorrect test results in the second type of test samples. PB j a ratio of the number of samples with incorrect test results in the second type of test samples to the number of the second type of test samples in the jth round of training; WB s a number of samples with incorrect test results in the second type of test samples in the sth round of training NB s a change rate compared with the (s-1)th round, and an error degree of test results in the second type of test samples MB sWeighted sum of the change rate compared with the s-1th round. In some embodiments, WB s The determination can be made by the following formula.

[0066]

[0067] Wherein, a, β are the weights of the sample quantity change rate and the error change rate, respectively, in some embodiments, a can take 20%, and β can take 80%.

[0068] In some embodiments, to ensure that the historical sample fluctuation degree and the historical sample deviation degree of the second type of sample can be obtained, after each round of testing, the number of samples with incorrect test results in the second type of test samples, the number of second type of test samples, and the error degree of the test results in the second type of test samples are obtained according to the test results, and the information is stored for subsequent rounds of testing.

[0069] In some embodiments, for the first type of test sample, the balance degree index H r The calculation can be obtained by the following formula.

[0070]

[0071] Wherein, Ratio ri The ratio of the number of samples with correct test results in the first type of test sample to the total number of test samples; The historical sample deviation degree of the sample with correct test results in the first type of test sample; The historical sample fluctuation degree of the first type of test sample. PR j The ratio of the number of samples with correct test results in the first type of test sample to the number of first type of test samples in the jth round of training; WR s The sample quantity of the first type of test sample in the sth round of training NR s The change rate compared with the s-1th round, and the correctness MR s The weighted sum of the change rate compared with the s-1th round.

[0072] In some embodiments, WR s The determination can be made by the following formula.

[0073]

[0074] Wherein, a, β are the weights of the sample quantity change rate and the error change rate, respectively, in some embodiments, a can take 20%, and β can take 80%.

[0075] In some embodiments, to ensure that the historical sample volatility and the historical sample deviation of the first type of samples can be obtained, after each round of testing, the number of samples in the first type of test samples with correct test results, the number of the first type of test samples, the correctness of the test results in the first type of test samples are obtained according to the test results, and the information is stored for use in subsequent rounds of testing.

[0076] Through the above operations, the balance degree index H a of the samples with correct test results in the second type of test samples is obtained b , the balance degree index H r of the samples with incorrect test results in the second type of test samples is obtained a , and the balance degree index H b of the first type of test samples is obtained r .

[0077] In some embodiments, the same or different predetermined balance degree threshold values are set for the above H a , H b and H r . If greater than or equal to the high threshold value, the corresponding sampling ratio can be reduced, if less than the low threshold value, the corresponding sampling ratio is increased, and if between the two, the sampling ratio remains unchanged. Through the above operations, the sampling ratio a of the samples with correct test results in the second type of test samples, the sampling ratio b of the samples with incorrect test results in the second type of test samples, and the sampling ratio r of the first type of test samples are determined respectively.

[0078] In 304, based on the sampling ratios a, b, r determined above, the test samples are sampled respectively, and combined with the i-th round of training set data, the i+1-th round of training set data is obtained, and the i+1-th round of training is performed. In addition, the part remaining after sampling in the i-th round of test set data will be used as the i+1-th round of test set data. In some other embodiments, a predetermined proportion of data in the i-th round of test set and training set can be extracted as the i+1-th round of test set data.

[0079] Through such a method, the samples of different categories and the samples with different test results can be evenly distributed in the model training process, so as to avoid the deviation of the trained model, speed up the model training speed, and improve the model recognition accuracy. In addition, such a method requires a lower amount of calculation, has a lower deployment cost, and has a more flexible deployment method, which is beneficial to the engineering implementation based on related technologies.

[0080] A schematic diagram of some embodiments of the model training device 40 of the present disclosure is shown in Figure 4 .

[0081] The test result obtaining unit 410 is capable of obtaining a test result of testing the classification model by test sample data, wherein the test sample data comprises at least two types of test samples. In some embodiments, if the current classification model is trained by the first round of training, the training sample is the initial predetermined training sample data, and the test sample is the initial predetermined test sample data. In some embodiments, the two types of test sample data, one of which can be the small category sample mentioned above, and the other of which can be the non-small category sample, such as the common sample data which is relatively easy to obtain.

[0082] The proportion determining unit 420 is capable of determining the sampling proportion of at least one type of sample in the test sample data according to the test result. In some embodiments, the sampling proportion of one or more types of samples can be determined according to the correct rate distribution of each type of sample. For example, when the test correct rate of the small category sample is low, the sampling proportion of the small category sample can be increased, so that more small category samples are included in the subsequent training, thereby improving the classification ability of the classification model for the small category sample.

[0083] The data set determining unit 430 is capable of sampling in the test sample data according to the sampling proportion, and incorporating the training sample data set, so as to train the model in the next round of training according to the training sample data set. In some embodiments, the updated training sample data set comprises all the training samples used in the previous round of training and the sampled samples of the test sample data. In some embodiments, the remaining samples in the test sample data can be used as the test sample after the next round of training. In other embodiments, a predetermined proportion of samples can be extracted from the test sample and the training sample of the current round as the test sample of the next round.

[0084] Such a device can adjust the sampling ratio of the sample according to the test condition in the training process, so as to realize the dynamic change of the sampling ratio of the sample, improve the degree of balanced distribution of the sample, and thus help to improve the model training speed and the accuracy of the trained model.

[0085] In some embodiments, the proportion determining unit 420 is capable of determining whether the test result is correct according to the test result and whether the classification label of the test sample is the same. Further, the correctness of the test result is counted according to the category to which the test sample belongs. In some embodiments, the training result parameter comprises the total number of test sample data, the number of at least one type of test sample, and the number of samples with correct test result in the corresponding type of test sample. Further, the sampling proportion of at least one type of sample in the test sample data is determined according to the training result parameter. Such a device can further process the test result as the data basis for subsequent determination of the sampling ratio, and realize the effective utilization of the test result.

[0086] In some embodiments, as Figure 4As shown, the model training apparatus 40 can further include a storage unit 440, which is capable of storing the training result parameters obtained by the proportion determining unit 420, such as the number of samples with correct test results in the first type of test samples, the number of the first type of test samples, the correctness of the test results in the first type of test samples, the number of samples with correct test results in the second type of test samples, the number of the second type of test samples, the correctness of the test results in the second type of test samples, the number of samples with incorrect test results in the second type of test samples, and the incorrectness of the test results in the second type of test samples, and store the information for use in subsequent rounds of testing.

[0087] Such an apparatus can realize the recording of the information of the pre-test, improve the reference degree of the balance degree index to the historical information, improve the accuracy of the balance degree judgment, also ensure the calculation efficiency of the balance degree index, and further improve the efficiency and accuracy of the model training.

[0088] The structural schematic diagram of one embodiment of the model training apparatus of the present disclosure is shown in Figure 5 The model training apparatus includes a memory 501 and a processor 502. The memory 501 can be a disk, a flash memory, or any other non-volatile storage medium. The memory is used to store the instructions in the corresponding embodiment of the model training method described above. The processor 502 is coupled to the memory 501 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 502 is used to execute the instructions stored in the memory, can realize the dynamic change of the sample sampling ratio, improve the model training speed, and also improve the accuracy of the trained model.

[0089] In one embodiment, as shown in Figure 6 The model training apparatus 600 includes a memory 601 and a processor 602. The processor 602 is coupled to the memory 601 through a BUS bus 603. The model training apparatus 600 can also be connected to an external storage apparatus 605 through a storage interface 604 to call external data, and can also be connected to a network or another computer system (not shown) through a network interface 606. Details are not described here.

[0090] In this embodiment, the data instructions are stored in the memory, and the above instructions are processed by the processor, which can realize the dynamic change of the sample sampling ratio, improve the model training speed, and also improve the accuracy of the trained model.

[0091] In another embodiment, a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the method in the corresponding embodiment of the model training method. Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, device, or computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer usable program code embodied therein.

[0092] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flowcharts and / or block diagrams.

[0093] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flowcharts and / or block diagrams.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flowcharts and / or block diagrams.

[0095] So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

[0096] The methods and apparatus of the present disclosure can be implemented in a number of ways. For example, the methods and apparatus of the present disclosure can be implemented via software, hardware, firmware, or any combination of the three. The above described order of steps for the methods is merely illustrative, and the steps of the methods of the present disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, which includes machine readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the methods according to the present disclosure.

[0097] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present disclosure rather than limit the present disclosure; although the present disclosure has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the specific embodiments of the present disclosure can be modified or equivalent replacements can be made to some technical features without departing from the spirit of the technical solutions of the present disclosure, and all of them should be covered in the technical solution range of the present disclosure.

Claims

1. A model training method, comprising: Obtain test results for a classification model using test sample data, wherein the test sample data includes at least two types of test samples, the classification model is a text data classification model, and the test sample data is text data; Determining the sampling ratio of at least one class of samples in the test sample data based on the test results includes: determining training result parameters based on the test results, the training result parameters including the total number of test sample data and the number of first class test samples; determining a first ratio of the number of first class test samples to the total number of test samples based on the training result parameters; determining a first sample balance index based on the first ratio, the historical sample deviation of the first class test samples, and the historical sample volatility of the first class test samples; determining a first sampling ratio in the first class test samples based on the first sample balance index, wherein the historical sample deviation of the first class test samples is determined based on the ratio of the number of correctly verified samples in the first class test samples to the total number of samples in the first class test samples in one or more previous rounds of model testing, and one round of model testing is performed after each round of model training; the historical sample volatility of the first class test samples is determined based on the change rate of the number of first class test samples and the change rate of test result accuracy in each round of model testing in one or more previous rounds; and Sampling is performed on the test sample data according to the sampling ratio, and the sampled samples are incorporated into the training sample dataset, including: extracting samples of the first sampling ratio from the first type of test samples, and incorporating the extracted samples into the training sample dataset, so as to perform the next round of training on the classification model based on the training sample dataset.

2. The method according to claim 1, wherein, Determining the sampling ratio of at least one type of sample in the test sample data based on the test results includes: Training result parameters are determined based on the test sample data and the test results. These parameters include the total number of test sample data, the number of test samples of at least one class, and the number of samples with correct test results within that class. The sampling ratio for at least one class of samples in the test sample data is determined based on the training result parameters.

3. The method according to claim 2, wherein, Determining the sampling ratio of at least one class of samples in the test sample data based on the training result parameters includes: Based on the training result parameters, determine the sample balance index for at least one class of samples; The range to which the sample balance belongs is determined based on the sample balance degree and a predetermined balance threshold; and Adjusting the sampling ratio according to the range to which the sample balance belongs includes: increasing the sampling ratio when the sample balance is within a first predetermined range, decreasing the sampling ratio when the sample balance is within a second predetermined range, and keeping the sampling ratio unchanged when the sample balance is within a third predetermined range.

4. The method according to any one of claims 1 to 3, wherein, The step of determining the sampling ratio of at least one type of sample in the test sample data based on the test results further includes: determining a second sampling ratio of samples with correct test results in the second type of test samples and a third sampling ratio of samples with incorrect test results in the second type of test samples based on the test results. The step of sampling from the test sample data according to the sampling ratio and incorporating it into the training sample dataset further includes: A second sampling proportion is drawn from the samples with correct test results in the second type of test sample, and a third sampling proportion is drawn from the samples with incorrect test results in the second type of test sample; and The extracted samples are incorporated into the training sample dataset.

5. The method according to claim 4, wherein, The step of determining the training result parameters based on the test results includes: determining the total number of test sample data and the number of samples with correct test results in the second type of test samples based on the test results; Determining the sampling ratio of at least one class of samples in the test sample data based on the training result parameters includes: Based on the training result parameters, determine the second ratio of the number of samples with correct test results in the second type of test samples to the total number of test samples; The second sample balance index is determined based on the second ratio, the historical sample deviation of correctly tested samples in the second type of test samples, and the historical sample volatility of correctly tested samples in the second type of test samples. The historical sample deviation of correctly tested samples in the second type of test samples is determined by the ratio of the number of incorrectly tested samples in the second type of test samples to the total number of samples in the second type of test samples in one or more previous rounds of model testing, with one round of model testing performed after each round of model training. The historical sample volatility of correctly tested samples in the second type of test samples is determined by the change ratio of the number of correctly tested samples in the second type of test samples and the change ratio of the test result accuracy in each round of one or more previous rounds of model testing. The second sampling proportion is determined based on the second sample balance index, which represents the samples with correct test results in the second type of test samples.

6. The method according to claim 5, wherein, The step of determining the training result parameters based on the test results further includes: determining the number of samples with incorrect test results in the second type of test samples based on the test results; The step of determining the sampling ratio of at least one class of samples in the test sample data based on the training result parameters further includes: Based on the training result parameters, determine the third ratio of the number of samples with incorrect test results in the second type of test samples to the total number of test samples; Based on the third ratio, the historical sample deviation of samples with incorrect test results in the second type of test samples, and the historical sample volatility of samples with incorrect test results in the second type of test samples, a third sample balance index is determined; and The third sampling proportion is determined based on the third sample balance index, which indicates that the test results in the second type of test samples are incorrect.

7. The method according to claim 1, wherein, The training result parameters also include the number of samples with correct test results in the first type of test samples; It also includes storing the training result parameters as historical data in order to generate the historical sample deviation and historical sample volatility.

8. The method according to claim 7, wherein, The training result parameters also include the number of samples in the second type of test samples.

9. The method according to claim 6, wherein, The historical sample deviation of samples with incorrect test results in the second type of test samples is determined by the ratio of the number of samples with incorrect test results in the second type of test samples to the total number of samples in the second type of test samples in one or more previous rounds of model testing, wherein one round of model testing is performed after each round of model training; and The historical sample fluctuation of the second type of test samples with incorrect test results is determined by the change rate of the number of samples with incorrect test results and the change rate of the test result error degree in each round of the previous one or more rounds of model testing.

10. The method according to claim 1, wherein, The sample data includes text data and classification result identifiers.

11. A model training device, comprising: The test result acquisition unit is configured to acquire test results of testing the classification model using test sample data, wherein the test sample data includes at least two types of test samples, the classification model is a text data classification model, and the test sample data is text data; A proportion determination unit is configured to determine a sampling proportion for at least one class of samples in the test sample data based on the test results, including: determining training result parameters based on the test results, the training result parameters including the total number of test sample data and the number of first class test samples; determining a first ratio of the number of first class test samples to the total number of test samples based on the training result parameters; determining a first sample balance index based on the first ratio, the historical sample deviation of the first class test samples, and the historical sample volatility of the first class test samples; determining a first sampling proportion in the first class test samples based on the first sample balance index, wherein the historical sample deviation of the first class test samples is determined based on the ratio of the number of correctly verified samples in the first class test samples to the total number of samples in the first class test samples in one or more previous rounds of model testing, and one round of model testing is performed after each round of model training; the historical sample volatility of the first class test samples is determined based on the change rate of the number of first class test samples and the change rate of test result accuracy in each round of model testing in one or more previous rounds; and The dataset determination unit is configured to sample from the test sample data according to the sampling ratio and incorporate the sampled samples into the training sample dataset, including: extracting samples from the first type of test samples at a first sampling ratio and incorporating the extracted samples into the training sample dataset, so as to perform the next round of training on the classification model based on the training sample dataset.

12. The apparatus according to claim 11, wherein, The ratio determination unit is configured as follows: The training result parameters are determined based on the test sample data and the test results. The training result parameters include the total number of test sample data, the number of test samples of at least one class, and the number of samples with correct test results in the corresponding class of test samples. and The sampling ratio for at least one class of samples in the test sample data is determined based on the training result parameters.

13. The apparatus of claim 12, further comprising: The storage unit is configured to store the training result parameters as historical data in order to generate the historical sample deviation and historical sample volatility.

14. A model training device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1 to 10 based on instructions stored in the memory.

15. A non-transitory computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images