A model optimization method, device and storage medium

Abstract

The embodiment of the present application provides a model optimization method, device and storage medium. In the embodiment of the present application, a number of sample data can be marked in advance to obtain a number of sample sets. On this basis, a target sample set that meets the preset requirements can be selected in batches from the number of sample sets; and based on the selected target sample set, the model to be improved is trained. Accordingly, in this embodiment, the sample data and the marking information can be combined to select the target sample set in batches and add it to the training set. Therefore, a large number of target sample sets can be efficiently mined, thereby giving full play to the value of massive reflow data; target sample sets carrying essential knowledge can also be mined more accurately and comprehensively from the reflow data, thereby optimizing the structure of the training set, improving the quality of the training set, and continuously improving the model performance; in addition, the method of batch mining the target sample set can greatly reduce the number of sample selections for the model to be improved, thereby effectively improving the efficiency of model optimization.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a model optimization method, device and storage medium. Background Art

[0002] Traditional active learning models are usually: A = (C, Q, S, L, U). C is a group or a classifier, L is a labeled sample for training. Q is a query function used to query information with a large amount of information from the unlabeled sample pool U, and S is a supervisor who can label the samples queried by Q. The model starts learning with a small number of initial labeled samples L, selects the most useful samples through a certain query function Q, and asks the supervisor for labels, and then uses the new knowledge to train the classifier and conduct the next round of queries.

[0003] However, the model trained in this way has reached a performance bottleneck and cannot meet the increasingly high model performance requirements. Summary of the invention

[0004] Various aspects of the present application provide a model optimization method, device, and storage medium to improve the performance of a machine learning model.

[0005] The present application provides a model optimization method, including:

[0006] Acquire a plurality of sample sets, wherein the sample sets include sample data and marking information;

[0007] Based on the sample data and labeling information respectively included in the plurality of sample sets, batch-selecting target sample sets that meet preset requirements from the plurality of sample sets;

[0008] The model to be improved is trained according to the target sample set.

[0009] The embodiment of the present application also provides a computing device, including a memory and a processor;

[0010] The memory is used to store one or more computer instructions;

[0011] The processor is coupled to the memory and configured to execute the one or more computer instructions for:

[0012] Acquire a plurality of sample sets, wherein the sample sets include sample data and marking information;

[0013] Based on the sample data and labeling information respectively included in the plurality of sample sets, batch-selecting target sample sets that meet preset requirements from the plurality of sample sets;

[0014] The model to be improved is trained according to the target sample set.

[0015] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the aforementioned model optimization method.

[0016] In the embodiment of the present application, several sample data can be marked in advance to obtain several sample sets. On this basis, target sample sets that meet preset requirements can be selected in batches from the several sample sets; and the model to be improved can be trained based on the selected target sample sets. Accordingly, in this embodiment, the sample data and marking information can be combined to select target sample sets in batches and add them to the training set. This can at least achieve the following technical effects:

[0017] 1. It can efficiently mine a large number of target sample sets, thereby greatly increasing the order of magnitude of the training set and giving full play to the value of massive reflux data;

[0018] 2. It can more accurately and comprehensively mine target sample sets that carry essential knowledge from the reflux data, thereby optimizing the structure of the training set, improving the quality of the training set, and continuously improving model performance;

[0019] 3. The method of batch mining the target sample set can greatly reduce the number of query function queries of the model to be improved, thereby effectively improving the efficiency of model optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 A flow chart of a model optimization method provided for an exemplary embodiment of the present application;

[0022] Figure 2 A logical schematic diagram of a model optimization solution provided for an exemplary embodiment of the present application;

[0023] Figure 3 A logical diagram of another model optimization solution provided as an example in this application;

[0024] Figure 4 The present invention further provides a schematic diagram of the structure of a computing device. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0026] In view of the technical problem that the existing model training scheme has reached the bottleneck of model performance, in some embodiments of the present application: a number of sample data can be labeled in advance to obtain a number of sample sets, on this basis, a target sample set that meets the preset requirements can be selected in batches from the several sample sets; and based on the selected target sample set, the model to be improved is trained. Accordingly, in this embodiment, the sample data and the labeling information can be combined to select the target sample set in batches and add it to the training set. In this way, the value of the massive amount of reflux data can be fully utilized; the structure of the training set can be optimized, the quality of the training set can be improved, and the model performance can be continuously improved; moreover, the efficiency of model optimization can be effectively improved.

[0027] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0028] Figure 1 A flowchart of a model optimization method provided for an exemplary embodiment of the present application. Figure 2 A logical diagram of a model optimization solution provided for an exemplary embodiment of the present application. The model optimization method provided in this embodiment can be executed by a model optimization device, which can be implemented as software or a combination of software and hardware, and the model optimization device can be integrated in a computing device.

[0029] like Figure 1 As shown, the method includes:

[0030] Step 100: Obtain several sample sets, where the sample sets contain sample data and marking information;

[0031] Step 101: based on the sample data and labeling information contained in each of the sample sets, batch-select target sample sets that meet preset requirements from the sample sets;

[0032] Step 102: train the model to be improved according to the target sample set.

[0033] The model optimization method provided in this embodiment can be used to improve the performance of machine learning models, especially active learning models. This embodiment does not limit the application scenario, and the model to be improved can be a machine learning model that may be used in various application scenarios.

[0034] In addition, the model to be improved can be a model that has been optimized using traditional training methods in various application scenarios. In this case, the model optimization method provided in this embodiment can further optimize the model to be improved; of course, the model to be improved can also be the initial model in various application scenarios. In this case, the model optimization method provided in this embodiment can efficiently optimize the model to be improved to the required performance.

[0035] In this embodiment, the models to be improved include but are not limited to the residual network Resnet model, the visual geometry group VGG model, the inceptionV3 model, etc. Of course, these are only exemplary, and the various models mentioned therein can be further subdivided into more types of models. This embodiment does not limit the model architecture used by the model to be improved, and this embodiment is universal to various machine learning models.

[0036] With the development of machine learning technology, machine learning models are introduced into more and more application scenarios to solve classification, regression and other problems. During the application of machine learning models, massive amounts of reflux data will be generated.

[0037] In this embodiment, the application scenario is not limited. In different application scenarios, the model architecture used by the model to be improved may not be exactly the same. The model optimization solution provided in this embodiment is universal for the models used in various application scenarios and can effectively improve the performance of the model. For example, the application scenarios may include: live broadcast scenarios, social scenarios, animation scenarios, e-commerce scenarios, financial scenarios, smart transportation scenarios, medical management scenarios, etc. Of course, this is only exemplary. The model optimization method provided in this embodiment can be applied to various application scenarios using machine learning models.

[0038] In addition, in this embodiment, there is no limitation on the problems to be handled by the model to be improved. In different application scenarios, the problems to be handled by the model to be improved may be different. For example, the model to be improved can be used to handle image classification problems, order allocation problems, and so on.

[0039] In this embodiment, the data pool can be constructed with these return data. Accordingly, in this embodiment, the order of magnitude of the data pool may be millions, tens of millions, or even higher.

[0040] In this embodiment, the reflow data can be directly used as sample data. Of course, in practical applications, the reflow data can also be pre-screened to filter out a portion of the data as sample data in this embodiment. For example, the reflow data in the data pool can be pre-screened from the dimension of information entropy, and some reflow data with insufficient information entropy can be discarded to filter out the sample data in this embodiment. This embodiment does not limit this.

[0041] In this embodiment, the number of sample data may also be in the millions, tens of millions, or even higher.

[0042] On this basis, this embodiment can pre-label the sample data.

[0043] In this embodiment, the marking method is not limited. For example, manual marking or batch marking using a marking model may be used. Of course, any marking method currently or in the future may also be used to mark the sample data.

[0044] In this embodiment, a piece of sample data and its corresponding marking information can be combined into a sample set. As mentioned above, the sample data can be in the order of millions, tens of millions or even higher. Accordingly, in this embodiment, a sample set of millions, tens of millions or even higher can be obtained.

[0045] In different scenarios, the content carried by the labeling information may not be exactly the same. For example, in a classification scenario, the labeling information may be category information, while in a regression scenario, the labeling information may be a regression result, and so on.

[0046] In step 101 , target sample sets that meet preset requirements may be selected in batches from a number of sample sets based on the sample data and labeling information contained in each of the number of sample sets.

[0047] Accordingly, in this embodiment, a target sample set can be selected based on the labeled sample data and added to the training set, wherein the training set refers to a training set used for model training.

[0048] In this embodiment, target sample sets can be selected in batches, and the number of selected target sample sets can also reach millions, tens of millions, or even higher.

[0049] In the process of selecting the target sample set, there is no need to iterate the model to be improved, which can effectively reduce the computing resource consumption caused by model iteration and improve the efficiency of selecting the target sample set.

[0050] Therefore, the model optimization solution provided in this embodiment is particularly suitable for situations where the number of training sets exceeds ten thousand, and can efficiently mine the target sample set of the required number.

[0051] In this embodiment, in the process of selecting the target sample set, attention is paid not only to the sample data itself, but also to the labeling information of the sample data. This can evaluate the quality of the sample data more comprehensively and reasonably, and can optimize the structure of the training set from a global level. In this way, the essence of knowledge in the massive amount of reflux data can be fully mined. Therefore, the sample selection method provided in this embodiment can greatly improve the quality of the training set.

[0052] Based on this, when the number of training sets exceeds 10,000, the model optimization method provided by this embodiment can more accurately and comprehensively mine target sample sets carrying essential knowledge from the reflow data, and obtain higher-quality training sets, compared with the traditional model training method, so as to break through the model performance bottleneck that can be reached by the traditional model training method and further improve the model performance. Moreover, as the reflow data is continuously refreshed, the model optimization solution provided by this embodiment can mine target sample sets carrying new essential knowledge from the new reflow data, thereby continuously improving the model performance.

[0053] In step 102, the target sample set may be input into the model to be improved at one time to train the model to be improved. Of course, the target sample set may also be input into the model to be improved in batches to train the model to be improved, which is not limited in this embodiment.

[0054] In summary, based on the model optimization solution provided in this embodiment, at least the following technical effects can be obtained:

[0055] 1. It can efficiently mine a large number of target sample sets, thereby greatly increasing the order of magnitude of the training set and giving full play to the value of massive reflux data;

[0056] 2. It can more accurately and comprehensively mine target sample sets that carry essential knowledge from the reflux data, thereby optimizing the structure of the training set, improving the quality of the training set, and continuously improving model performance;

[0057] 3. The method of batch mining the target sample set can greatly reduce the number of sample selections for the model to be improved, thereby effectively improving the efficiency of model optimization.

[0058] In the above or below embodiments, the sample values ​​corresponding to the plurality of sample sets may be calculated according to the sample data and labeling information contained in the plurality of sample sets.

[0059] In this embodiment, the sample value can be used to measure the quality of the sample set. The sample set with a larger sample value plays a greater role in the model optimization process.

[0060] In an exemplary implementation, labeling information prediction can be performed on sample data contained in each of the several sample sets to obtain the prediction probability of the prediction result corresponding to each sample data; based on the labeling information contained in each of the several sample sets, the labeling quality parameters corresponding to each of the several sample sets can be determined; based on the labeling quality parameters corresponding to each of the several sample sets and the prediction probability of the prediction result, the sample value corresponding to each of the several sample sets can be calculated.

[0061] The model to be improved can be used to perform the labeling information prediction process. That is, each sample data is input into the model to be improved, so that the model to be improved can be used to output the prediction results corresponding to each sample data respectively; and the prediction probability of the prediction result corresponding to each sample data is obtained.

[0062] For example, in a classification scenario, the model to be improved can be used to output the classification results corresponding to each sample data, and obtain the predicted probability of the classification results corresponding to each sample data.

[0063] In practical applications, if the sample set is represented by {x j ,y j}, j is the number of sample sets, then the prediction probability of the prediction result corresponding to each sample data can be expressed as Formula 1:

[0064]

[0065] Among them, P m (y i |x j ) represents the sample data x j Prediction is labeling information y i The predicted probability of Indicates the labeling information with the highest prediction probability, that is, the prediction result.

[0066] In this implementation, multiple schemes may be used to determine the marking quality parameters corresponding to each of the sample sets.

[0067] Among them, the marking quality parameter is used to measure the quality of marking information.

[0068] In an exemplary solution, the marking quality parameters corresponding to each of the sample sets may be determined according to the prediction results and marking information corresponding to each of the sample data.

[0069] In this exemplary solution, if the prediction result corresponding to the first sample data is consistent with the labeling information, the labeling quality parameter of the sample set where the first sample data is located is determined to be correctly labeled; if the prediction result corresponding to the first sample data is inconsistent with the labeling information, the labeling quality parameter of the sample set where the first sample data is located is determined to be incorrectly labeled. The first sample data is sample data included in any one of the sample sets.

[0070] In practical applications, when the marking quality parameter indicates that the marking is correct, the marking quality parameter may be recorded as 1; and, when the marking quality parameter indicates that the marking is correct, the marking quality parameter may be recorded as 0.

[0071] Following formula 1, the marking quality parameters can be expressed as Based on this, if The marking quality parameter is 1, otherwise, the marking quality parameter is 0.

[0072] Of course, other schemes may also be used to determine the marking quality parameters corresponding to the respective sample sets, such as manual evaluation schemes, which are not limited here.

[0073] Based on this, the marking quality parameters corresponding to several sample sets and the predicted probabilities of the predicted results can be obtained.

[0074] On this basis, in an exemplary calculation scheme: the first factor can be obtained by subtracting the predicted probability of the predicted result corresponding to the first sample set from 1; the marking quality parameter corresponding to the first sample set is used as the second factor; the product of the first factor and the second factor is calculated as the sample value corresponding to the first sample set; wherein the first sample set is any one of several sample sets.

[0075] The above exemplary calculation scheme can be expressed as Formula 2:

[0076]

[0077] Continuing the above description in which the marking quality parameter is expressed as 0 and 1, in this exemplary calculation scheme, when the marking quality parameter is 0, the sample value of the sample set will also be 0. However, when the marking quality parameter is 1, the higher the prediction probability of the prediction result, the lower the sample value of the sample set.

[0078] Of course, the above is only an exemplary implementation of sample value. In this embodiment, based on the sample data and labeling information contained in each of the sample sets, other implementations can also be used to calculate the sample value of the sample set. For example, according to a preset evaluation strategy, the labeling information and the sample data are scored respectively, and the scores of the two aspects are weighted and summed to determine the sample value of the sample set, etc. This embodiment is not limited to this.

[0079] Based on this, the sample values ​​corresponding to several sample sets can be obtained.

[0080] On this basis, in this embodiment, target sample sets that meet preset requirements can be selected in batches from a number of sample sets according to the sample values ​​corresponding to each of the number of sample sets.

[0081] In practical applications, based on the sample values ​​corresponding to each of the several sample sets, a target sample set that meets the preset requirements can be selected from the several sample sets from at least one selection dimension.

[0082] Figure 3 A logical schematic diagram of another model optimization solution provided as an example in the present application.

[0083] refer to Figure 3, at least one selection dimension includes but is not limited to a selection total quantity dimension, a marking quality dimension, or a sample balance dimension.

[0084] The following will take the above-mentioned exemplary dimensions as examples to illustrate the selection schemes of target sample sets under several exemplary dimensions.

[0085] Select the total dimension

[0086] Under the dimension of total selection amount, N sample sets with the largest sample values ​​can be selected from several sample sets as target sample sets, where N is the preset total selection amount.

[0087] Under some optimization requirements, there may be a limit on the total number of sample sets in the training set. In this case, the total number N of target sample sets can be determined, and N sample sets with the largest sample values ​​can be selected from several sample sets as target sample sets.

[0088] Based on this, sample sets with high sample values ​​can be used preferentially for model optimization, so that when the size of the training set is limited, the most essential sample sets can be used for model optimization to achieve higher optimization effects.

[0089] Marking quality dimensions

[0090] Under the dimension of labeling quality, several sample sets can be divided into correctly labeled sample sets and incorrectly labeled sample sets according to their corresponding sample values; according to the preset requirements on the proportion of incorrectly labeled sample sets, some incorrectly labeled sample sets in several sample sets are discarded to obtain a target sample set that meets the requirements on the proportion.

[0091] Since the sample set in this embodiment is pre-labeled, there may be inaccurate labeling. In order to avoid the incorrectly labeled sample set from lowering the model performance, in this embodiment, several sample sets can be cleaned from the labeling quality dimension to adjust the proportion of dirty data. Dirty data is also an incorrectly labeled sample set.

[0092] This can effectively ensure the quality of the target sample set. Especially when the training set is large in magnitude, incorrectly labeled sample sets may lower the performance of the model. This can be avoided by selecting sample sets under the labeling quality dimension.

[0093] Sample Balance Dimension

[0094] In this embodiment, on the one hand, the several sample sets all contain labeling information, and on the other hand, the sample value of the several sample sets is evaluated.

[0095] Based on the first aspect above, in this embodiment, the labeling information can be used as a grouping basis to accurately group the sample set, thereby more comprehensively covering the prediction performance of the model to be improved for different prediction results.

[0096] Under different optimization requirements, the sample balance ratio can be flexibly set, where the sample balance ratio refers to the ratio between sample data under different annotation information. According to the sample balance ratio, the number of target sample sets that can be selected under different annotation information can be determined.

[0097] Therefore, based on the labeling information, it can be ensured that the selected target sample set meets the sample balance ratio, which can greatly optimize the structure of the training set and make the distribution of sample data in the training set more reasonable and balanced; moreover, the coverage scheme for improving the prediction performance of the model for different prediction results can be more flexibly adjusted. For example, for prediction results whose prediction performance does not meet the standard, a higher proportion of the labeling information corresponding to the prediction result can be set in the sample balance ratio, so as to focus on optimizing the prediction performance of the prediction result.

[0098] Furthermore, the above second aspect can be combined: based on the sample values ​​corresponding to the respective sample sets, sample sets matching the number of sample sets required to be selected under the multiple groups of sample sets are selected as target sample sets.

[0099] In practical applications, sample data with different annotation information can be sorted according to sample value, that is, multiple groups of sample sets can be sorted within the group.

[0100] Therefore, according to the preset sample balance ratio, from multiple groups of sample sets, sample sets that match the number of sample sets to be selected and have the largest sample value can be selected as target sample sets.

[0101] Based on this, combining the above two aspects can not only make the distribution of sample data in the training set more reasonable and balanced, but also the selected target sample set has higher quality and carries more essential knowledge.

[0102] In the above, the target sample set selection schemes are respectively described from several exemplary selection dimensions. It should be understood that at least one selection dimension can be used alone or flexibly combined with each other to select the target sample set.

[0103] In an exemplary solution, the sample balance dimension and the marking quality dimension may be combined to select the target sample set.

[0104] According to the aforementioned marking quality dimension, some incorrectly labeled sample sets can be discarded from several sample sets; on this basis, according to the aforementioned sample balance dimension, the target sample set can be further selected from the remaining sample sets. The detailed process can refer to the selection scheme of the aforementioned sample balance dimension and marking quality dimension, which will not be repeated here.

[0105] Of course, the selection order of the two selection dimensions may also be swapped, that is, the selection result of the sample balance dimension is used as the selection basis of the marking quality dimension, which is not limited in this embodiment.

[0106] Similarly, in this embodiment, the combination method, selection order, etc. of each selection dimension can also be flexibly adjusted, which will not be enumerated here, but this should not cause loss of the protection scope of this application.

[0107] In summary, in this embodiment, a new solution for measuring sample value is proposed. Compared with the traditional solution of measuring sample value from dimensions such as information entropy, this can determine the sample value of sample data more comprehensively and reasonably. On this basis, based on the sample values ​​of several sample sets, the target sample set carrying the essence of knowledge can be mined more accurately and comprehensively from the reflux data, thereby optimizing the structure of the training set, improving the quality of the training set, and continuously improving the model performance.

[0108] The following uses the image classification model as the model to be improved to explain the model optimization solution.

[0109] In the e-commerce field, when searching for products by image, it is often necessary to use an image classification model to classify the images input by the user. The categories may include dresses, pants, etc. The accuracy of the classification results will directly affect the quality of the search results.

[0110] Based on traditional model training methods, the bottleneck of model performance has been reached and the model performance can no longer be improved further.

[0111] The model optimization method provided in this embodiment can break through this bottleneck and further improve the model performance.

[0112] Among them, a large number of images input by users can be used as the return data corresponding to the image search product function. Usually, the order of magnitude of the return data may reach tens of millions.

[0113] Based on this, first, the reflow data can be labeled, for example, it can be labeled manually or by a labeling model. And an image sample in the reflow data and its corresponding labeling information can be combined into a sample set. In this way, a sample set of tens of millions can be obtained. For example, a sample set can be [Picture A, Dress]

[0114] Afterwards, a sufficient number of high-quality sample sets can be selected at one time from tens of millions of sample sets.

[0115] In this embodiment, the following three aspects can be used for optimization preparation:

[0116] The image classification model can be used to perform classification prediction on the image samples in each sample set. If the prediction result is inconsistent with the labeling information, such sample set will be marked as dirty data.

[0117] The sample value V of each sample set can also be calculated based on the output probability y corresponding to the above prediction results. For non-dirty data, the larger the output probability y is, the lower the sample value V will be.

[0118] In addition, since the labeling information contained in the sample set actually indicates the category, the number of sample sets under different categories can also be counted.

[0119] Based on the preparations in the above three aspects, sample sets can be selected from several dimensions, such as the total amount of sample sets required, dirty data ratio control, and sample balance ratio.

[0120] For example, if 900,000 to 1,000,000 sample sets are needed, the dirty data ratio is less than 1%, and the sample balance ratio is as close to 3:4:3 as possible, an exemplary sample set selection process may be:

[0121] First, the sample sets are divided into three groups according to the labeling information. The labeling information of the same group of sample sets is consistent. Under each group, the sample sets are sorted according to the sample value.

[0122] Among the three groups of sample sets, according to the sample balance ratio, the sample sets with the highest sample value of 300,000, 400,000 and 300,000 were selected respectively.

[0123] Afterwards, it is determined whether the proportion of dirty data in the selected 1 million sample sets is less than 1%. If not, some dirty data is deleted from the 1 million sample sets to ensure that the proportion of dirty data is less than 1%.

[0124] At this point, we can select about 1 million high-quality sample sets from tens of millions of sample sets at one time. In the process of selecting sample sets, there is no need to iterate the image classification model, so the number of sample selections can be effectively reduced.

[0125] After that, the performance of the image classification model can be optimized by simply inputting the selected sample set of about 1 million into the aforementioned image classification model. Since the selected sample set is evenly distributed and has a high enough training value, the performance of the image classification model can be further improved by breaking through the bottleneck on the basis of the traditional training scheme. In addition, the above sample set selection scheme can be continued to be executed on new reflux data, and the image classification model can be further optimized, so that the performance of the image classification model can be continuously improved.

[0126] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 100 to 102 can be device A; for another example, the execution subject of steps 100 and 101 can be device A, and the execution subject of step 102 can be device B; and so on.

[0127] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 100, 101, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different sample data, sample sets, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to be different types.

[0128] Figure 4 A schematic diagram of a computing device provided by another exemplary embodiment of the present application is shown in FIG. Figure 4 As shown, the computing device includes: a memory 40 and a processor 41 .

[0129] The memory 40 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0130] The memory 40 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0131] The processor 41 is coupled to the memory 40 and is used to execute the computer program in the memory to:

[0132] Obtain several sample sets, which contain sample data and labeling information;

[0133] Based on the sample data and labeling information contained in each of the sample sets, batch select target sample sets that meet preset requirements from the sample sets;

[0134] Train the model to be improved based on the target sample set.

[0135] In an optional embodiment, when the processor 41 selects target sample sets that meet preset requirements in batches from a plurality of sample sets based on the sample data and labeling information contained in each of the plurality of sample sets, it is configured to:

[0136] Calculate the sample values ​​corresponding to the sample sets according to the sample data and labeling information contained in the sample sets;

[0137] According to the sample values ​​corresponding to each of the several sample sets, target sample sets that meet the preset requirements are selected in batches from the several sample sets.

[0138] In an optional embodiment, when the processor 41 calculates the sample value corresponding to each of the plurality of sample sets according to the sample data and the marking information respectively included in the plurality of sample sets, it is configured to:

[0139] Predicting the labeling information of the sample data contained in each of the several sample sets respectively, so as to obtain the prediction probability of the prediction result corresponding to each sample data;

[0140] Determine the marking quality parameters corresponding to each of the sample sets according to the marking information contained in each of the sample sets;

[0141] According to the marking quality parameters corresponding to the sample sets and the predicted probabilities of the predicted results, the sample values ​​corresponding to the sample sets are calculated.

[0142] In an optional embodiment, when the processor 41 determines the marking quality parameters corresponding to the respective sample sets according to the marking information respectively included in the respective sample sets, it is configured to:

[0143] According to the prediction results and marking information corresponding to each sample data, the marking quality parameters corresponding to each of the sample sets are determined.

[0144] In an optional embodiment, when the processor 41 calculates the marking quality parameters of each of the sample sets according to the prediction results and marking information corresponding to each sample data, it is configured to:

[0145] If the prediction result corresponding to the first sample data is consistent with the marking information, it is determined that the marking quality parameter of the sample set where the first sample data is located is correct;

[0146] If the prediction result corresponding to the first sample data is inconsistent with the marking information, it is determined that the marking quality parameter of the sample set where the first sample data is located is a marking error.

[0147] The first sample data is sample data included in any one of the sample sets.

[0148] In an optional embodiment, when the processor 41 calculates the sample values ​​corresponding to the plurality of sample sets according to the marking quality parameters corresponding to the plurality of sample sets and the predicted probabilities of the predicted results, it is configured to:

[0149] Subtracting the predicted probability of the predicted result corresponding to the first sample set from 1 to obtain a first factor;

[0150] Using the marking quality parameter corresponding to the first sample set as the second factor;

[0151] Calculate the product of the first factor and the second factor as the sample value corresponding to the first sample set;

[0152] The first sample set is any one of several sample sets.

[0153] In an optional embodiment, when the processor 41 performs labeling information prediction on the sample data respectively included in the plurality of sample sets to obtain the prediction probability of the prediction result corresponding to each sample data, it is configured to:

[0154] Input the sample data contained in each of the several sample sets into the model to be improved;

[0155] The model to be improved is used to predict the results of the sample data contained in each of the several sample sets to obtain the prediction probability of the prediction result corresponding to each sample data.

[0156] In an optional embodiment, when the processor 41 selects target sample sets that meet preset requirements in batches from a plurality of sample sets according to the sample values ​​corresponding to the plurality of sample sets, it is configured to:

[0157] Based on the sample values ​​corresponding to each of the several sample sets, a target sample set that meets the preset requirements is selected from the several sample sets from at least one selection dimension.

[0158] In an optional embodiment, the at least one selection dimension includes a selection total dimension. When the processor 41 selects a target sample set that meets the preset requirements from among the plurality of sample sets from the at least one selection dimension based on the sample values ​​corresponding to the plurality of sample sets, the processor 41 is configured to:

[0159] From several sample sets, select N sample sets with the largest sample values ​​as target sample sets, where N is the preset total number of selections.

[0160] In an optional embodiment, the at least one selection dimension includes a marking quality dimension. When the processor 41 selects a target sample set that meets the preset requirements from the plurality of sample sets from the at least one selection dimension based on the sample values ​​corresponding to the plurality of sample sets, the processor 41 is configured to:

[0161] According to the sample values ​​corresponding to the sample sets, the sample sets are divided into correctly marked sample sets and incorrectly marked sample sets;

[0162] According to the preset requirement on the proportion of incorrectly labeled sample sets, some incorrectly labeled sample sets in several sample sets are discarded to obtain a target sample set that meets the requirement on the proportion.

[0163] In an optional embodiment, the at least one selection dimension includes a sample balance dimension. When the processor 41 selects a target sample set that meets the preset requirements from among the plurality of sample sets from the at least one selection dimension based on the sample values ​​corresponding to the plurality of sample sets, the processor 41 is configured to:

[0164] Grouping several sample sets according to the labeling information to obtain multiple groups of sample sets, where the sample sets in different groups contain different labeling information;

[0165] According to the preset sample balance ratio, the number of sample sets to be selected under multiple sample sets is determined respectively;

[0166] Based on the sample values ​​corresponding to the respective sample sets, sample sets matching the number of sample sets required to be selected are selected from the multiple groups of sample sets as target sample sets.

[0167] In an optional embodiment, the processor 41 selects sample sets that match the number of sample sets to be selected from multiple groups of sample sets as target sample sets based on the sample values ​​corresponding to the respective sample sets, and is configured to:

[0168] Based on the sample values ​​corresponding to the respective sample sets, the sample sets in the multiple groups are sorted within the groups;

[0169] Under multiple groups of sample sets, select the sample sets that match the number of sample sets to be selected and have the largest sample value as the target sample sets.

[0170] In an optional embodiment, the processor 41 is further configured to:

[0171] If the proportion of incorrectly labeled sample sets in the determined target sample set does not meet the preset proportion requirement;

[0172] Then, some incorrectly labeled sample sets are discarded from the determined target sample set so that the remaining target sample set meets the proportion requirement;

[0173] The incorrectly labeled sample set is determined according to the sample value of the sample set.

[0174] In an optional embodiment, the target sample set for batch selection is on the order of tens of thousands or more.

[0175] In an optional embodiment, when training the model to be improved based on the target sample set, the processor 41 is used to:

[0176] The target sample set is input into the model to be improved at one time to train the model to be improved.

[0177] Further, if Figure 4 As shown, the computing device also includes: a communication component 42, a power supply component 43 and other components. Figure 4 Only some components are shown schematically, and it does not mean that the computing device only includes Figure 4 Components shown.

[0178] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a computing device in the above method embodiment.

[0179] Above Figure 4 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0180] Above Figure 4 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0181] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0182] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0185] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0186] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0187] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0188] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0189] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A model optimization method, characterized in that: include: Acquire a plurality of sample sets, wherein the sample sets include sample data and marking information; Predicting the labeling information of the sample data respectively contained in the plurality of sample sets to obtain the prediction probability of the prediction result corresponding to each sample data; determining the labeling quality parameters corresponding to each of the plurality of sample sets respectively according to the labeling information respectively contained in the plurality of sample sets; Subtracting the predicted probability of the predicted result corresponding to the first sample set from 1 to obtain a first factor; Taking the marking quality parameter corresponding to the first sample set as the second factor; calculating the product of the first factor and the second factor as the sample value corresponding to the first sample set; wherein the first sample set is any one of the plurality of sample sets; Selecting, in batches, target sample sets that meet preset requirements from the sample sets according to the sample values ​​corresponding to the sample sets; The model to be improved is trained according to the target sample set.

2. The method according to claim 1, characterized in that: The determining, according to the marking information respectively included in the plurality of sample sets, the marking quality parameters respectively corresponding to the plurality of sample sets comprises: According to the prediction results and marking information corresponding to each sample data, the marking quality parameters corresponding to each of the plurality of sample sets are determined.

3. The method according to claim 2, characterized in that The calculating the marking quality parameters of each of the plurality of sample sets according to the prediction results and marking information corresponding to each of the sample data includes: If the prediction result corresponding to the first sample data is consistent with the marking information, determining that the marking quality parameter of the sample set where the first sample data is located is correct marking; If the prediction result corresponding to the first sample data is inconsistent with the marking information, determining that the marking quality parameter of the sample set where the first sample data is located is a marking error; The first sample data is sample data included in any one of the sample sets.

4. The method according to claim 1, characterized in that: The performing labeling information prediction on the sample data respectively included in the plurality of sample sets to obtain the prediction probability of the prediction result corresponding to each sample data respectively includes: Inputting the sample data contained in each of the plurality of sample sets into the model to be improved; The model to be improved is used to predict the results of the sample data contained in each of the several sample sets, so as to obtain the prediction probability of the prediction result corresponding to each sample data.

5. The method according to claim 1, characterized in that The step of batch-selecting target sample sets that meet preset requirements from the sample sets according to the sample values ​​corresponding to the sample sets respectively includes: Based on the sample values ​​corresponding to the several sample sets respectively, a target sample set that meets the preset requirements is selected from the several sample sets from at least one selection dimension.

6. The method according to claim 5, characterized in that The at least one selection dimension includes a selection total dimension, and the selecting, based on the sample values ​​corresponding to the plurality of sample sets, a target sample set that meets preset requirements from the plurality of sample sets from the at least one selection dimension includes: From the several sample sets, select N sample sets with the largest sample values ​​as target sample sets, where N is a preset total number of selections.

7. The method according to claim 5, characterized in that The at least one selection dimension includes a marking quality dimension, and the selecting, based on the sample values ​​corresponding to the plurality of sample sets, a target sample set that meets preset requirements from the plurality of sample sets from the at least one selection dimension includes: According to the sample values ​​corresponding to the sample sets, the sample sets are divided into correctly marked sample sets and incorrectly marked sample sets; According to the preset requirement on the proportion of incorrectly marked sample sets, some incorrectly marked sample sets in the plurality of sample sets are discarded to obtain a target sample set that meets the requirement on the proportion.

8. The method according to claim 5, characterized in that The at least one selection dimension includes a sample balance dimension, and the selecting, based on the sample values ​​corresponding to the plurality of sample sets, a target sample set that meets preset requirements from the plurality of sample sets from the at least one selection dimension includes: The plurality of sample sets are grouped according to the labeling information to obtain a plurality of groups of sample sets, wherein the sample sets in different groups contain different labeling information; According to a preset sample balance ratio, determining the number of sample sets to be selected from the multiple groups of sample sets respectively; Based on the sample values ​​corresponding to the plurality of sample sets, sample sets matching the number of sample sets required to be selected are selected from the plurality of sample sets as target sample sets.

9. The method according to claim 8, characterized in that The selecting, based on the sample values ​​corresponding to the plurality of sample sets, sample sets that match the number of sample sets to be selected from the plurality of sample sets as target sample sets, respectively, includes: Based on the sample values ​​corresponding to the plurality of sample sets, respectively, the plurality of sample sets are sorted within the groups; Among the multiple groups of sample sets, sample sets that match the number of sample sets to be selected and have the largest sample value are selected as target sample sets.

10. The method according to claim 8, characterized in that Also includes: If the proportion of incorrectly labeled sample sets in the determined target sample set does not meet the preset proportion requirement; Then, some incorrectly labeled sample sets are discarded from the determined target sample set, so that the remaining target sample set meets the proportion requirement; The incorrectly marked sample set is determined according to the sample value of the sample set.

11. The method according to claim 1, characterized in that: The target sample set for batch selection is on the order of tens of thousands or more.

12. The method according to claim 11, characterized in that The step of training the model to be improved based on the target sample set includes: The target sample set is input into the model to be improved at one time to train the model to be improved.

13. The method according to claim 1, characterized in that The model to be improved includes one or more of a residual network Resnet model, a visual geometry group VGG model or an inceptionV3 model.

14. A computing device, characterized in that including memory and processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and configured to execute the one or more computer instructions for: Acquire a plurality of sample sets, wherein the sample sets include sample data and marking information; Predicting the labeling information of the sample data respectively contained in the plurality of sample sets to obtain the prediction probability of the prediction result corresponding to each sample data; determining the labeling quality parameters corresponding to each of the plurality of sample sets respectively according to the labeling information respectively contained in the plurality of sample sets; Subtracting the predicted probability of the predicted result corresponding to the first sample set from 1 to obtain a first factor; Taking the marking quality parameter corresponding to the first sample set as the second factor; calculating the product of the first factor and the second factor as the sample value corresponding to the first sample set; wherein the first sample set is any one of the plurality of sample sets; Selecting, in batches, target sample sets that meet preset requirements from the sample sets according to the sample values ​​corresponding to the sample sets; The model to be improved is trained according to the target sample set.

15. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the model optimization method according to any one of claims 1 to 13.

Images