A training method and device for a classification model

Abstract

This specification discloses a method and device for training a classification model. After selecting a first number of training samples from an unlabeled sample set for labeling, the characteristic vectors corresponding to the data of each mode in each training sample are determined respectively, and the classification difficulty that characterizes the degree of data association of each mode in each training sample is determined. The classification model is trained in batches in the order of classification difficulty from small to large. If the number of samples for training the classification model does not reach the preset number, other unlabeled samples are classified according to the classification model, and the uncertainty of the classification results of other samples is determined, so that other samples that are difficult to classify can be selected from them to continue labeling, and the classification model is trained again. By repeating the above process, when training the classification model, samples with obvious training classification effects can be preferentially used for training, and the classification model can be trained in batches, so that on the one hand, the training efficiency can be improved, and on the other hand, the number of training samples can be reduced, and the cost of manual labeling can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and device for training a classification model. Background Art

[0002] At present, with the development of computer technology, in order to realize the rational use of data, it is necessary to classify data based on the content of the data.

[0003] In the prior art, a trained classification model is generally used to classify data. Since the classification models for data in different formats, such as character format data or image format data, are not universal, different models are usually used for classification of data in different formats.

[0004] However, with the development of the Internet, more and more data is composed of data in multiple formats. For example, in order to increase the attractiveness of articles, users add pictures or videos to articles when posting comments, which makes the content published by users contain multiple formats. The classification model for a single format has a low accuracy rate for classifying such data and does not take into account the association between data in different formats. Summary of the invention

[0005] This specification provides a classification model training method and device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a classification model training method, including:

[0008] Selecting a first number of training samples from the sample set for manual labeling to determine a training set, wherein each training sample consists of data of at least two modalities;

[0009] For each training sample in the training set, determining the feature vector corresponding to the data of each mode in the training sample;

[0010] Determine the similarity between each feature vector of the training sample, and determine the classification difficulty representing the degree of association of each modality data in the training sample according to each determined similarity;

[0011] The classification model is trained in the order of the classification difficulty of each training sample from small to large;

[0012] If the number of training samples used to train the classification model does not reach a preset number, classifying other samples in the sample set that are not labeled by using the trained classification model;

[0013] According to the order of uncertainty of the classification results of each other sample from large to small, a first number of other samples are determined as training samples for manual labeling and added to the training set, and the classification difficulty of each training sample is re-determined to continue training the classification model until the classification model is trained using the preset number of training samples.

[0014] Optionally, determining the similarity between the feature vectors of the training sample, and determining the classification difficulty representing the degree of association of each modality data in the training sample according to the determined similarities, specifically includes:

[0015] For each modality data's eigenvector, determine the similarity between the eigenvector and other modality data's eigenvectors;

[0016] Determine a similarity mean according to the determined similarities;

[0017] According to the similarity mean, the classification difficulty characterizing the degree of association of each modality data in the training sample is determined.

[0018] Optionally, the classification model is trained in the order of the classification difficulty of each training sample from small to large, specifically including:

[0019] Transferring a second number of training samples from the training set and adding them to the update set in the order of the difficulty of classification of each training sample from small to large;

[0020] Training a classification model according to each training sample in the update set, and determining whether the training set is empty;

[0021] If yes, it is determined that the training of the classification model is completed;

[0022] If not, the second number of training samples are transferred from the training set to the update set in order of classification difficulty from small to large, and the classification model is continuously trained until the training set is empty.

[0023] Optionally, classifying other unlabeled samples in the sample set by using the trained classification model specifically includes:

[0024] Determine other samples in the sample set that are not labeled and selected as training samples;

[0025] For each other sample, input the other sample into the trained classification model, and determine the probability that the other sample belongs to each classification type according to the output result of the classification model;

[0026] Determine the first probability and the second probability according to the order of the probability that the other samples belong to each classification type from large to small;

[0027] Determine a first parameter according to a magnitude relationship between the first probability and a preset first threshold;

[0028] Determine a second parameter according to a magnitude relationship between a difference between the first probability and the second probability and a preset second threshold;

[0029] The uncertainty of the other sample is determined according to the first parameter and the second parameter.

[0030] Optionally, determining a first number of other samples as training samples for manual labeling and adding them to the training set according to the order of uncertainty of the classification results of the other samples from large to small specifically includes:

[0031] Determine the ranking of other samples according to the uncertainty of the classification results from large to small;

[0032] determining the uncertainty of other samples at the first number position in the sequence as a standard value;

[0033] Determine whether the number of other samples whose uncertainty is not less than the standard value is greater than the first number;

[0034] If so, manually annotate other samples whose uncertainty is greater than the standard value and add them to the training set, and select other samples whose uncertainty is equal to the standard value by the difference value from other samples according to the difference between the first number and the number of other samples whose uncertainty is greater than the standard value, manually annotate them, and add them to the training set;

[0035] If not, then determine the first number of other samples in the ranking, manually annotate them, and add them to the training set.

[0036] Optionally, re-determining the classification difficulty of each training sample and continuing to train the classification model until the classification model is trained with the preset number of training samples, specifically includes:

[0037] Re-determining the classification difficulty of each training sample according to the training set, and continuing to train the classification model;

[0038] Determining whether the number of training samples used to train the classification model does not reach the preset number;

[0039] If yes, then reclassify the other samples in the sample set that are not added to the training set according to the classification model obtained by continued training, update the training set according to the uncertainty of the reclassified classification results, and continue to train the classification model until the number of training samples used to train the classification model reaches the preset number;

[0040] If not, it is determined that the classification model training is completed.

[0041] Optionally, before the classification model is trained using the preset number of training samples, the method further includes:

[0042] Determining whether the number of training samples in the sample set is greater than a preset detection threshold, wherein the detection threshold is less than the preset number;

[0043] If so, a fourth number of other samples are selected from other samples in the sample set that are not selected as training samples, and are manually annotated as test samples; a classification result of the test sample is determined according to the classification model; and a classification accuracy is determined according to the classification result and the manual annotation; when the classification accuracy is greater than a preset accuracy threshold, it is determined that the training of the classification model is completed; when the classification accuracy is not greater than the preset accuracy threshold, the classification model continues to be trained;

[0044] If not, continue training the classification model.

[0045] This specification provides an information display device, including:

[0046] A selection module, configured to select a first number of training samples from the sample set for manual labeling to determine a training set, wherein each training sample is composed of data of at least two modalities;

[0047] A feature vector determination module is used to determine, for each training sample in the training set, a feature vector corresponding to the data of each modality in the training sample;

[0048] A classification difficulty determination module is used to determine the similarity between the feature vectors of the training sample, so as to determine the classification difficulty representing the degree of association of each modality data in the training sample according to the determined similarities;

[0049] The initial training module is used to train the classification model in the order of the classification difficulty of each training sample from small to large;

[0050] A pre-classification module, used for classifying other unlabeled samples in the sample set by using the trained classification model if the number of training samples used to train the classification model does not reach a preset number;

[0051] The continuing training module is used to determine a first number of other samples as training samples for manual labeling according to the uncertainty of the classification results of each other sample from large to small, and add them to the training set, and re-determine the classification difficulty of each training sample to continue training the classification model until the classification model is trained with the preset number of training samples.

[0052] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the training method of the above-mentioned classification model is implemented.

[0053] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned classification model training method when executing the program.

[0054] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0055] In the training method of the classification model provided in the present specification, a first number of training samples can be selected from an unlabeled sample set for labeling, and the labeled training samples are added to the training set, and then the feature vectors corresponding to the data of each modality in each training sample are determined respectively, and the classification difficulty characterizing the degree of data association of each modality in each training sample is determined, and then the classification model is trained in batches in the order of classification difficulty from small to large, so as to classify other unlabeled samples according to the classification model. If the number of training samples used to train the classification model does not reach a preset number, the uncertainty of the classification results of other unlabeled samples in the sample set is determined by the trained classification model, so as to select other samples that are difficult to classify from them for further labeling, and the labeled samples are added to the training set, and the training samples in the training set are used again to train the classification model. By repeating the above process, when selecting samples, samples with obvious training classification effects can be preferentially selected, and the classification model can be trained in batches. When training the model, training samples with high data association of each modality can be preferentially used for training, so that on the one hand, the training efficiency can be improved, and on the other hand, the number of training samples can be reduced, reducing the cost of manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0057] Figure 1 A flowchart of a training method for a classification model in this specification;

[0058] Figure 2 A schematic diagram of the training process of a classification model provided in this manual;

[0059] Figure 3 A schematic diagram of a training device for a classification model provided in this specification;

[0060] Figure 4 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] In this specification, the modality of data includes the format of the data. For example, data such as characters, pictures, and videos belong to data of different modalities. Since modality is a professional term in the field of data processing, data such as text, pictures, and videos are referred to as data of different modalities in this specification.

[0063] At present, the existing classification models usually support the classification of data of one modality. When training the classification model, only the data of the corresponding modality will be used as training samples to train the classification model. However, since the data provided by the service provider (website, application software, etc.) mostly comes from users, such as comments posted by users, articles written by users, etc., and this type of data is often composed of multiple modalities in order to express their own demands and attract the attention of others. For example, the comments posted by users contain characters, pictures and videos to indicate their own consumption experience. In this scenario, the existing classification model has a low accuracy rate in classifying this type of data. Therefore, this specification provides a classification model training method to train a classification model specifically for data of multiple modalities.

[0064] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0065] Figure 1 The following is a flow chart of a training method for a classification model in this specification, which specifically includes the following steps:

[0066] S100: Selecting a first number of training samples from a sample set for manual labeling to determine a training set, wherein each training sample consists of data of at least two modalities.

[0067] Since the training of the model usually requires a lot of computing power, it can be performed by a server. Therefore, in this specification, the training method of the classification model can be performed by a server. Of course, the server is specifically a single device, or a distributed server system composed of multiple devices. This specification does not limit it and can be set as needed.

[0068] Specifically, first of all, since the training method of the classification model provided in this specification is used to train a classification model for classifying multimodal data, the server can select a large amount of data including at least two modalities from massive data such as user-generated content and community-generated content to form a sample set, and determine that each data in the sample set is an original sample.

[0069] Among them, if the sample includes two or more of the data modes such as characters, pictures, videos, etc., it is multimodal data.

[0070] Secondly, the server may randomly select the first number of original samples from the sample set according to the preset first number, and manually label the selected first number of original samples, and use the manually labeled original samples as training samples.

[0071] The training set is determined based on the labeled training samples. In the above manner, a certain number of original samples can be selected from a sample set containing a large number of original samples for manual labeling as training samples, and a training set can be determined to train a classification model based on the training samples in the training set, thereby reducing the labeling cost.

[0072] S102: For each training sample in the training set, determine the feature vector corresponding to the data of each mode in the training sample.

[0073] Since each determined training sample includes multiple data modalities, for each training sample, if the similarity of the feature vectors corresponding to the data of each data modality in a training sample is high, it means that the classification difficulty of the training sample is low. If the similarity of the feature vectors corresponding to the data of each data modality in a training sample is low, it means that the classification difficulty of the training sample is high.

[0074] For example, one training sample is a review of gourmet food. The text part is a description of the taste and texture of the gourmet food, and the picture part is a photo of the gourmet food. Obviously, the classification result of this training sample is "gourmet food". Another training sample is a review of the gourmet food that a user posted on an e-commerce platform. The text part is a description of the taste and texture of the gourmet food, and the picture part is a screenshot of the gourmet food logistics. Then it is difficult to determine whether the classification result of this training sample is "gourmet food" or "express logistics", that is, the classification difficulty of this training sample is relatively high. In addition, if the classification results of the data "pointing" of different data modalities in the training sample are similar, it is also easy to label the training samples.

[0075] Therefore, in one or more embodiments of the present specification, in order to improve the training efficiency of the classification model, the classification difficulty that characterizes the degree of association of the data of each data modality in each training sample can be determined. Specifically, for each training sample in the training set, the server inputs the training sample into the pre-trained feature extraction model, and determines the feature vectors corresponding to the data of each data modality in the training sample according to the output result of the feature extraction model.

[0076] Among them, the feature extraction model can be pre-trained, or it can be trained together with models of other business scenarios and task scenarios. There are many mature methods in the prior art for training the feature extraction model, and this specification will not go into details. In terms of model structure, the feature extraction model at least includes a feature extraction layer, a classification layer, and an attention layer. The feature extraction model can adopt a neural network model structure, an encoder-decoder structure, etc. The specific structure of the feature extraction model can be set as needed, and this specification does not limit it. The feature extraction model can extract features from data of multiple modalities, and use feature vectors to represent the data features of each modality.

[0077] Furthermore, in one or more embodiments of the present specification, the dimensions of the feature vectors corresponding to each data modality may be the same or different, as long as the similarity corresponding to the feature vectors of different data modalities can be determined. If the dimensions of the feature vectors corresponding to each modality are different, a variety of methods can be used to unify the dimensions of the feature vectors corresponding to different modalities. For example, a certain data includes sub-data a of modality A, with a feature vector of (1,1,1,1,1,1), and sub-data b of modality B, with a feature vector of (2,2,2,2,2,2,2,2,2), then when determining the similarity of the feature vectors corresponding to sub-data a and sub-data b, the dimension of the feature vector with a higher dimension can be used as the basis, and the feature vector with a lower dimension can be expanded. That is, based on the dimension of the feature vector of sub-data b, the dimension of the feature vector of sub-data a is expanded, and the feature vector of sub-data a is determined to be (1,1,1,1,1,1,0,0,0), and then the similarity is calculated. Of course, it is also possible to determine that the feature vectors output by the feature extraction model are feature vectors of the same dimension.

[0078] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the corresponding device owner or user.

[0079] S104: Determine the similarities between the feature vectors of the training sample, and determine the classification difficulty representing the degree of association of each modality data in the training sample according to the determined similarities.

[0080] In one or more embodiments of the present specification, after each feature vector corresponding to each training sample is determined, the classification difficulty of each training sample can be determined according to each feature vector corresponding to each training sample.

[0081] Specifically, the server determines the similarity between the feature vector and other feature vectors for the feature vector corresponding to each data modality in the training sample, based on the feature vectors corresponding to each data modality in the training sample. According to the similarities determined by the feature vectors corresponding to each data modality in the training sample, the average of the similarities is determined as the similarity mean. And according to the similarity mean, the classification difficulty of the training sample is determined. Of course, according to the determined similarities, the median and mode of each similarity can also be determined as the classification difficulty. How to determine the classification difficulty can be set according to needs, and this manual does not limit it.

[0082] Among them, when determining the similarity of different feature vectors, the similarity of different feature vectors can be determined based on the distance of different feature vectors. For example, the similarity is determined based on the Euclidean distance, Manhattan distance, etc. between different feature vectors. The cosine similarity of different feature vectors can also be determined as the similarity of different feature vectors. The specific method used to determine the similarity of different feature vectors can be set as needed, and this manual does not limit it. When determining the classification difficulty of the training sample based on the similarity mean, the similarity mean can be directly used as the classification difficulty, or the reciprocal of the similarity mean can be determined as the classification difficulty, or the difference between 1 and the similarity mean can be determined as the classification difficulty. How to determine the classification difficulty based on the similarity mean can be set as needed, and this manual does not limit it.

[0083] By adopting the above method, the degree of association (similarity) of the feature vectors corresponding to the data of each modality in the training sample is determined, and the degree of association of the data of each modality in the training sample is determined to determine the classification difficulty of each training sample. The server can train the classification model in a certain order according to the classification difficulty of each training sample, thereby improving the training efficiency of the classification model.

[0084] S106: Training the classification model in order of the classification difficulty of each training sample from small to large.

[0085] In one or more embodiments of the present specification, after determining the classification difficulty of each training sample in the training set, each training sample can be used to train the classification model in the order of the classification difficulty of each training sample from small to large. This achieves the training of the classification model by selecting training samples in the order from easy to difficult, thereby improving the training efficiency.

[0086] Specifically, the server transfers the second number of training samples from the training set to the update set in the order of the classification difficulty from small to large, and trains the classification model according to each training sample in the update set. The training of the classification model according to each training sample in the update set is regarded as a round of training. When the accuracy of the classification model reaches a preset accuracy threshold in a round of training, the round of training is determined to be over. For example, when the accuracy of the classification model reaches 98%, the round of training is over. The initial state of the update set is empty, and the second number is less than the first number.

[0087] Secondly, since the number of training samples in the training set is greater than the number of training samples in the update set, it is necessary to transfer training samples from the training set to the update set multiple times, and each time the update set is updated, the classification model needs to be retrained based on the training samples in the update set. Therefore, after each round of training of the classification model is completed, the server can determine whether the training set is empty. If so, there is no need to update the update set again, and it can be determined that the classification model training is completed.

[0088] If not, the next round of training can be carried out, that is, the second number of training samples are transferred from the training set to the update set in order of classification difficulty from small to large, so as to update the update set, and the classification model is continued to be trained using the training samples in the updated update set until this round of training of the classification model is completed, and then it is re-determined whether the training set is empty.

[0089] In addition, in one or more embodiments of the present specification, since the second number is smaller than the first number, the update set is updated multiple times according to the training set. If the classification model is trained according to the training samples in the update set in each round, the accuracy of the classification model is required to reach the accuracy threshold, and problems such as overfitting and local optimality may occur. Therefore, an accuracy threshold is set in each round of training the classification model, and as the number of training rounds increases, the value of the accuracy threshold is gradually increased so that it achieves the ideal effect just in the last round of training.

[0090] For example, the number of training samples in the training set is 10,000, and the second number is 1,000. In the last round of training, the accuracy of the classification model needs to be 98%. A total of 10 rounds of training are required. The accuracy threshold can be set to 90.8% in the first round of training, 91.6% in the second round, 92.4% in the third round, and so on to 98% in the tenth round.

[0091] By adopting the above method, training samples with less classification difficulty are preferentially used to train the classification model. When the number of training samples is certain, the classification effect of the classification model can be better, the training efficiency is improved, and the training cost is reduced.

[0092] S108: If the number of training samples used to train the classification model does not reach a preset number, classify other unlabeled samples in the sample set using the trained classification model.

[0093] In one or more embodiments of the present specification, since a large number of samples are usually required to train a classification model, if the number of samples used to train the classification model is too small, phenomena such as overfitting of the classification model and poor accuracy of the classification model may occur. Therefore, a preset number is set in the present specification. If the number of training samples used to train the classification model does not reach the preset number, the classification model can continue to be trained.

[0094] Furthermore, since the cost of manual labeling is high, in order to save costs and improve training efficiency, samples with higher training difficulty can be used to train the model. Since the training difficulty of each original sample in the sample set varies, in order to select original samples with higher training difficulty from the sample set as training samples to train the classification model, in one or more embodiments of this specification, the classification model can be used to determine the uncertainty of the classification results of each other sample in the sample set that is not selected as a training sample. The uncertainty of the classification results of each other sample can represent the training difficulty of each other sample.

[0095] Specifically, first, the server determines each original sample in the sample set, and determines each original sample selected from the sample set to the training set as a training sample, and then determines each original sample in the sample set that is not selected as a training sample as other samples. For each other sample in the sample set, the other sample is input into the classification model, and the probability of the other sample belonging to each classification type is determined according to the output result of the classification model.

[0096] Secondly, the probabilities that the other samples belong to each classification type are sorted in order from large to small, and the probability at the first position of the sort is determined as the first probability, and the probability at the second position of the sort is determined as the second probability.

[0097] Finally, a first parameter is determined based on the relationship between the first probability and a preset first threshold, wherein when the first probability is greater than the first threshold, the first parameter is smaller. A second parameter is determined based on the relationship between the difference between the first probability and the second probability and a preset second threshold, wherein when the difference between the first probability and the second probability is greater than the second threshold, the second parameter is smaller. The uncertainty of the classification result of the other sample is determined based on the sum of the first parameter and the second parameter.

[0098] For example, the probability that the original sample A belongs to each classification type is type 1: 0.8, type 2: 0.2. The probability that the original sample B belongs to each classification type is type 1: 0.5, type 2: 0.5, the first threshold is 0.6, the second threshold is 0.4, the larger first parameter and the second parameter are both 1, and the smaller first parameter and the second parameter are both 0. Then for the original sample A, the first probability is 0.8, which is greater than the first threshold, so the first parameter is 0. The first probability is 0.8, the second probability is 0.2, and the difference between the first probability and the second probability is 0.6, which is greater than the second threshold 0.4, so the second parameter is 0. Then the uncertainty of the original sample A is 0+0=0, that is, the classification result of the original sample is very clear. For the original sample B, the first probability is 0.5, which is less than the first threshold, so the first parameter is 1. The first probability is 0.5, the second probability is 0.5, and the difference between the first probability and the second probability is 0, which is less than the second threshold 0.4, so the second parameter is 1. Then the uncertainty of the original sample B is 1+1=1, that is, the classification result of the original sample is unclear.

[0099] By adopting the above method, the uncertainty of the classification results of other samples in the sample set can be determined through the classification model that has been preliminarily trained, so that other samples with higher uncertainty in classification results can be selected from the sample set to train the classification model, thereby improving training efficiency.

[0100] S110: According to the order of uncertainty of the classification results of each other sample from large to small, determine a first number of other samples as training samples for manual labeling, and add them to the training set, re-determine the classification difficulty of each training sample and continue to train the classification model until the classification model is trained with the preset number of training samples.

[0101] In one or more embodiments of the present specification, after determining the uncertainty of the classification results of each other sample in the sample set, other samples can be selected from the sample set to train the classification model in the order of the uncertainty of the classification results of each other sample.

[0102] Specifically, first, the uncertainties of the classification results of the other samples in the sample set are determined, and the ranking of the other samples is determined in descending order, and the uncertainty of the other samples in the first quantitative position in the ranking is determined as the standard value.

[0103] Secondly, according to the uncertainty of the classification results of other samples in the sample set, determine whether the number of other samples whose uncertainty is not less than the standard value is greater than the first number. If so, manually label the other samples whose uncertainty is greater than the standard value, and add the manually labeled other samples as training samples to the training set, and determine the difference between the first number and the number of other samples whose uncertainty is greater than the standard value. Randomly select other samples of the difference number from other samples whose uncertainty is equal to the standard value for manual labeling, and add the manually labeled other samples as training samples to the training set. If not, manually label the other samples before the first number in the ranking, and add the manually labeled other samples as training samples to the training set. Continue to train the classification model according to the training samples in the training set.

[0104] Since it usually costs money to manually label samples, in order to save costs, a preset number is set in this specification. When the number of training samples reaches the preset number, that is, when the number of manually labeled original samples reaches the preset number, the training of the classification model is completed. Therefore, each time the classification model is trained according to each training sample in the training set, the number of training samples in the training set can be judged to determine whether to continue training the classification model.

[0105] Specifically, it is determined whether the number of training samples used to train the classification model does not reach a preset number.

[0106] If so, according to the determined uncertainty of the classification results of each other sample, in the order of the uncertainty of the classification results from large to small, a first number of other samples are selected from each other sample in the sample set, and manually labeled, and the manually labeled other samples are used as training samples and added to the training set to update the training set. For the re-updated training set, repeat steps S102, S104 and S106, and re-judge whether the number of training samples used to train the classification model does not reach the preset number. That is, according to each training sample of the updated training set and the latest trained classification model, the classification difficulty of each training sample is re-determined, and the classification model is trained in batches in the order of the classification difficulty from small to large, and re-judge whether the number of training samples used to train the classification model does not reach the preset number, until the judgment result is no. That is, until the last round of training, the classification model is trained using the preset number of training samples. Among them, the classification model is a classification model trained according to the training set after the last update.

[0107] If not, it is determined that the classification model training is completed.

[0108] In the training method of the classification model provided in the present specification, a first number of training samples can be selected from an unlabeled sample set for labeling, and the labeled training samples are added to the training set, and then the characteristic vectors corresponding to the data of each modality in each training sample are determined respectively, and the classification difficulty characterizing the degree of data association of each modality in each training sample is determined, and then the classification model is trained in batches in order of classification difficulty from small to large, so as to classify other unlabeled samples according to the classification model. If the number of training samples used to train the classification model does not reach the preset number, the uncertainty of the classification results of other unlabeled samples in the sample set is determined by the trained classification model, so as to select other samples that are difficult to classify from them for further labeling, and the labeled samples are added to the training set, and the training samples in the training set are used to train the classification model again. By repeating the above process, the training of the classification model is achieved.

[0109] From the above method, it can be seen that this method can give priority to samples with obvious training classification effects when selecting samples, train the classification model in batches, and give priority to training samples with high data correlation of each modality when training the model. On the one hand, it can improve the training efficiency, and on the other hand, it can reduce the number of training samples and reduce the cost of manual labeling.

[0110] In addition, in one or more embodiments of the present specification, in order to improve training efficiency, when determining the second number, the quotient of the number of training samples and the number of training times can be determined as the second number based on the number of training samples in the training set and the preset number of training times.

[0111] In addition, in one or more embodiments of the present specification, since under normal circumstances, a certain number of samples are required to train the classification model in order to achieve the expected effect of the classification model, when the number of training samples meets certain conditions, the effect of the classification model may reach the expected effect, and in order to save the labeling cost, the accuracy of the classification model may be tested when the number of training samples reaches the detection threshold.

[0112] Specifically, first, determine whether the number of training samples in the training set is greater than a preset detection threshold. If so, select a fourth number of other samples from other samples in the sample set that are not selected as training samples for manual annotation as test samples. Input each test sample into the classification model, and determine the classification accuracy of the classification model based on the output result of the classification model and the manual annotation. When the classification accuracy is greater than the preset accuracy threshold, determine that the classification model training is completed, and when the classification accuracy is not greater than the accuracy threshold, continue to train the classification model. If not, continue to train the classification model.

[0113] Wherein, the detection threshold is less than the preset number. The classification model is the classification model that has completed the latest update. For example, when the detection threshold is 1000 and the number of training samples is 1100, the classification model is the classification model updated according to the 1100 training samples. When the detection threshold is 1000 and the number of training samples is 1500, the classification model is the classification model updated according to the 1500 training samples.

[0114] In addition, in one or more embodiments of the present specification, in order to more strictly determine whether the effect of the classification model meets expectations, when selecting test samples, the uncertainty of the classification results of other samples in the sample set that are not selected as training samples can be determined based on the classification model, and a fourth number of training samples can be selected as test samples in order of uncertainty from large to small.

[0115] In addition, in one or more embodiments of the present specification, in order to more accurately determine the uncertainty of each original sample in the training set, the first number in step S100 may be different from the first number in step S110.

[0116] In addition, in one or more embodiments of the present specification, in order to reduce the labeling cost, when labeling the original samples, the labeling difficulty of each original sample can be determined according to the uncertainty of each original sample, and when performing manual labeling, the total labeling difficulty of the original samples assigned to each user is determined to be consistent, thereby avoiding the situation that the original samples labeled by some users are too difficult, while the original samples labeled by some users are too simple.

[0117] In addition, this manual also provides the training process of the corresponding classification model, such as Figure 2 shown.

[0118] Figure 2 This is a schematic diagram of a training process of a classification model provided in this specification. After starting training, step S200 can be executed, that is, a first number of original samples are selected from the sample set, manually annotated, and added to the training set as training samples. It should be noted that the training set in the initial state is empty.

[0119] After the training set is not empty, step S202 can be executed, that is, the classification model is trained according to each training sample in the training set, which is equivalent to pre-training the classification model. If the number of training samples used to train the classification model does not reach the preset number, step S204 can be executed, that is, according to the pre-trained classification model, the uncertainty of the classification results of each other sample not selected in the sample set is determined, and then step S206 is executed, that is, in the order of the uncertainty of each other sample from large to small, the first number of original samples are selected for manual labeling, and added to the training set as training samples, and the classification model is trained according to each training sample in the updated training set, and then it is determined whether the number of training samples reaches the preset number. If so, step S208 is executed, that is, it is determined that the training is completed. If not, steps S204, S206 and S202 are repeated to continue training the training model.

[0120] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0121] The above is a classification model training method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding classification model training method device, such as Figure 3 shown.

[0122] Figure 3 A schematic diagram of a training device for a classification model provided in this specification specifically includes:

[0123] A selection module 300 selects a first number of training samples from the sample set for manual labeling to determine a training set, wherein each training sample is composed of data of at least two modalities;

[0124] A feature vector determination module 302 is configured to determine, for each training sample in the training set, a feature vector corresponding to data of each modality in the training sample;

[0125] A classification difficulty determination module 304 determines the similarities between the feature vectors of the training sample, and determines the classification difficulty representing the degree of association of each modality data in the training sample according to the determined similarities;

[0126] Initial training module 306, training the classification model in order of the classification difficulty of each training sample from small to large;

[0127] A pre-classification module 308, if the number of training samples used to train the classification model does not reach a preset number, classifies other samples in the sample set that are not labeled by using the trained classification model;

[0128] The training module 310 continues to determine a first number of other samples as training samples for manual labeling according to the order of uncertainty of the classification results of each other sample from large to small, and add them to the training set, and re-determines the classification difficulty of each training sample to continue training the classification model until the classification model is trained with the preset number of training samples.

[0129] Optionally, the module 304 for determining classification difficulty determines, for each modality of data, a feature vector corresponding to the feature vector, and a feature vector corresponding to other modalities of data, and determines a mean similarity based on the determined similarities; and determines, based on the mean similarity, a classification difficulty that characterizes the degree of association between the modal data in the training sample.

[0130] Optionally, the initial training module 306 transfers a second number of training samples from the training set in an ascending order of classification difficulty of each training sample, and adds them to an update set; trains a classification model based on each training sample in the update set, and determines whether the training set is empty; if so, determines that the training of the classification model is completed; if not, transfers the second number of training samples from the training set to the update set in an ascending order of classification difficulty, and continues to train the classification model until the training set is empty.

[0131] Optionally, the pre-classification module 308 determines other samples in the sample set that are not labeled and selected as training samples, inputs the other sample into the trained classification model for each other sample, and determines the probability that the other sample belongs to each classification type based on the output result of the classification model, determines a first probability and a second probability in descending order of the probability that the other sample belongs to each classification type, determines a first parameter based on the relationship between the first probability and a preset first threshold, determines a second parameter based on the relationship between the difference between the first probability and the second probability and a preset second threshold, and determines the uncertainty of the other sample based on the first parameter and the second parameter.

[0132] Optionally, the pre-classification module 308 determines the ranking of each other sample in order of uncertainty of the classification result from large to small, determines the uncertainty of the other samples in the first number position in the ranking as the standard value, and judges whether the number of other samples whose uncertainty is not less than the standard value is greater than the first number. If so, the other samples whose uncertainty is greater than the standard value are manually labeled and added to the training set; and according to the difference between the first number and the number of other samples whose uncertainty is greater than the standard value, from the other samples whose uncertainty is equal to the standard value, select the other samples of the difference number, manually label them, and add them to the training set; if not, determine the first number of other samples in the ranking, manually label them, and add them to the training set.

[0133] Optionally, the continued training module 310 re-determines the classification difficulty of each training sample based on the training set, continues to train the classification model, and determines whether the number of training samples used to train the classification model has not reached the preset number. If so, then based on the classification model obtained by continued training, re-classify other samples in the sample set that have not been added to the training set, update the training set based on the uncertainty of the reclassified classification results, and continue to train the classification model until the number of training samples used to train the classification model reaches the preset number. If not, it is determined that the classification model training is completed.

[0134] Optionally, the continued training module 310 determines whether the number of training samples in the sample set is greater than a preset detection threshold, wherein the detection threshold is less than the preset number. If so, a fourth number of other samples are selected from other samples in the sample set that have not been selected as training samples for manual labeling as test samples, and the classification results of the test samples are determined according to the classification model, and the classification accuracy is determined according to the classification results and manual labeling. When the classification accuracy is greater than a preset accuracy threshold, it is determined that the classification model training is completed. When the classification accuracy is not greater than the preset accuracy threshold, the classification model continues to be trained. If not, the classification model continues to be trained.

[0135] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The training method of the provided classification model.

[0136] This manual also provides Figure 4 The schematic structure diagram of the electronic device shown in FIG. Figure 4As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0137] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0138] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0139] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0140] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0141] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0149] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0151] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0152] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A classification model training method, characterized in that: include: A first number of training samples are selected from a sample set for manual annotation to determine a training set, wherein each training sample is composed of data of at least two modes, and the training samples include two or more of characters, pictures, and videos, i.e., multimodal data; for each training sample in the training set, the feature vectors corresponding to the data of each mode in the training sample are determined respectively; the similarity between the feature vectors of the training sample is determined, so as to determine the classification difficulty that characterizes the degree of association of the data of each mode in the training sample according to the determined similarities; the classification model is trained in the order of the classification difficulty of each training sample from small to large; if the number of training samples used to train the classification model does not reach a preset number, the other samples in the sample set that are not annotated are classified by the trained classification model; according to the order of the uncertainty of the classification results of each other sample from large to small, the first number of other samples are determined as training samples for manual annotation, and added to the training set, the classification difficulty of each training sample is re-determined, and the classification model is continued to be trained until the classification model is trained with the preset number of training samples.

2. The method according to claim 1, characterized in that: Determine the similarity between each feature vector of the training sample to determine the classification difficulty that characterizes the degree of association of each modality data in the training sample based on the determined similarities, specifically including: for the feature vector corresponding to the data of each modality, determine the similarity between the feature vector and the feature vectors corresponding to the data of other modalities; determine the mean similarity based on the determined similarities; determine the classification difficulty that characterizes the degree of association of each modality data in the training sample based on the similarity mean.

3. The method according to claim 1, characterized in that The classification model is trained in the order of the classification difficulty of each training sample from small to large, specifically comprising: transferring a second number of training samples from the training set and adding them to the update set in the order of the classification difficulty of each training sample from small to large; training the classification model according to each training sample in the update set, and judging whether the training set is empty; if so, determining that the training of the classification model is completed; if not, transferring the second number of training samples from the training set to the update set in the order of the classification difficulty from small to large, and continuing to train the classification model until the training set is empty.

4. The method according to claim 1, characterized in that The other samples in the sample set that are not labeled are classified by the trained classification model, specifically including: determining other samples in the sample set that are not labeled and selected as training samples; for each other sample, inputting the other sample into the trained classification model, and determining the probability that the other sample belongs to each classification type according to the output result of the classification model; determining a first probability and a second probability in descending order of the probability that the other sample belongs to each classification type; determining a first parameter according to the relationship between the first probability and a preset first threshold; determining a second parameter according to the relationship between the difference between the first probability and the second probability and a preset second threshold; and determining the uncertainty of the other sample according to the first parameter and the second parameter.

5. The method according to claim 4, characterized in that According to the order of uncertainty of the classification results of each other sample from large to small, a first number of other samples are determined as training samples for manual labeling and added to the training set, specifically including: determining the order of each other sample according to the order of uncertainty of the classification result from large to small; determining the uncertainty of the other samples at the first number position in the order as the standard value; judging whether the number of other samples whose uncertainty is not less than the standard value is greater than the first number; if so, manually labeling the other samples whose uncertainty is greater than the standard value and adding them to the training set, and according to the difference between the first number and the number of other samples whose uncertainty is greater than the standard value, selecting the other samples of the difference number from the other samples whose uncertainty is equal to the standard value, manually labeling them, and adding them to the training set; if not, determining the first number of other samples in the order, manually labeling them, and adding them to the training set.

6. The method according to claim 1, characterized in that Re-determining the classification difficulty of each training sample and continuing to train the classification model until the classification model is trained with the preset number of training samples, specifically comprising: re-determining the classification difficulty of each training sample according to the training set and continuing to train the classification model; judging whether the number of training samples used to train the classification model does not reach the preset number; If yes, then reclassify the other samples in the sample set that are not added to the training set according to the classification model obtained by continued training, update the training set according to the uncertainty of the reclassified classification results, and continue to train the classification model until the number of training samples used to train the classification model reaches the preset number; If not, it is determined that the classification model training is completed.

7. The method according to claim 1, characterized in that Until the classification model is trained with the preset number of training samples, the method also includes: determining whether the number of training samples in the sample set is greater than a preset detection threshold, wherein the detection threshold is less than the preset number; if so, selecting a fourth number of other samples from other samples in the sample set that are not selected as training samples for manual labeling as test samples, determining the classification results of the test samples according to the classification model, and determining the classification accuracy based on the classification results and manual labeling, when the classification accuracy is greater than the preset accuracy threshold, determining that the classification model training is completed, and when the classification accuracy is not greater than the preset accuracy threshold, continuing to train the classification model; if not, continuing to train the classification model.

8. A training device for a data classification model, characterized in that: include: A selection module is used to select a first number of training samples from the sample set for manual annotation to determine the training set, wherein each training sample is composed of at least two modal data, and the training sample includes two or more of characters, pictures, and videos, that is, multimodal data; a feature vector determination module is used to determine, for each training sample in the training set, the feature vectors corresponding to the data of each modality in the training sample; a classification difficulty determination module is used to determine the similarity between the feature vectors of the training sample, so as to determine the classification difficulty representing the degree of association of the modal data in the training sample according to the determined similarities; an initial training module is used to The invention discloses a method for training a classification model in the order of the classification difficulty of each training sample from small to large; a pre-classification module, which is used to classify other samples in the sample set that are not labeled by the trained classification model if the number of training samples used to train the classification model does not reach a preset number; and a continuous training module, which is used to determine a first number of other samples as training samples for manual labeling according to the order of the uncertainty of the classification results of each other sample from large to small, and add them to the training set, and re-determine the classification difficulty of each training sample to continue training the classification model until the classification model is trained with the preset number of training samples.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images