Classification and training method, device and equipment of classification model, and medium

Abstract

The present disclosure relates to a classification method and a training method of a classification model, an apparatus, a device and a medium. The classification method comprises: obtaining a to-be-classified object; obtaining a first category and intermediate features of the to-be-classified object by using a classification branch model in a classification model based on the to-be-classified object; obtaining a second category by using a verification branch model in the classification model based on the first category and the intermediate features; and determining the second category as the category of the to-be-classified object. According to the classification method and the training method of the classification model, the verification branch model is added to the traditional classification model, so that the classification result of the classification branch model is verified again, the precision rate of the classification branch model is further improved, and the classification accuracy is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a method, apparatus, device, and medium for classifying and training classification models. Background Technology

[0002] Currently, with the rapid development of artificial intelligence technology, more and more application scenarios are calling classification models to classify target objects. The classification accuracy of the classification model directly affects the subsequent actions performed on the target object. For example, in scenarios such as precise video push or video content review, the accuracy of video classification can directly affect the push effect or review efficiency. Therefore, it is necessary to continuously improve the classification accuracy of the classification model in order to improve the classification accuracy of target objects in various scenarios. Summary of the Invention

[0003] This disclosure provides a classification method, apparatus, device, and medium for training classification models, in order to at least solve the problems in the aforementioned related technologies.

[0004] According to a first aspect of the present disclosure, a classification method is provided, comprising: obtaining an object to be classified; based on the object to be classified, using a classification branch model in a classification model to obtain a first category and intermediate features of the object to be classified, wherein the first category is at least one category of the object to be classified estimated by the classification branch model, and the intermediate features are features output by an intermediate layer of the classification branch model; based on the first category and the intermediate features, using a verification branch model in the classification model to obtain a second category of the object to be classified, wherein the second category is at least one category obtained after verifying the first category by the verification branch model; and determining the second category as the category of the object to be classified.

[0005] Optionally, the verification branch model includes a category feature acquisition module, a feature interaction module, and a classification module. The step of obtaining the second category of the object to be classified based on the first category and the intermediate features using the verification branch model in the classification model includes: obtaining category features of the first category using the category feature acquisition module based on the first category; obtaining intermediate features with configured weights using the feature interaction module based on the category features and the intermediate features, where the weights represent the correlation between the category features and the intermediate features; and obtaining the second category of the object to be classified using the classification module based on the intermediate features with configured weights.

[0006] Optionally, obtaining the category features of the first category based on the first category using the category feature acquisition module includes: performing one-hot encoding on the first category to obtain a category list of the first category; querying the category features from a dictionary matrix according to the category list, wherein the dictionary matrix is ​​a vector representation in a dimensional space, the number of rows of the dictionary matrix is ​​the same as the number of columns of the sparse matrix, and the number of columns of the dictionary matrix is ​​the same as the dimension of the intermediate feature.

[0007] Optionally, obtaining the first category of the object to be classified based on the object to be classified using the classification branch model in the classification model includes: obtaining at least one first category probability of the object to be classified using the classification branch model based on the object to be classified; when the at least one first category probability is greater than or equal to the corresponding first threshold, determining at least one category corresponding to the at least one first category probability as the first category of the object to be classified.

[0008] Optionally, obtaining the second category of the object to be classified based on the first category and the intermediate features using the validation branch model in the classification model includes: obtaining at least one second category probability of the object to be classified using the validation branch model based on the first category and the intermediate features; when the at least one second category probability is greater than or equal to the corresponding second threshold, determining at least one category corresponding to the at least one second category probability as the second category of the object to be classified.

[0009] According to a second aspect of the present disclosure, a method for training a classification model is provided, comprising: acquiring a sample of an object to be classified, the sample of the object to be classified corresponding to a true classification label; based on the sample of the object to be classified, obtaining a first estimation result and intermediate features of the sample of the object to be classified through a classification branch model in the classification model, the first estimation result representing at least one category of the sample of the object to be classified estimated by the classification branch model, the intermediate features being features output by an intermediate layer of the classification branch model; based on the true classification label and the intermediate features, obtaining a second estimation result of the sample of the object to be classified through a validation branch model in the classification model, the second estimation result representing at least one category of the sample of the object to be classified estimated by the validation branch model; calculating a loss based on the first estimation result, the second estimation result, and the true classification label; and training the classification model by adjusting the model parameters of the classification branch model and the validation branch model according to the loss.

[0010] Optionally, the verification branch model includes a category feature acquisition module, a feature interaction module, and a classification module. The step of obtaining a second estimation result of the object sample to be classified based on the true classification label and the intermediate features through the verification branch model includes: acquiring the category features of the true classification label using the category feature acquisition module; obtaining intermediate features with configured weights using the feature interaction module based on the category features and the intermediate features; and obtaining a second estimation result of the object sample to be classified using the classification module based on the intermediate features with configured weights.

[0011] Optionally, the step of obtaining the category features of the real classification labels using the category feature acquisition module based on the real classification labels includes: performing one-hot encoding on the real classification labels to obtain a category list of the real classification labels; and querying the category features from a dictionary matrix according to the category list, wherein the dictionary matrix is ​​a vector representation in a dimensional space, the number of rows of the dictionary matrix is ​​the same as the number of columns of the category list, and the number of columns of the dictionary matrix is ​​the same as the dimension of the intermediate features.

[0012] Optionally, calculating the loss based on the first estimation result, the second estimation result, and the true classification label includes: calculating a first loss based on the first estimation result and the true classification label; calculating a second loss based on the second estimation result and the true classification label; and obtaining the loss based on the first loss and the second loss.

[0013] Optionally, obtaining the loss based on the first loss and the second loss includes: taking the weighted sum of the first loss and the second loss as the loss, wherein the weight of the first loss is greater than the weight of the second loss.

[0014] According to a third aspect of the present disclosure, a classification apparatus is provided, comprising: an object acquisition unit configured to: acquire an object to be classified; a first category acquisition unit configured to: obtain a first category and intermediate features based on the object to be classified using a classification branch model in a classification model, wherein the first category is at least one category of the object to be classified estimated by the classification branch model, and the intermediate features are features output by an intermediate layer of the classification branch model; a second category acquisition unit configured to: obtain a second category of the object to be classified based on the first category and the intermediate features using a verification branch model in the classification model, wherein the second category is at least one category obtained after verifying the first category by the verification branch model; and a category determination unit configured to: determine the second category as the category of the object to be classified.

[0015] Optionally, the verification branch model includes a category feature acquisition module, a feature interaction module, and a classification module. The second category acquisition unit is configured to: obtain the category features of the first category using the category feature acquisition module based on the first category; obtain intermediate features with configured weights using the feature interaction module based on the category features and the intermediate features, where the weights represent the degree of correlation between the category features and the intermediate features; and obtain the second category of the object to be classified using the classification module based on the intermediate features with configured weights.

[0016] Optionally, the second category acquisition unit is configured to: perform one-hot encoding on the first category to obtain a category list of the first category; and query the category feature from a dictionary matrix according to the category list, wherein the dictionary matrix is ​​a vector representation in a dimensional space, the number of rows of the dictionary matrix is ​​the same as the number of columns of the category list, and the number of columns of the dictionary matrix is ​​the same as the dimension of the intermediate feature.

[0017] Optionally, the first category acquisition unit is configured to: based on the object to be classified, use the classification branch model to obtain at least one first category probability of the object to be classified; when the at least one first category probability is greater than or equal to the corresponding first threshold, determine at least one category corresponding to the at least one first category probability as the first category of the object to be classified.

[0018] Optionally, the second category acquisition unit is configured to: based on the first category and the intermediate features, use the verification branch model to obtain at least one second category probability of the object to be classified; when the at least one second category probability is greater than or equal to the corresponding second threshold, determine at least one category corresponding to the at least one second category probability as the second category of the object to be classified.

[0019] According to a fourth aspect of the present disclosure, a training apparatus for a classification model is provided, comprising: a sample acquisition unit configured to: acquire a sample of an object to be classified, the sample of the object to be classified corresponding to a true classification label; a first estimation unit configured to: obtain a first estimation result and intermediate features of the sample of the object to be classified based on the sample of the object to be classified, through a classification branch model in the classification model, wherein the first estimation result represents at least one category of the sample of the object to be classified estimated by the classification branch model, and the intermediate features are features output by an intermediate layer of the classification branch model; a second estimation unit configured to: obtain a second estimation result of the sample of the object to be classified based on the true classification label and the intermediate features, through a validation branch model in the classification model, wherein the second estimation result represents at least one category of the sample of the object to be classified estimated by the validation branch model; a loss calculation unit configured to: calculate a loss based on the first estimation result, the second estimation result, and the true classification label; and a model training unit configured to: train the classification model by adjusting the model parameters of the classification branch model and the validation branch model according to the loss.

[0020] Optionally, the verification branch model includes a category feature acquisition module, a feature interaction module, and a classification module. The second estimation unit is configured to: acquire the category features of the real classification label using the category feature acquisition module based on the real classification label; obtain intermediate features with configured weights using the feature interaction module based on the category features and the intermediate features; and obtain the second estimation result of the sample to be classified using the classification module based on the intermediate features with configured weights.

[0021] Optionally, the second estimation unit is configured to: perform one-hot encoding on the true classification labels to obtain a list of categories of the true classification labels; and query the category features from a dictionary matrix based on the category list, wherein the dictionary matrix is ​​a vector representation in a dimensional space, the number of rows of the dictionary matrix is ​​the same as the number of columns of the sparse matrix, and the number of columns of the dictionary matrix is ​​the same as the dimension of the intermediate features.

[0022] Optionally, the loss calculation unit is configured to: calculate a first loss based on the first estimation result and the true classification label; calculate a second loss based on the second estimation result and the true classification label; and obtain the loss based on the first loss and the second loss.

[0023] Optionally, the model training unit is configured to use the weighted sum of the first loss and the second loss as the loss, wherein the weight of the first loss is greater than the weight of the second loss.

[0024] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to execute a classification method according to a first aspect of the present disclosure or a training method for a classification model according to a second aspect of the present disclosure.

[0025] According to a sixth aspect of the present disclosure, a computer-readable storage medium for storing instructions is provided, which, when executed by at least one processor, causes the at least one processor to perform a classification method according to a first aspect of the present disclosure or a training method for a classification model according to a second aspect of the present disclosure.

[0026] According to a seventh aspect of the present disclosure, a computer program product is provided, wherein the instructions in the computer program product are executable by a processor of a computer device to perform a classification method according to a first aspect of the present disclosure or a training method for a classification model according to a second aspect of the present disclosure.

[0027] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0028] According to the classification and classification model training method, apparatus, equipment and medium disclosed herein, a verification branch model is added to the traditional classification model, thereby realizing secondary verification of the classification results of the classification branch model, which can further improve the precision and recall of the classification branch model and improve the classification accuracy.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0031] Figure 1 This is a schematic diagram illustrating the structure of a classification model according to an exemplary embodiment of the present disclosure.

[0032] Figure 2 This is a flowchart illustrating a classification method according to an exemplary embodiment of the present disclosure.

[0033] Figure 3 This is a schematic diagram illustrating the structure of a classification branch model according to an exemplary embodiment of the present disclosure.

[0034] Figure 4 This is a flowchart illustrating a training method for a classification model according to an exemplary embodiment of the present disclosure.

[0035] Figure 5 This is an overall schematic diagram illustrating the training and application of a classification model according to exemplary embodiments of the present disclosure.

[0036] Figure 6 This is a block diagram illustrating a classification apparatus according to an exemplary embodiment of the present disclosure.

[0037] Figure 7 This is a block diagram illustrating a training apparatus for a classification model according to an exemplary embodiment of the present disclosure.

[0038] Figure 8 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0039] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0040] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0041] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0042] Currently, with the rapid development of artificial intelligence technology, more and more application scenarios are calling classification models to classify target objects. The classification accuracy of the classification model directly affects the subsequent actions performed on the target object. For example, in scenarios such as precise video push or video content review, the accuracy of video classification can directly affect the push effect or review efficiency. Therefore, it is necessary to continuously improve the classification accuracy of the classification model.

[0043] One relevant technique involves an image multi-classification model: after extracting image features from the original image using a multi-layer convolutional neural network, multiple fully connected layers (FC) are applied for feature compression and integration, and finally, a FC layer with C classes is applied to achieve multi-class classification. However, this image multi-classification model suffers from inaccurate classification, resulting in suboptimal classification performance when applied to specific scenarios.

[0044] To improve classification accuracy, this disclosure proposes a classification method, apparatus, device, and medium for training classification models. It adds a validation branch model to the traditional classification model, thereby enabling secondary validation of the classification results and further improving the precision and recall of the classification branch model, ultimately enhancing classification accuracy. The following will refer to… Figures 1 to 8 The specific description includes a classification method and apparatus, and a method and apparatus for training a classification model, according to exemplary embodiments of the present disclosure.

[0045] Figure 1 This is a schematic diagram illustrating the structure of a classification model according to an exemplary embodiment of the present disclosure.

[0046] Here, the classification model can be used to classify images, such as videos, pictures, and text, without limitation. The classification can be multi-label classification, for example, multi-label classification of videos. (See reference...) Figure 1 According to the classification model 100 of this disclosure, there are a classification branch model 101 and a verification branch model 102. The object to be classified (video or image, etc.) is preprocessed (or not preprocessed) and then input into the classification branch model 101 to obtain the estimated category of the object to be classified. In related technologies, the estimated category is directly used as the final classification result. However, the estimated category usually deviates from the true category of the object to be classified (i.e., there is inaccurate classification). Therefore, this disclosure sets up a verification branch model 102 to verify the estimated category. Categories with low accuracy (e.g., prediction probability less than the preset threshold of the verification branch model 102) can be removed from the estimated category, while categories with high accuracy (e.g., prediction probability greater than or equal to the preset threshold of the verification branch model 102) are output as the final classification result.

[0047] Since the classification model disclosed herein includes a classification branch model and a verification branch model, the estimated category output by the classification branch model can be verified twice through the verification branch model, thereby improving the classification accuracy of the classification model and thus improving the accuracy of classifying target objects in various application scenarios.

[0048] Figure 2This is a flowchart illustrating a classification method according to an exemplary embodiment of the present disclosure. Here, the classification method is performed based on the aforementioned classification model, which includes a classification branch model and a verification branch model.

[0049] Reference Figure 2 In step 201, the object to be classified can be obtained. Here, the object to be classified can be a video, image, or text, etc., without restriction. Classification refers to multi-label classification of the object to be classified; that is, the object to be classified may have multiple categories. For example, an image with a nautical theme may be categorized as "the sea" and "ships." Multi-label classification can be performed on videos; in this scenario, the video to be classified can be obtained.

[0050] In step 202, based on the object to be classified, a first category and intermediate features of the object can be obtained using a classification branch model in the classification model. Here, the first category is at least one category of the object to be classified estimated by the classification branch model, and the intermediate features are features output by the intermediate layer of the classification branch model.

[0051] According to an exemplary embodiment of this disclosure, based on an object to be classified, a classification branch model can be used to obtain at least one first category probability of the object to be classified. When the probability of at least one first category is greater than or equal to the corresponding first threshold, the at least one category corresponding to the probability of at least one first category is determined as the first category of the object to be classified. Here, the first thresholds corresponding to the at least one first category are the same or different. Specifically, the classification branch model includes multiple output channels, each output channel corresponding to a category. Each output channel includes multiple predicted probabilities of the category corresponding to the channel predicted by the classification branch model. The maximum probability among them is taken as the predicted probability of the output channel. When the predicted probability is greater than or equal to the first threshold of the output channel, the category corresponding to the output channel can be determined as the category of the object to be classified. The classification branch model can be any classification model in related technologies. For example, in a scenario of multi-label classification of video, the structure of the classification branch model can be as follows: Figure 3 As shown. Figure 3 This is a schematic diagram illustrating the structure of a classification branching model according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 3 First, M global images are sampled from the video to be classified, where M is a natural number greater than or equal to 1. If M frames cannot be sampled, the video's cover frame image is used to supplement the sampled images. Here, the number of sampled frames is related to the structure of the classification branch model. Then, the M frames are input into the image classification network to extract image features. Here, the image classification network can be Inception-V3, ResNet-50d, or EfficientNet-B3, etc., without restriction. Figure 3In this example, EfficientNet-B3 is used. Assuming the feature dimension is D, the image feature dimension extracted by EfficientNet-B3 is M*D. Next, the M*D feature is input into the transformer structure for feature interaction to obtain D-dimensional content features (i.e., intermediate features). Finally, the D-dimensional content features are processed through multiple fully connected layers to obtain at least one category of the video (i.e., the first category).

[0052] In step 203, a second category can be obtained based on the first category and the intermediate features using the validation branch model in the classification model. Here, the second category is at least one category obtained after validating the first category using the validation branch model.

[0053] According to exemplary embodiments of this disclosure, the verification branch model may include a category feature acquisition module, a feature interaction module, and a classification module. First, based on a first category, the category feature acquisition module can obtain the category features of the first category. In some embodiments, the category feature acquisition module can perform one-hot encoding on the first category to obtain a category list of the first category; and query the category features from a dictionary matrix according to the category list, wherein the dictionary matrix is ​​a vector representation in dimensional space, the number of rows of the dictionary matrix is ​​the same as the number of columns of the sparse matrix, and the number of columns of the dictionary matrix is ​​the same as the dimension of the intermediate features obtained through the intermediate layer of the classification branch model. Here, by performing one-hot encoding on the first category and querying the trained dictionary matrix through the encoded category list, the first category accurately represented in dimensional space can be obtained, thereby improving the verification accuracy of the verification branch model. Specifically, the first category can first be one-hot encoded, for example, as (0, 1, 2, ..., T-1), with a total of T categories. Then, the embeddings (feature vectors) at the corresponding positions in the dictionary matrix (T, D) are obtained. For example, if the encoding is 0, the first row of the dictionary matrix's embedding is taken to obtain an array of length D. This process is repeated to obtain the category features of all categories in the first category. Here, the dictionary matrix, after training, can largely represent the first category in the dimensional space. After obtaining the category features of the first category, based on these category features and the intermediate features obtained through the intermediate layer of the classification branch model in step 202, the feature interaction module can be used to obtain intermediate features with configured weights. Here, the weights represent the degree of correlation between the category feature and the intermediate features. Specifically, feature interaction can be achieved through a multi-head attention mechanism. The categorical feature is used as Q (Query) in Attention(Q,K,V), and the intermediate features are used as K (Key) and V (Value). Similarity weights between the categorical feature and the intermediate feature are obtained and then added to the intermediate feature to obtain a weighted intermediate feature. This means different categorical features can activate different intermediate features. If there is an inaccurately classified category in the first category, its similarity to the intermediate feature is low, and its corresponding weight is also low; conversely, its weight is high. Finally, based on the weighted intermediate feature, a classification module can be used to obtain the second category. Specifically, the weighted intermediate feature can be processed through multiple fully connected (FC) layers to obtain the second category. Here, because the intermediate features are weighted, the probability of a low-weight intermediate feature being classified as a certain category decreases after passing through multiple FC layers. If the probability falls below the threshold for that category, that category will not be output, thus achieving the purpose of verification and improving classification accuracy.

[0054] According to an exemplary embodiment of this disclosure, based on a first category and intermediate features, a verification branch model can be used to obtain at least one second category probability of the object to be classified. When the probability of at least one second category is greater than or equal to the corresponding second threshold, the at least one category corresponding to the probability of at least one second category is determined as the second category of the object to be classified. Here, the second thresholds corresponding to the at least one second category are the same or different. Specifically, the verification branch model includes multiple output channels, each output channel corresponding to a category. Each output channel includes multiple predicted probabilities of the category corresponding to the channel predicted by the verification branch model. The maximum probability among them is taken as the predicted probability of the output channel. When the predicted probability is greater than or equal to the second threshold of the output channel, it can be determined that the category corresponding to the output channel is indeed the category of the object to be classified.

[0055] In step 204, the second category can be determined as the category of the object to be classified. Here, the second category is the category after verification by the validation branch model, thus having higher accuracy.

[0056] Figure 4 This is a flowchart illustrating a training method for a classification model according to an exemplary embodiment of the present disclosure. Here, the classification model refers to a model to be trained for classifying objects, including a classification branch model and a validation branch model.

[0057] Reference Figure 4 In step 401, samples of objects to be classified can be obtained, each corresponding to a real classification label. Here, the samples of objects to be classified refer to object samples used to train the classification model, such as video samples, image samples, or text samples. The specific application scenario of the classification model can be used to obtain the corresponding samples, and there is no limitation on this. The number of samples of objects to be classified can be one or more. For example, to ensure the training effect of the classification model, the number of samples of objects to be classified can be multiple. Each sample of objects to be classified corresponds to a real classification label, which indicates the real category of the object to be classified. This real category describes the actual content contained in the object to be classified. Here, there are multiple real labels for the object to be classified. For example, the category of an image with the theme of navigation might be "sea and ships." In the scenario of multi-label classification of videos, video samples can be obtained. The number of video samples can be one or more. For example, to ensure the training effect of the model, the number of video samples can be multiple.

[0058] In step 402, based on the sample of the object to be classified, a first estimation result and intermediate features can be obtained through the classification branch model in the classification model. The first estimation result represents at least one category of the sample of the object to be classified estimated by the classification branch model, and the intermediate features are the features output by the intermediate layer of the classification branch model. Here, the classification branch model can be any classification model in related technologies. For example, in the scenario of multi-label classification of videos, the structure of the classification branch model can be as follows: Figure 3 As shown. (Refer to...) Figure 3 First, M global images are sampled from the video samples, where M is a natural number greater than or equal to 1. If M frames cannot be sampled, the cover frame image of the video is used to supplement the sampled images. Here, the number of sampled frames is related to the structure of the classification branch model. Then, the M frames are input into the image classification network to extract image features. Here, the image classification network can be Inception-V3, ResNet-50d, or EfficientNet-B3, etc., without restriction. Figure 3 In this example, EfficientNet-B3 is used. Assuming the feature dimension is D, the image feature dimension extracted by EfficientNet-B3 is M*D. Next, the M*D feature is input into the transformer structure for feature interaction to obtain D-dimensional content features (i.e., intermediate features). Finally, the D-dimensional content features are passed through multiple fully connected layers to obtain the estimated category of the video sample (i.e., the first estimation result).

[0059] In step 403, based on the real classification label and the intermediate features, a second estimation result of the object sample to be classified can be obtained through the validation branch model in the classification model. The second estimation result represents the category of the object sample to be classified estimated by the validation branch model.

[0060] According to exemplary embodiments of this disclosure, the verification branch model may include a category feature acquisition module, a feature interaction module, and a classification module. First, based on the true classification labels of the objects to be classified, the category feature acquisition module can obtain the category features of the first category. In some embodiments, the category feature acquisition module can perform one-hot encoding on the true classification labels to obtain a list of categories of true category labels, and can obtain the category features of the true category labels based on the list of categories and a dictionary matrix. The dictionary matrix is ​​a vector representation in a dimensional space, and is a randomly initialized matrix with the same number of rows as the number of columns in the category list, and the same number of columns as the dimension of the intermediate features obtained through the intermediate layer of the classification branch model. Specifically, the true classification labels can first be one-hot encoded, for example, encoded as (0, 1, 2, ..., T-1), for a total of T true categories. Then, the embedding (feature vector) at the corresponding position in the dictionary matrix (T, D) is obtained. For example, if the encoding is 0, the first row of the dictionary matrix's embedding is taken to obtain an array of length D, and so on, to obtain the category features of all categories in the true category labels. After obtaining the category features of the true classification labels, intermediate features with configured weights can be obtained using the feature interaction module, based on these category features and the intermediate features obtained through the intermediate layers of the classification branch model. Here, the weights represent the degree of correlation between the category features and the intermediate features. Specifically, feature interaction can be performed through a multi-head attention mechanism, using the category features as Q (Query) in Attention(Q,K,V), and the intermediate features as K (Key) and V (Value). The similarity weights between the category features and the intermediate features are obtained, and these weights are superimposed on the intermediate features to obtain the intermediate features with configured weights. Passing the intermediate features with configured weights through multiple fully connected (FC) layers yields the estimated category of the object to be classified. Specifically, since the dictionary matrix is ​​randomly initialized, it may not accurately represent the features of the true classification labels in the initial stage of training. Therefore, although the input to the validation branch model is the true classification label, the correlation between the classification features and the intermediate features in the feature interaction module may be low in the initial stage of training. By repeatedly training and adjusting the relevant parameters, a dictionary matrix that accurately reflects the features of the true classification labels can be obtained.

[0061] In step 404, the loss can be calculated based on the first estimation result, the second estimation result, and the true classification label. Here, the loss measures the difference between the estimated category of the object to be classified by the classification model and the true classification label. This loss can be calculated using a loss function, which, for example, but not limited to, can be the cross-entropy loss function. According to an exemplary embodiment of this disclosure, the first loss can be calculated first based on the first estimation result and the true classification label, then the second loss can be calculated based on the second estimation result and the true classification label, and finally the total loss of the classification model can be obtained based on the first loss and the second loss. Specifically, the weighted sum of the first loss and the second loss can be used as the loss, where the weight of the first loss is greater than the weight of the second loss. Here, using the weighted sum of the first loss and the second loss as the loss of the classification model for training the classification model can take into account both the classification effect of the classification branch model and the verification effect of the verification branch model. Furthermore, making the weight of the first loss greater than the weight of the second loss can ensure the main classification effect of the classification branch model and the auxiliary classification effect of the verification branch model. Exemplarily, the relationship between the total loss and the first loss and the second loss can be expressed as:

[0062] Loss = a * Loss c +b*Loss t

[0063] Where Loss represents the total loss of the classification model; c Indicates the first loss; Loss t Let represent the second loss; 'a' represent the weight of the first loss; and 'b' represent the weight of the second loss. For example, a = 1; b = 0.4. The values ​​of a and b can be dynamically adjusted provided that a > b.

[0064] In step 405, the classification model is trained by adjusting the model parameters of the classification branch model and the validation branch model according to the loss. That is, the parameters of the classification model can be adjusted through backpropagation of the loss calculated by the loss function. Furthermore, during model training, batches of samples of objects to be classified (e.g., batches of videos in a multi-label video classification scenario) can be used to adjust or update the model parameters, and the model parameters are iteratively adjusted or updated with the goal of minimizing the loss function value until the loss no longer decreases or the required number of iterations is reached.

[0065] Figure 5 This is an overall schematic diagram illustrating the training and application of a classification model according to exemplary embodiments of the present disclosure.

[0066] Reference Figure 5(a) The classification model includes a classification branch model and a validation branch model. During training, the sample to be classified is input into the classification branch model to obtain intermediate features (obtained by feature extraction from the sample in the intermediate layer) and a first estimation result. The first loss is calculated using the first estimation result and the true classification label corresponding to the sample to be classified. The true classification label and the intermediate features are input into the validation branch model to obtain a second estimation result. The second loss is calculated using the second estimation result and the true classification label corresponding to the sample to be classified. The model parameters of the classification branch model are adjusted using the value of the first loss, and the model parameters of the validation branch model are adjusted using the value of the second loss, thus training the classification model. (Refer to...) Figure 5 (b) The object to be classified can be input into the classification branch model to obtain the predicted first category. The predicted first category and the intermediate features obtained through the classification branch model are then input into the validation branch model to obtain the second category after validating the first category. Here, by adding a validation branch model to the traditional classification model, the classification results of the classification branch model can be validated a second time, which can further improve the precision and recall of the classification branch model and improve the accuracy of classification.

[0067] Figure 6 This is a block diagram illustrating a classification apparatus according to an exemplary embodiment of the present disclosure.

[0068] Reference Figure 6 According to an exemplary embodiment of the present disclosure, the classification device 600 performs operations based on a classification model. The classification device 600 may include an object acquisition unit 601, a first category acquisition unit 602, a second category acquisition unit 603, and a category determination unit 604.

[0069] The object acquisition unit 601 can acquire objects to be classified. The first category acquisition unit 602 can, based on the objects to be classified, use a classification branch model in the classification model to obtain a first category and intermediate features for the objects to be classified. Here, the first category is at least one category of the objects to be classified estimated by the classification branch model, and the intermediate features are features output by the intermediate layers of the classification branch model. The second category acquisition unit 602 can, based on the first category and intermediate features, use a validation branch model in the classification model to obtain a second category. Here, the second category is at least one category obtained after validating the first category using the validation branch model. The category determination unit 604 can determine the second category as the category of the objects to be classified.

[0070] because Figure 2 The classification method shown can be derived from Figure 6 The classification device 600 shown is used to perform the classification, and the object acquisition unit 601, the first category acquisition unit 602, the second category acquisition unit 603, and the category determination unit 604 can respectively perform the classification. Figure 2The operations corresponding to steps 201, 202, 203, and 204 in the text, therefore, regarding Figure 6 Any relevant details regarding the operations performed by each unit within the [system / component] can be found in the [document / reference / etc.]. Figure 2 The corresponding descriptions will not be repeated here.

[0071] Furthermore, it should be noted that although the classification device 600 has been divided into units for performing corresponding processes in the above description, those skilled in the art will understand that the processes performed by the aforementioned units can also be performed without any specific unit division in the classification device 600 or without clear boundaries between the units. In addition, the classification device 600 may also include other units, such as a data processing unit, a storage unit, etc.

[0072] Figure 7 This is a block diagram illustrating a training apparatus for a classification model according to an exemplary embodiment of the present disclosure. Here, the classification model includes a classification branch model and a verification branch model.

[0073] Reference Figure 7 The classification model training apparatus 700 according to an exemplary embodiment of the present disclosure may include a sample acquisition unit 701, a first estimation unit 702, a second estimation unit 703, a loss calculation unit 704, and a model training unit 705.

[0074] The sample acquisition unit 701 acquires samples of objects to be classified, each corresponding to a true classification label. The first estimation unit 702, based on the samples, obtains a first estimation result and intermediate features of the samples through a classification branch model in the classification model. Here, the first estimation result represents at least one category of the samples estimated by the classification branch model, and the intermediate features are the features output by the intermediate layers of the classification branch model. The second estimation unit 703, based on the true classification label and intermediate features, obtains a second estimation result of the samples through a validation branch model in the classification model. Here, the second estimation result represents at least one category of the samples estimated by the validation branch model. The loss calculation unit 704 calculates the loss based on the first estimation result, the second estimation result, and the true classification label. The model training unit 705 trains the classification model by adjusting the model parameters of the classification branch model and the validation branch model according to the loss.

[0075] because Figure 4 The training method for the classification model shown can be derived from... Figure 7 The training device 700 of the classification model shown is used to execute the training, and the sample acquisition unit 701, the first estimation unit 702, the second estimation unit 703, the loss calculation unit 704, and the model training unit 705 can respectively execute the training. Figure 4The operations corresponding to steps 401, 402, 403, 404, and 405 in the text, therefore, regarding Figure 7 Any relevant details regarding the operations performed by each unit within the [system / component] can be found in the [document / reference / etc.]. Figure 4 The corresponding descriptions will not be repeated here.

[0076] Furthermore, it should be noted that although the training apparatus 700 for the classification model is divided into units for performing corresponding processes in the above description, those skilled in the art will understand that the processes performed by the aforementioned units can also be performed without any specific unit division in the training apparatus 700 for the classification model, or without clear boundaries between the units. In addition, the training apparatus 700 for the classification model may also include other units, such as a data processing unit, a storage unit, etc.

[0077] Figure 8 This is a block diagram of an electronic device 800 according to an exemplary embodiment of the present disclosure.

[0078] Reference Figure 8 The electronic device 800 includes at least one memory 801 and at least one processor 802. The at least one memory 801 stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by the at least one processor 802, a classification method or a training method for a classification model according to an exemplary embodiment of the present disclosure is executed.

[0079] As an example, electronic device 800 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 800 is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 800 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0080] In electronic device 800, processor 802 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0081] The processor 802 can execute instructions or code stored in the memory 801, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0082] The memory 801 may be integrated with the processor 802, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 801 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 801 and the processor 802 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 802 to read files stored in the memory.

[0083] In addition, the electronic device 800 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 800 can be interconnected via a bus and / or network.

[0084] According to exemplary embodiments of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, they cause at least one processor to perform a training method for a classification method or classification model according to the present disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0085] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, wherein the instructions in the computer program product can be executed by a processor of a computer device to perform a classification method or a training method for a classification model according to exemplary embodiments of the present disclosure.

[0086] According to the classification and classification model training method, apparatus, equipment and medium disclosed herein, a verification branch model is added to the traditional classification model, thereby realizing secondary verification of the classification results of the classification branch model, which can further improve the precision and recall of the classification branch model and improve the classification accuracy.

[0087] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0088] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A classification method, characterized in that, include: Obtain the object to be classified, which includes at least one of video, image and text; Based on the object to be classified, the first category and intermediate features of the object to be classified are obtained by using the classification branch model in the classification model. The first category is at least one category of the object to be classified estimated by the classification branch model, and the intermediate features are features output by the intermediate layer of the classification branch model. Based on the first category and the intermediate features, the second category of the object to be classified is obtained by using the verification branch model in the classification model. The second category is at least one category obtained after verifying the first category through the verification branch model. The second category is determined as the category of the object to be classified; The verification branch model includes a category feature acquisition module, a feature interaction module, and a classification module. The step of obtaining the second category of the object to be classified based on the first category and the intermediate features, using the verification branch model in the classification model, includes: Based on the first category, the category features of the first category are obtained using the category feature acquisition module; Based on the category features and the intermediate features, the feature interaction module is used to obtain intermediate features with configured weights, whereby the weights represent the degree of correlation between the category features and the intermediate features. Based on the intermediate features of the configured weights, the classification module is used to obtain the second category of the object to be classified.

2. The classification method as described in claim 1, characterized in that, The step of obtaining the category features of the first category using the category feature acquisition module based on the first category includes: Perform one-hot encoding on the first category to obtain a category list of the first category; The category features are obtained by querying the dictionary matrix from the category list. The dictionary matrix is ​​a vector representation in a dimensional space. The number of rows in the dictionary matrix is ​​the same as the number of columns in the category list, and the number of columns in the dictionary matrix is ​​the same as the dimension of the intermediate features.

3. The classification method as described in claim 1, characterized in that, The step of obtaining the first category of the object to be classified based on the object to be classified using the classification branch model in the classification model includes: Based on the object to be classified, the classification branch model is used to obtain at least one first category probability of the object to be classified. When the probability of at least one first category is greater than or equal to the corresponding first threshold, the at least one category corresponding to the probability of at least one first category is determined as the first category of the object to be classified.

4. The classification method as described in claim 1, characterized in that, The step of obtaining the second category of the object to be classified based on the first category and the intermediate features, using the validation branch model in the classification model, includes: Based on the first category and the intermediate features, the verification branch model is used to obtain at least one second category probability of the object to be classified. When the probability of at least one second category is greater than or equal to the corresponding second threshold, the at least one category corresponding to the probability of at least one second category is determined as the second category of the object to be classified.

5. A method for training a classification model, characterized in that, include: Obtain a sample of an object to be classified, wherein the sample of the object to be classified includes at least one of video, image and text, and the sample of the object to be classified corresponds to a real classification label; Based on the object sample to be classified, a first estimation result and intermediate features of the object sample to be classified are obtained through the classification branch model in the classification model. The first estimation result represents at least one category of the object sample to be classified estimated by the classification branch model, and the intermediate features are features output by the intermediate layer of the classification branch model. Based on the true classification label and the intermediate features, a second estimation result of the object sample to be classified is obtained through the validation branch model in the classification model. The second estimation result represents at least one category of the object sample to be classified as estimated by the validation branch model. The loss is calculated based on the first estimation result, the second estimation result, and the true classification label; The classification model is trained by adjusting the model parameters of the classification branch model and the validation branch model according to the loss. The verification branch model includes a category feature acquisition module, a feature interaction module, and a classification module. The step of obtaining a second estimation result of the sample to be classified based on the true classification label and the intermediate features through the verification branch model includes: Based on the real classification label, the category features of the real classification label are obtained using the category feature acquisition module; Based on the category features and the intermediate features, the intermediate features with configured weights are obtained using the feature interaction module; Based on the intermediate features of the configured weights, the classification module is used to obtain a second estimation result of the sample of the object to be classified.

6. The training method for the classification model as described in claim 5, characterized in that, The step of obtaining the category features of the real classification label using the category feature acquisition module based on the real classification label includes: One-hot encoding is performed on the actual classification labels to obtain a list of categories for the actual classification labels; The category features are obtained by querying the dictionary matrix from the category list. The dictionary matrix is ​​a vector representation in a dimensional space. The number of rows in the dictionary matrix is ​​the same as the number of columns in the category list, and the number of columns in the dictionary matrix is ​​the same as the dimension of the intermediate features.

7. The training method for the classification model as described in claim 5, characterized in that, The step of calculating the loss based on the first estimation result, the second estimation result, and the true classification label includes: Calculate the first loss based on the first estimation result and the true classification label; Calculate the second loss based on the second estimation result and the true classification label; The loss is obtained based on the first loss and the second loss.

8. The training method for the classification model as described in claim 7, characterized in that, The loss is obtained based on the first loss and the second loss, including: The weighted sum of the first loss and the second loss is taken as the loss, wherein the weight of the first loss is greater than the weight of the second loss.

9. A sorting device, characterized in that, include: The object acquisition unit is configured to acquire an object to be classified, wherein the object to be classified includes at least one of video, image and text. The first category acquisition unit is configured to: based on the object to be classified, use the classification branch model in the classification model to obtain the first category and intermediate features of the object to be classified, wherein the first category is at least one category of the object to be classified estimated by the classification branch model, and the intermediate features are features output by the intermediate layer of the classification branch model. The second category acquisition unit is configured to: based on the first category and the intermediate features, use the verification branch model in the classification model to obtain the second category of the object to be classified, wherein the second category is at least one category obtained after verifying the first category through the verification branch model; The category determination unit is configured to: determine the second category as the category of the object to be classified; The verification branch model includes a category feature acquisition module, a feature interaction module, and a classification module, wherein the second category acquisition unit is configured as follows: Based on the first category, the category features of the first category are obtained using the category feature acquisition module; Based on the category features and the intermediate features, the feature interaction module is used to obtain intermediate features with configured weights, whereby the weights represent the degree of correlation between the category features and the intermediate features. Based on the intermediate features of the configured weights, the classification module is used to obtain the second category of the object to be classified.

10. The sorting device as described in claim 9, characterized in that, The second category acquisition unit is configured as follows: Perform one-hot encoding on the first category to obtain a category list of the first category; The category features are obtained by querying the dictionary matrix from the category list. The dictionary matrix is ​​a vector representation in a dimensional space. The number of rows in the dictionary matrix is ​​the same as the number of columns in the category list, and the number of columns in the dictionary matrix is ​​the same as the dimension of the intermediate features.

11. The sorting device as described in claim 9, characterized in that, The first category acquisition unit is configured as follows: Based on the object to be classified, the classification branch model is used to obtain at least one first category probability of the object to be classified. When the probability of at least one first category is greater than or equal to the corresponding first threshold, the at least one category corresponding to the probability of at least one first category is determined as the first category of the object to be classified.

12. The sorting device as described in claim 9, characterized in that, The second category acquisition unit is configured as follows: Based on the first category and the intermediate features, the verification branch model is used to obtain at least one second category probability of the object to be classified. When the probability of at least one second category is greater than or equal to the corresponding second threshold, the at least one category corresponding to the probability of at least one second category is determined as the second category of the object to be classified.

13. A training device for a classification model, characterized in that, include: The sample acquisition unit is configured to: acquire a sample of an object to be classified, wherein the sample of the object to be classified includes at least one of video, image and text, and the sample of the object to be classified corresponds to a real classification label; The first estimation unit is configured to: based on the object sample to be classified, obtain a first estimation result and intermediate features of the object sample to be classified through the classification branch model in the classification model, wherein the first estimation result represents at least one category of the object sample to be classified estimated by the classification branch model, and the intermediate features are features output by the intermediate layer of the classification branch model. The second estimation unit is configured to: obtain a second estimation result of the object sample to be classified based on the true classification label and the intermediate features through the validation branch model in the classification model, wherein the second estimation result represents at least one category of the object sample to be classified estimated by the validation branch model; The loss calculation unit is configured to calculate the loss based on the first estimation result, the second estimation result, and the true classification label. The model training unit is configured to train the classification model by adjusting the model parameters of the classification branch model and the validation branch model according to the loss. The verification branch model includes a category feature acquisition module, a feature interaction module, and a classification module, and the second estimation unit is configured as follows: Based on the real classification label, the category features of the real classification label are obtained using the category feature acquisition module; Based on the category features and the intermediate features, the intermediate features with configured weights are obtained using the feature interaction module; Based on the intermediate features of the configured weights, the classification module is used to obtain a second estimation result of the sample of the object to be classified.

14. The training apparatus for the classification model as described in claim 13, characterized in that, The second estimation unit is configured as follows: One-hot encoding is performed on the actual classification labels to obtain a list of categories for the actual classification labels; The category features are obtained by querying the dictionary matrix from the category list. The dictionary matrix is ​​a vector representation in a dimensional space. The number of rows in the dictionary matrix is ​​the same as the number of columns in the category list, and the number of columns in the dictionary matrix is ​​the same as the dimension of the intermediate features.

15. The training apparatus for the classification model as described in claim 13, characterized in that, The loss calculation unit is configured as follows: Calculate the first loss based on the first estimation result and the true classification label; Calculate the second loss based on the second estimation result and the true classification label; The loss is obtained based on the first loss and the second loss.

16. The training apparatus for the classification model as described in claim 15, characterized in that, The model training unit is configured as follows: The weighted sum of the first loss and the second loss is taken as the loss, wherein the weight of the first loss is greater than the weight of the second loss.

17. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. The computer-executable instructions, when executed by the at least one processor, cause the at least one processor to execute the classification method as described in any one of claims 1 to 4 or the training method for the classification model as described in any one of claims 5 to 8.

18. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one processor, it causes the at least one processor to perform the classification method as described in any one of claims 1 to 4 or the training method for the classification model as described in any one of claims 5 to 8.

19. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, they implement the classification method as described in any one of claims 1 to 4 or the training method for the classification model as described in any one of claims 5 to 8.

Images