Method and apparatus for classifying dynamic images, vehicle and storage medium

Abstract

The application relates to the technical field of fine-grained dynamic image classification, in particular to a dynamic image classification method and device, a vehicle and a storage medium, which comprises the following steps: acquiring a dynamic image to be classified; identifying a feature map of each frame of image in the dynamic image; inputting the feature map of each frame of image into a classification model pre-trained to output an actual classification category of the dynamic image; wherein the classification model comprises a first branch network, the first branch network comprises a fine-grained classifier and a coarse-grained classifier, the fine-grained classifier and the coarse-grained classifier are used to generate a fine-grained prediction score and a coarse-grained prediction score of the dynamic image respectively, and the actual classification category of the dynamic image is matched according to the fine-grained prediction score and / or the coarse-grained prediction score. Thus, the method for the fine-grained classification task in the related art cannot improve the performance of the fine-grained image classification model, and the generalization and robustness of the image classification model are low.

Description

Technical Field

[0001] This application relates to the field of fine-grained animated image classification technology, and in particular to a method, apparatus, vehicle, and storage medium for classifying dynamic images. Background Technology

[0002] As an ideal medium for information transmission, GIFs have been widely used in various social media platforms, advertising, digital forums, and online education. They are even used in emails as an enhanced version of emoticons, unconsciously becoming a part of people's daily online lives. Efficiently utilizing massive amounts of low-quality internet GIFs to achieve low-cost GIF content classification based on deep learning has significant research value. However, the biggest challenge in learning fine-grained GIF classification from internet data lies in the existence of coarse-grained and fine-grained "subordinate" relationships between semantic tags in internet GIFs, and the availability of very limited fine-grained labeled GIF data.

[0003] In related technologies, methods for handling fine-grained classification tasks under the same problem setting mainly target static image data. A dual-classification branch network structure based on CNN (Convolutional Neural Network) achieves the goal of using coarse-grained labeled images to help train a fine-grained image classification model. However, the coarse-grained classification method based on minpooling is limited by datasets with perfect hierarchical structures. For fine-grained datasets with large class spans and unclear common features between classes, the features extracted by minpooling are likely to be background features, lacking generality. Moreover, fine-grained image classification is much more difficult than ordinary coarse-grained image classification such as image recognition. Using only coarse-grained labels for supervised training can only improve the model's general feature extraction ability, with very limited performance improvement for fine-grained image classification models. Summary of the Invention

[0004] This application provides a method, apparatus, vehicle, and storage medium for classifying dynamic images to address the problems in related technologies where methods for fine-grained classification tasks cannot improve the performance of fine-grained image classification models, resulting in low generalization and robustness of image classification models.

[0005] The first aspect of this application provides a method for classifying dynamic images, comprising the following steps: acquiring a dynamic image to be classified; identifying feature maps of each frame in the dynamic image; inputting the feature maps of each frame into a pre-trained classification model; and outputting the actual classification category of the dynamic image. The classification model includes a first branch network, which includes a fine-grained classifier and a coarse-grained classifier. The fine-grained classifier and the coarse-grained classifier are used to generate fine-grained prediction scores and coarse-grained prediction scores for the dynamic image, respectively. The actual classification category of the dynamic image is matched based on the fine-grained prediction scores and / or the coarse-grained prediction scores, facilitating more accurate identification of the actual category of the dynamic image.

[0006] Based on the above technical means, this application embodiment acquires the dynamic image to be classified and identifies the feature map of each frame of the image, inputs it into a pre-trained classification model for classification matching, and outputs the actual classification category corresponding to the dynamic image. It uses a learnable neural network structure to replace the previous coarse-grained classification method based on the Minpooling rule to help train the fine-grained model, thereby improving the generalization and robustness of the model.

[0007] Furthermore, the first branch network also includes an attention module. The step of inputting the feature map of each frame image into the pre-trained classification model and outputting the actual classification category of the dynamic image includes: inputting the feature map of each frame image into the fine-grained classifier, the coarse-grained classifier, and the attention module respectively, and outputting the fine-grained frame prediction score, coarse-grained frame prediction score, and weight of each frame image; performing a weighted sum based on the fine-grained frame prediction score and the weight of each frame image to obtain the fine-grained prediction score of the dynamic image, and performing a weighted sum based on the coarse-grained frame prediction score and the weight to obtain the coarse-grained prediction score of the dynamic image; if it is in the forward propagation calculation of the network, then matching the actual classification category of the dynamic image according to the fine-grained prediction score; otherwise, matching the actual classification category of the dynamic image according to the fine-grained prediction score and the coarse-grained prediction score.

[0008] Based on the above technical means, in this embodiment of the application, each frame of the dynamic image is input into a fine-grained classifier, a coarse-grained classifier, and an attention module, respectively, and the fine-grained frame prediction score, coarse-grained frame prediction score, and weight of each frame are output to obtain the coarse-grained prediction score of the dynamic image. When the network is in the forward propagation calculation, the actual classification category is matched according to the fine-grained prediction score; otherwise, the actual classification category of the dynamic image is matched according to the prediction scores of both fine-grained and coarse-grained classes. A dual-branch network structure that can learn both coarse-grained and fine-grained classification is adopted to perform coarse-grained classification prediction and fine-grained classification prediction simultaneously, so as to improve the generalization and robustness of the model.

[0009] Furthermore, before inputting the feature map of each frame image into the pre-trained classification model, the method includes: spatial pooling of the feature map of each frame image to obtain the dimensionality reduction features of each frame image.

[0010] Based on the above technical means, the embodiments of this application perform spatial pooling on the feature map of each frame image to perform fine-grained and coarse-grained classification of the animation, and obtain the dimensionality reduction features of each frame image to reduce the dimensionality of the feature map. This is to avoid the excessive number of neurons in the subsequent network, which would lead to an overly complex network structure and cause overfitting of the classification model.

[0011] Furthermore, the classification model further includes a second branch network, wherein the first branch network and the second branch network have the same structure. The classification model training process includes: acquiring a training dataset, wherein the training dataset includes coarse-grained labeled dynamic images; perturbing the coarse-grained labeled dynamic images to obtain a first dynamic image and a second dynamic image, wherein the perturbation level of the first dynamic image is less than that of the second dynamic image; inputting the first dynamic image into the first branch network, outputting a fine-grained prediction score for the first dynamic image, generating a fine-grained pseudo-label based on the fine-grained prediction score of the first dynamic image, and using the fine-grained pseudo-label as the true fine-grained label for the second dynamic image; and so on. The second dynamic image is input into the second branch network, which outputs a fine-grained prediction score and a coarse-grained prediction score for the second dynamic image. A fine-grained cross-entropy classification loss is calculated based on the fine-grained prediction score and the true fine-grained label, and a coarse-grained cross-entropy classification loss is calculated based on the coarse-grained prediction score and the coarse-labeled dynamic image. The total training loss is calculated based on the fine-grained cross-entropy classification loss and the coarse-grained cross-entropy classification loss. If the total loss is greater than a preset threshold, the second branch network continues to be trained based on the trained dataset, and the network parameters of the second branch network and the first branch network are updated synchronously until the total loss is less than or equal to the preset threshold. Training then stops, and the classification model is obtained.

[0012] According to the above technical means, this application embodiment obtains a training dataset and perturbs the data of the coarse-grained labeled dynamic images to obtain a first dynamic image and a second dynamic image; the first dynamic image is input into a first branch network to obtain a fine-grained prediction score and generate a fine-grained pseudo-label, which is used as the real fine-grained label of the second dynamic image; the second dynamic image is input into a second branch network to output fine-grained and coarse-grained prediction scores, and the fine-grained cross-entropy classification loss and coarse-grained cross-entropy classification loss are calculated respectively, and then the total training loss is calculated. When the total loss is greater than a threshold, training continues and network parameters are synchronized; if it is less than the threshold, training stops and a classification model is obtained. The coarse-grained assisted fine-grained self-learning realizes efficient utilization of the dynamic images in the coarse-grained label, and improves the generalization and robustness of the model.

[0013] Further, the step of generating fine-grained pseudo-labels based on the fine-grained prediction score of the first dynamic image includes: normalizing the fine-grained prediction score of the first dynamic image to obtain a fine-grained prediction probability matrix; converting the fine-grained prediction probability matrix into a one-hot encoded probability distribution matrix, mapping the fine-grained category corresponding to the maximum class probability in the probability distribution matrix to the corresponding coarse-grained category, and converting it into a one-hot encoded coarse-grained label; determining whether the maximum class probability in the probability distribution matrix is ​​higher than a preset confidence threshold, and whether the one-hot encoded coarse-grained label is consistent with the true coarse-grained label of the coarse-labeled dynamic image; if the maximum class probability is higher than the preset confidence threshold, and the one-hot encoded coarse-grained label is consistent with the true coarse-grained label, then the probability distribution matrix is ​​used as the fine-grained pseudo-label; otherwise, the probability distribution matrix is ​​not used as the fine-grained pseudo-label.

[0014] Based on the above technical means, this embodiment of the application obtains a fine-grained prediction probability matrix by normalizing the fine-grained prediction score of the first dynamic image, and converts it into a one-hot encoded probability distribution matrix. The fine-grained category corresponding to the maximum class probability is mapped to the corresponding coarse-grained category and converted into a one-hot encoded coarse-grained label. It is determined whether the maximum class probability in the probability distribution matrix is ​​higher than the confidence threshold and whether the coarse-grained label is consistent with the true coarse-grained label of the dynamic image. If the maximum class probability is higher than the confidence threshold and the labels are consistent, the probability distribution matrix is ​​used as a fine-grained pseudo-label. Its coarse-grained label is used to further filter the incorrect fine-grained pseudo-labels generated by the model, so as to avoid the fine-grained animation classification model being misled by incorrect pseudo-labels and to prevent the model from incorrectly labeling a fine-grained category as a fine-grained category that is very similar to it in another coarse category, thereby improving the accuracy of the model.

[0015] A second aspect of this application provides a dynamic image classification apparatus, comprising: an acquisition module for acquiring a dynamic image to be classified; and a processing module for identifying feature maps of each frame in the dynamic image, inputting the feature maps of each frame into a pre-trained classification model, and outputting the actual classification category of the dynamic image. The classification model includes a first branch network, which includes a fine-grained classifier and a coarse-grained classifier. The fine-grained classifier and the coarse-grained classifier are used to generate fine-grained prediction scores and coarse-grained prediction scores for the dynamic image, respectively, and the actual classification category of the dynamic image is matched based on the fine-grained prediction scores and / or the coarse-grained prediction scores.

[0016] Furthermore, the processing module is further configured to: input the feature map of each frame image into the fine-grained classifier, the coarse-grained classifier, and the attention module respectively, and output the fine-grained frame prediction score, coarse-grained frame prediction score, and weight of each frame image; perform a weighted sum based on the fine-grained frame prediction score and the weight of each frame image to obtain the fine-grained prediction score of the dynamic image, and perform a weighted sum based on the coarse-grained frame prediction score and the weight to obtain the coarse-grained prediction score of the dynamic image; if the network is in forward propagation calculation, match the actual classification category of the dynamic image according to the fine-grained prediction score; otherwise, match the actual classification category of the dynamic image according to the fine-grained prediction score and the coarse-grained prediction score.

[0017] Furthermore, the processing module is further configured to: perform spatial pooling on the feature map of each frame image to obtain the dimensionality-reduced features of each frame image.

[0018] Further, the processing module is further configured to: acquire a training dataset, wherein the training dataset includes coarse-grained labeled dynamic images; perturb the coarse-grained labeled dynamic images to obtain a first dynamic image and a second dynamic image, wherein the perturbation level of the first dynamic image is less than the perturbation level of the second dynamic image; input the first dynamic image into the first branch network, output the fine-grained prediction score of the first dynamic image, generate fine-grained pseudo-labels based on the fine-grained prediction score of the first dynamic image, and use the fine-grained pseudo-labels as the true fine-grained labels of the second dynamic image; input the second dynamic image into the second branch network, and output... The fine-grained prediction score and coarse-grained prediction score of the second dynamic image are used to calculate the fine-grained cross-entropy classification loss based on the fine-grained prediction score of the second dynamic image and the real fine-grained label, and the coarse-grained cross-entropy classification loss is calculated based on the coarse-grained prediction score and the coarse-grained labeled dynamic image; the total training loss is calculated based on the fine-grained cross-entropy classification loss and the coarse-grained cross-entropy classification loss. If the total loss is greater than a preset threshold, the second branch network is trained on the trained dataset, and the network parameters of the second branch network and the first branch network are updated synchronously until the total loss is less than or equal to the preset threshold, at which point training stops and the classification model is obtained.

[0019] Further, the processing module is further configured to: normalize the fine-grained prediction score of the first dynamic image to obtain a fine-grained prediction probability matrix; convert the fine-grained prediction probability matrix into a one-hot encoded probability distribution matrix, map the fine-grained category corresponding to the maximum class probability in the probability distribution matrix to the corresponding coarse-grained category, and convert it into a one-hot encoded coarse-grained label; determine whether the maximum class probability in the probability distribution matrix is ​​higher than a preset confidence threshold, and whether the one-hot encoded coarse-grained label is consistent with the true coarse-grained label of the dynamic image labeled with coarse-grained information; if the maximum class probability is higher than the preset confidence threshold, and the one-hot encoded coarse-grained label is consistent with the true coarse-grained label, then use the probability distribution matrix as the fine-grained pseudo-label; otherwise, do not use the probability distribution matrix as the fine-grained pseudo-label.

[0020] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic image classification method as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the dynamic image classification method as described in the above embodiments.

[0022] Therefore, this application has at least the following beneficial effects:

[0023] (1) In this embodiment of the application, the dynamic image to be classified is obtained and the feature map of each frame of the image is identified. The feature map is then input into the pre-trained classification model for classification matching and the actual classification category corresponding to the dynamic image is output. The learnable neural network structure replaces the previous coarse-grained classification method based on the Minpooling rule to help train the fine-grained model, which improves the generalization and robustness of the model and makes it easier to more accurately identify the actual category of the dynamic image.

[0024] (2) In this embodiment, each frame of the dynamic image is input into the fine-grained classifier, the coarse-grained classifier and the attention module respectively, and the fine-grained frame prediction score, coarse-grained frame prediction score and weight of each frame are output respectively to obtain the coarse-grained prediction score of the dynamic image. When the network is in the forward propagation calculation, the actual classification category is matched according to the fine-grained prediction score. Otherwise, the actual classification category of the dynamic image is matched according to the prediction scores of both fine-grained and coarse-grained classes. A dual-branch network structure that can learn both coarse-grained and fine-grained classification is adopted to perform coarse-grained classification prediction and fine-grained classification prediction at the same time to improve the generalization and robustness of the model.

[0025] (3) In this embodiment, spatial pooling is performed on the feature map of each frame image to perform fine-grained and coarse-grained classification of the animation and obtain the dimensionality reduction features of each frame image to reduce the dimensionality of the feature map. This is to avoid the network structure becoming too complex due to the excessive number of neurons in the subsequent network, which would cause the classification model to overfit.

[0026] (4) In this embodiment, a training dataset is obtained, and the data perturbation of the coarse-grained labeled dynamic images is performed to obtain a first dynamic image and a second dynamic image; the first dynamic image is input into the first branch network to obtain a fine-grained prediction score, and a fine-grained pseudo label is generated and used as the real fine-grained label of the second dynamic image; the second dynamic image is input into the second branch network, and the fine-grained and coarse-grained prediction scores are output. The fine-grained cross-entropy classification loss and the coarse-grained cross-entropy classification loss are calculated respectively, and then the total training loss is calculated. When the total loss is greater than the threshold, training continues and the network parameters are synchronized. If it is less than the threshold, training is stopped and a classification model is obtained. The coarse-grained assisted fine-grained self-learning realizes the efficient use of the animation in the coarse-grained label, and improves the generalization and robustness of the model.

[0027] (5) In this embodiment, the fine-grained prediction probability matrix is ​​obtained by normalizing the fine-grained prediction score of the first dynamic image and converting it into a one-hot encoded probability distribution matrix. The fine-grained category corresponding to the maximum class probability is mapped to the corresponding coarse-grained category and converted into a one-hot encoded coarse-grained label. It is determined whether the maximum class probability in the probability distribution matrix is ​​higher than the confidence threshold and whether the coarse-grained label is consistent with the real coarse-grained label of the dynamic image. If the maximum class probability is higher than the confidence threshold and the label is consistent, the probability distribution matrix is ​​used as a fine-grained pseudo-label. Its coarse-grained label is used to further filter the incorrect fine-grained pseudo-labels generated by the model, so as to avoid the fine-grained animation classification model being misled by incorrect pseudo-labels and to avoid the model incorrectly labeling a fine-grained category as another coarse category that is very similar to it, thereby improving the accuracy of the model.

[0028] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0029] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0030] Figure 1 This is a flowchart of a dynamic image classification method according to an embodiment of this application;

[0031] Figure 2 This is an overall framework diagram of the dynamic image classification method according to an embodiment of this application;

[0032] Figure 3 This is a diagram of the adaptive multi-granularity animated graph classification network structure according to an embodiment of this application;

[0033] Figure 4 This is a schematic diagram illustrating the impact of different confidence thresholds on the WGIF dataset according to embodiments of this application on the model's fine-grained animation classification performance;

[0034] Figure 5 This is a schematic diagram comparing the fine-grained classification accuracy of the CRAFT method and the comparison method on the WGIF dataset according to an embodiment of this application;

[0035] Figure 6 This is a schematic diagram comparing the fine-grained classification performance of the CRAFT method and the comparison method on the WGIF dataset according to embodiments of this application;

[0036] Figure 7 This is a schematic diagram illustrating the ablation study of different components of the CRAFT method on the WGIF dataset according to embodiments of this application;

[0037] Figure 8 This is a schematic diagram illustrating the classification performance of fine-grained labeled data of different scales on the WGIF dataset according to embodiments of this application;

[0038] Figure 9 This is a block diagram of a dynamic image classification device according to an embodiment of this application;

[0039] Figure 10 This is a structural schematic diagram of a vehicle according to an embodiment of this application. Detailed Implementation

[0040] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0041] As an ideal medium for information transmission, GIFs have been widely used in various social media platforms, advertising, digital forums, and online education. They are even used in emails as an enhanced version of emoticons, unconsciously becoming a part of people's daily online lives. Efficiently utilizing massive amounts of low-quality internet GIFs to achieve low-cost GIF content classification based on deep learning has significant research value. However, the biggest challenge in learning fine-grained GIF classification from internet data lies in the existence of coarse-grained and fine-grained "subordinate" relationships between semantic tags in internet GIFs, and the availability of very limited fine-grained labeled GIF data.

[0042] To fully utilize the large-scale coarse-grained labeled animated images on the Internet to improve the performance of fine-grained animated image classification models, the research direction is mainly as follows: a subset of the animated image training set has fine-grained labels (e.g., dog breeds, types of ball sports), while the remaining animated images only have coarse-grained labels (e.g., dogs, ball sports). Furthermore, there is a subordinate relationship between the coarse and fine-grained category labels of the animated images, where "any fine-grained category belongs to one and only one of the coarse-grained categories". The task objective is to perform fine-grained classification on the animated images in the test set.

[0043] Existing methods for handling fine-grained classification tasks under the same problem setting mainly target static image data. They use a CNN-based dual-branch network structure to achieve the goal of using coarse-grained labeled images to help train fine-grained image classification models. However, the coarse-grained classification method based on min-pooling is limited by datasets with perfect hierarchical structure. For fine-grained datasets with large class spans and indistinct common features between classes, the features extracted by min-pooling are likely to be background features, thus lacking generality. Moreover, fine-grained image classification is much more difficult than ordinary coarse-grained image classification such as image recognition. Using only coarse-grained labels for supervised training can only improve the model's general feature extraction ability, with very limited performance improvement for fine-grained image classification models.

[0044] The core idea of ​​coarse-grained assisted fine-grained self-learning is to automatically generate fine-grained pseudo-labels with coarse label constraints for coarse-grained labeled animations to supervise the training of fine-grained models. Previous fine-grained image classification methods used to solve the same problem setting only enhanced the model's general feature extraction capabilities with coarse-grained labeled data, offering very limited help for fine-grained classification tasks.

[0045] To address this issue, a feasible solution is to automatically assign fine-grained pseudo-labels to the coarse-grained annotated animated samples in the training set, maximizing the utilization of the coarse-labeled data. Although pseudo-labeling technology has been widely used and researched, it is currently mainly used to generate labels for unlabeled data, handling incompletely supervised learning tasks. This application proposes a novel coarse-grained assisted fine-grained self-learning method, extending pseudo-labeling technology to weakly supervised learning tasks with imprecise supervision. This method generates fine-grained pseudo-labels for coarse-grained annotated animated images and further filters out erroneous fine-grained pseudo-labels generated by the model using the coarse-grained labels, preventing the fine-grained animated image classification model from being misled by incorrect pseudo-labels.

[0046] The following description, with reference to the accompanying drawings, describes a method, apparatus, vehicle, and storage medium for classifying dynamic images according to embodiments of this application.

[0047] Specifically, Figure 1 This is a flowchart illustrating the dynamic image classification method provided in an embodiment of this application.

[0048] like Figure 1 As shown, the classification method for this dynamic image includes the following steps:

[0049] In step S101, the dynamic image to be classified is acquired.

[0050] It is understood that the embodiments of this application acquire dynamic images to be classified, so as to facilitate subsequent classification according to the actual type of the dynamic images.

[0051] In step S102, the feature map of each frame in the dynamic image is identified, and the feature map of each frame is input into the pre-trained classification model to output the actual classification category of the dynamic image. The classification model includes a first branch network, which includes a fine-grained classifier and a coarse-grained classifier. The fine-grained classifier and the coarse-grained classifier are used to generate fine-grained prediction scores and coarse-grained prediction scores for the dynamic image, respectively, and the actual classification category of the dynamic image is matched according to the fine-grained prediction scores and / or coarse-grained prediction scores.

[0052] The pre-trained classification model can be a fine-grained animation classification model with coarse-grained assistance and fine-grained self-learning, without any specific limitations.

[0053] The first branch network can be a fine-grained pseudo-label generation branch network, used for fine-grained classification, and no specific limitation is made here.

[0054] Among them, the fine-grained classifier can classify subtle differences among images of the same type. For example, it can distinguish the fine-grained subclasses of the category "dog", namely "husky" and "Eskimo dog", by classifying them based on subtle differences such as ear shape and fur color. No specific limitation is made here.

[0055] Among them, the coarse-grained classifier can classify dynamic images into broad categories. For example, in the category of animals, the coarse-grained classifier can classify them into poultry, insects, carnivores, etc., without making specific limitations here.

[0056] It is understood that the embodiments of this application acquire dynamic images to be classified and identify the feature maps of each frame of the images, input them into a pre-trained classification model for classification matching, and output the actual classification category corresponding to the dynamic image. The learnable neural network structure replaces the previous coarse-grained classification method based on the Minpooling rule to help train the fine-grained model, improves the generalization and robustness of the model, and facilitates more accurate identification of the actual category of the dynamic image.

[0057] In this embodiment, the first branch network further includes an attention module, which inputs the feature map of each frame image into a pre-trained classification model and outputs the actual classification category of the dynamic image. This includes: inputting the feature map of each frame image into a fine-grained classifier, a coarse-grained classifier, and the attention module respectively, and outputting the fine-grained frame prediction score, coarse-grained frame prediction score, and weight of each frame image; performing a weighted sum based on the fine-grained frame prediction score and weight of each frame image to obtain the fine-grained prediction score of the dynamic image, and performing a weighted sum based on the coarse-grained frame prediction score and weight to obtain the coarse-grained prediction score of the dynamic image; if the network is in the forward propagation calculation stage, the actual classification category of the dynamic image is matched according to the fine-grained prediction score; otherwise, the actual classification category of the dynamic image is matched according to the fine-grained prediction score and the coarse-grained prediction score.

[0058] The attention module receives feature input from the feature map and generates attention weights for each frame of the animation, which are used to identify the correlation between the animation and the true class label. No specific limitations are made here.

[0059] Weight refers to the degree of importance of a factor or indicator relative to a thing. It is different from the general proportion. It reflects not only the percentage of a factor or indicator, but also the relative importance of the factor or indicator. It tends to be a contribution or importance, and no specific limitation is made here.

[0060] In this context, forward propagation computation can refer to the output of the previous layer being used as the input of the next layer. For example, a dynamic image is input to a coarse-grained classifier to determine that the type is animal. Then, the dynamic image of the animal is input to a fine-grained classifier to determine that the animal is a canine. Finally, the attention module determines that it is a "husky". This is a progressive relationship between layers, and no specific limitation is made here.

[0061] It is understood that in this embodiment, each frame of the dynamic image is input into a fine-grained classifier, a coarse-grained classifier, and an attention module, respectively, and the fine-grained frame prediction score, coarse-grained frame prediction score, and weight of each frame are output to obtain the coarse-grained prediction score of the dynamic image. When the network is in the forward propagation calculation, the actual classification category is matched according to the fine-grained prediction score; otherwise, the actual classification category of the dynamic image is matched according to the prediction scores of both fine-grained and coarse-grained classes. A dual-branch network structure that can learn both coarse-grained and fine-grained classification is adopted to perform coarse-grained classification prediction and fine-grained classification prediction simultaneously, so as to improve the generalization and robustness of the model.

[0062] In this embodiment of the application, before inputting the feature map of each frame of the image into the pre-trained classification model, the method includes: spatial pooling of the feature map of each frame of the image to obtain the dimensionality reduction features of each frame of the image.

[0063] Spatial pooling can extract spatial features by calculating the weighted sum of the center pixel and its neighboring pixels to form a feature map. By aggregating and statistically processing different features, a relatively lower dimension can be obtained, while avoiding overfitting. No specific limitations are made here.

[0064] It is understood that the embodiments of this application perform spatial pooling on the feature map of each frame image to perform fine-grained and coarse-grained classification of the animation and obtain the dimensionality reduction features of each frame image to reduce the dimensionality of the feature map. This is to avoid the subsequent network structure becoming too complex due to an excessive number of neurons, which would cause the classification model to overfit.

[0065] In this embodiment, the classification model further includes a second branch network, wherein the first branch network and the second branch network have the same structure. The classification model training process includes: acquiring a training dataset, wherein the training dataset includes coarse-grained labeled dynamic images; perturbing the coarse-grained labeled dynamic images to obtain a first dynamic image and a second dynamic image, wherein the perturbation level of the first dynamic image is less than that of the second dynamic image; inputting the first dynamic image into the first branch network, outputting a fine-grained prediction score for the first dynamic image, generating a fine-grained pseudo-label based on the fine-grained prediction score of the first dynamic image, and using the fine-grained pseudo-label as the true fine-grained label for the second dynamic image. The second dynamic image is input into the second branch network, which outputs the fine-grained prediction score and coarse-grained prediction score of the second dynamic image. The fine-grained cross-entropy classification loss is calculated based on the fine-grained prediction score and the true fine-grained label of the second dynamic image, and the coarse-grained cross-entropy classification loss is calculated based on the coarse-grained prediction score and the coarse-grained labeled dynamic image. The total training loss is calculated based on the fine-grained cross-entropy classification loss and the coarse-grained cross-entropy classification loss. If the total loss is greater than a preset threshold, the second branch network is trained again based on the trained dataset, and the network parameters of the second branch network and the first branch network are updated synchronously until the total loss is less than or equal to the preset threshold. At this point, training stops, and the classification model is obtained.

[0066] The second branch network can be a fine-grained pseudo-label generation backbone branch used for training the classification model; no specific limitations are made here.

[0067] Data perturbation can be performed during testing to evaluate the robustness of the trained model; no specific limitations are specified here.

[0068] The first dynamic image can be obtained by slightly perturbing the data of a coarse-grained labeled dynamic image, and no specific limitation is made here.

[0069] The second dynamic image can be obtained by subjecting a coarse-grained labeled dynamic image to drastic data perturbation, and no specific limitation is made here.

[0070] Fine-grained pseudo-labels can be used to make predictions on unlabeled data using a model trained on labeled data, and then filter the samples based on the prediction results to generate labels for the unlabeled data. No specific limitations are made here.

[0071] Cross-entropy is an indicator of output uncertainty caused by the mismatch between the actual output probability and the perceived probability, and is not specifically defined here.

[0072] The preset threshold can be a threshold set by the user in advance. For example, the default loss threshold is 0.5, which can be adjusted or set according to the actual situation. No specific limitation is made here.

[0073] It is understood that in this embodiment, a training dataset is obtained, and the coarse-grained labeled dynamic images are perturbed to obtain a first dynamic image and a second dynamic image. The first dynamic image is input into a first branch network to obtain a fine-grained prediction score, and a fine-grained pseudo-label is generated and used as the real fine-grained label for the second dynamic image. The second dynamic image is input into a second branch network, and the fine-grained and coarse-grained prediction scores are output. The fine-grained cross-entropy classification loss and the coarse-grained cross-entropy classification loss are calculated respectively. Then, the total training loss is calculated. When the total loss is greater than a threshold, training continues and the network parameters are synchronized. If it is less than the threshold, training stops and a classification model is obtained. The coarse-grained assisted fine-grained self-learning realizes the efficient use of the dynamic images in the coarse-grained label, improving the generalization and robustness of the model.

[0074] In this embodiment, generating fine-grained pseudo-labels based on the fine-grained prediction score of the first dynamic image includes: normalizing the fine-grained prediction score of the first dynamic image to obtain a fine-grained prediction probability matrix; converting the fine-grained prediction probability matrix into a one-hot encoded probability distribution matrix, mapping the fine-grained category corresponding to the maximum class probability in the probability distribution matrix to the corresponding coarse-grained category, and converting it into a one-hot encoded coarse-grained label; determining whether the maximum class probability in the probability distribution matrix is ​​higher than a preset confidence threshold, and whether the one-hot encoded coarse-grained label is consistent with the real coarse-grained label of the dynamic image; if the maximum class probability is higher than the preset confidence threshold, and the one-hot encoded coarse-grained label is consistent with the real coarse-grained label, then the probability distribution matrix is ​​used as a fine-grained pseudo-label; otherwise, the probability distribution matrix is ​​not used as a fine-grained pseudo-label.

[0075] One-hot encoding uses an N-bit state register to encode N states. Each state has its own independent register bit, and at any given time, only one bit is valid, that is, only one bit is 1, and the rest are zero values.

[0076] The preset confidence threshold can be a confidence threshold set by the user in advance, for example, a predefined confidence threshold of 0.7, without any specific limitation here.

[0077] It is understood that in this embodiment, the fine-grained prediction probability matrix is ​​obtained by normalizing the fine-grained prediction score of the first dynamic image, and then converted into a one-hot encoded probability distribution matrix. The fine-grained category corresponding to the maximum class probability is mapped to the corresponding coarse-grained category and converted into a one-hot encoded coarse-grained label. It is determined whether the maximum class probability in the probability distribution matrix is ​​higher than the confidence threshold and whether the coarse-grained label is consistent with the true coarse-grained label of the dynamic image. If the maximum class probability is higher than the confidence threshold and the label is consistent, then the probability distribution matrix is ​​used as a fine-grained pseudo-label, which avoids the model from incorrectly labeling a fine-grained category as a fine-grained category that is very similar to it in another coarse category, thereby improving the accuracy of the model.

[0078] The dynamic image classification method proposed in this application acquires the dynamic image to be classified and identifies the feature map of each frame. This feature map is then input into a pre-trained classification model for classification matching, outputting the actual classification category corresponding to the dynamic image. A learnable neural network structure replaces the previous coarse-grained classification method based on minpooling rules to aid in the training of the fine-grained model, improving the model's generalization and robustness. This solves the problem in related technologies where methods for fine-grained classification tasks fail to improve the performance of fine-grained image classification models, resulting in low generalization and robustness.

[0079] The following will combine Figure 2 The classification method for dynamic images is described in detail, and the specific process is as follows:

[0080] 1. Determine the dataset:

[0081] In addition to conducting extensive experiments on the WGIF animation dataset to verify the powerful performance of the CRAFT method on fine-grained animation classification tasks based on coarse supervision information, this application also conducted generalization experiments on the CIFAR-100 fine-grained image classification benchmark dataset under the same problem setting to test the universality and effectiveness of the proposed coarse-grained assisted fine-grained self-learning framework.

[0082] The WGIF dataset is comprehensive and diverse, consisting of over 8,000 animated images from 73 categories. These 73 categories belong to 14 coarse classes. Ten of these coarse classes contain 2-5 subclasses, three coarse classes contain 7-8 subclasses, and one coarse class contains 12 subclasses. Each subclass belongs to exactly one of the coarse classes. The CIFAR-100 dataset, which shows the hierarchical relationships between class labels, consists of 60,000 color images from 20 coarse-grained categories, each image being 32x32 pixels. These 20 coarse-grained categories are further divided into 100 fine-grained categories, with exactly five fine-grained categories corresponding to one coarse-grained category. Each fine-grained category includes 500 training images and 100 test images.

[0083] 2. Obtain different perturbation versions of the same input using two types of data perturbation.

[0084] (1) Slight perturbation: Randomly flip the input data horizontally with a 50% probability, then fill the input data vertically and horizontally to 9 / 8 of the original, and then randomly crop it according to a fixed size of 224x224 pixels.

[0085] (2) Drastic perturbation: In addition to performing all operations with slight perturbation, inspired by UDA (Unsupervised Data Augmentation) and Fixmatch (semi-supervised learning method), it continues to perturb the input data with RandAugment (data augmentation strategy) and Cutout (data cutting strategy) to produce a severely distorted version of the given input.

[0086] 3. Model Design

[0087] The CRAFT framework described above consists of two parallel branches, named the fine-grained pseudo-label generation branch and the trunk branch, respectively. Both branches use the same network: an adaptive multi-granularity animation classification network, which performs both fine-grained and coarse-grained classification predictions on the animation simultaneously. The trunk branch is used for training the classification model, while the pseudo-label generation branch directly shares the network parameters of the trunk branch and does not participate in the training.

[0088] This application's embodiments primarily utilize a novel coarse-grained auxiliary fine-grained self-learning method formed by combining an adaptive multi-granularity animated image classification network and a coarse-grained auxiliary method. This extends pseudo-label technology to weakly supervised learning tasks with imprecise supervision, generating fine-grained pseudo-labels for coarse-grained annotated animated images. This allows the fine-grained classification model to be trained in a strongly supervised manner (i.e., with fine-grained labels as supervision) by jointly using subsets of coarse-grained and fine-grained annotations, thereby improving model performance.

[0089] Specifically, the principles of the adaptive multi-granularity animated graph classification network and the principle of fine-grained self-learning with coarse-grained assistance in the model will be explained in detail below:

[0090] (1) The principle of adaptive multi-granularity animated graph classification network is introduced, such as... Figure 3 As shown, specifically:

[0091] For animated images with fine-grained annotations, the adaptive multi-granularity animated image classification network only uses the fine-grained classification branch to perform fine-grained classification; while for animated images with only coarse-grained labels, in order to maximize the use of them to improve the performance of the fine-grained classification model, the adaptive multi-granularity animated image classification network uses both fine-grained and coarse-grained classification branches to predict its fine-grained and coarse-grained labels.

[0092] Next, taking the main branch as an example, the adaptive multi-granularity animated graph classification network will be explained in detail as follows:

[0093] Given a coarse-grained labeled dynamic with a violent perturbation Figure X s First, a common 2D convolutional neural network is used as the backbone network to extract the violently perturbed version of the dynamics. Figure X s Feature map F s ∈R T×C×H×W Where T is from X in the animation. s The number of image frames extracted is given, and C, H, and W represent the number of channels, height, and width of the feature map in each frame, respectively. To avoid excessive complexity in the subsequent network structure due to an excessive number of neurons, which could lead to overfitting of the classification model, the adaptive multi-granularity animated image classification network then uses a GAP operation to spatially pool the generated feature maps, thereby reducing the dimensionality of the feature maps. Figure X s The feature representation after dimensionality reduction is as follows

[0094] To address the limitation of existing fine-grained image classification methods for solving the same problem setting, where coarse-grained classification based on min-pooling rules is only applicable to datasets with perfect hierarchical structure, this application employs a dual-branch network structure that can learn both coarse and fine-grained classification, simultaneously performing coarse-grained and fine-grained classification predictions to improve the model's generalization and robustness. Simultaneously, the adaptive multi-granularity animated image classification network utilizes a keyframe attention mechanism to address the problem of sparse semantically relevant keyframes in internet animated image content classification tasks.

[0095] Specifically, three parallel modules—the fine-grained classifier, the attention module, and the coarse-grained classifier—follow the spatial pooling operation and are used to perform fine-grained and coarse-grained classification on the animation: Let N...f N represents the number of fine-grained categories. c This indicates the number of coarse-grained categories, representing the dynamic version of the violent perturbation. Figure X s Features After being input into two classifiers, fine-grained T-frame prediction scores are obtained respectively. and coarse-grained T-frame prediction scores

[0096] At the same time, the attention module also receives features. As input, generate a violently perturbated version of the dynamics. Figure X s Attention weights of T-frame images in It indicates that it is a movement. Figure X s The correlation between the t-th frame image and the true category label of the animation is considered; the closer the value is to 1, the more important the image frame is for classification. Since both the fine-grained and coarse-grained classifiers are single-layer linear classifiers, the network structure of the attention module is roughly the same. Finally, a... s Fine-grained frame prediction scores S for T-frame images sf and coarse-grained frame prediction score S sc Perform a weighted summation to obtain a version of the dynamics with significant perturbation. Figure X s Fine-grained prediction score and coarse-grained prediction score g sf Formalized as:

[0097]

[0098] Similarly, g sc Represented as:

[0099]

[0100] in They represent movement respectively Figure X s The fine-grained prediction score and coarse-grained prediction score of the t-th frame image.

[0101] It is important to note that although the fine-grained pseudo-label generation branch and the main branch use the same network structure, the coarse-grained animation prediction score output by the adaptive multi-granularity animation classification network of the fine-grained pseudo-label generation branch is directly discarded during the forward propagation of the network because the fine-grained pseudo-label generation branch is only used to generate fine-grained pseudo-labels based on the fine-grained animation prediction scores it generates.

[0102] (2) The principle of fine-grained self-learning with coarse-grained assistance is introduced as follows:

[0103] The core idea of ​​coarse-grained assisted fine-grained self-learning is to automatically generate fine-grained pseudo-labels with coarse label constraints for coarse-grained labeled animated images to supervise the training of fine-grained models. As mentioned earlier, previous fine-grained image classification methods for solving the same problem setting only enhanced the model's general feature extraction capabilities using coarse-grained labeled data, offering very limited help for fine-grained classification tasks. To address this, fine-grained pseudo-labels are automatically assigned to the coarse-grained labeled animated image samples in the training set to maximize the utilization of the coarse-labeled data. The pseudo-label technique is primarily used to generate labels for unlabeled data, handling incompletely supervised learning tasks.

[0104] This application extends pseudo-labeling technology to weakly supervised learning tasks with imprecise supervision, generating fine-grained pseudo-labels for coarse-grained annotated animations to supervise fine-grained model training. Furthermore, for coarse-grained annotated animations, their coarse-grained labels are used to further filter out erroneous fine-grained pseudo-labels generated by the model, preventing the fine-grained animation classification model from being misled by incorrect pseudo-labels. The coarse-grained-assisted fine-grained self-learning achieves the processing of the coarse-grained annotation subset S. coarse The efficient use of animated graphs in the network is achieved through two parallel branches: a fine-grained pseudo-label generation branch and a main branch, both with the same network structure. Specifically: given a coarse-grained labeled animated graph x∈S coarse Its coarse-grained label is an N c A one-hot encoded vector of dimension y is denoted as y c .

[0105] First, two different levels of data perturbation were applied to the animated image x to obtain a slightly perturbated version. Figure X w and violently disturbed versions of motion Figure X s Then move Figure X s The adaptive multi-granularity animated graph classification network fed into the main branch performs both fine-grained and coarse-grained classification predictions, obtaining a fine-grained prediction score g. sf and coarse-grained prediction score g sc ,move Figure X w The data is then simultaneously fed into the adaptive multi-granularity animated image classification network of the fine-grained pseudo-label generation branch for both fine-grained and coarse-grained classification predictions. However, unlike the main branch, the fine-grained pseudo-label generation branch directly discards slightly perturbed versions of the animated image. Figure X w The coarse-grained prediction score is used, and only the fine-grained prediction score is retained. Use it to generate fine-grained pseudo-labels from the coarse-grained animated image x. This pseudo-tag was used as the dynamic version of the violent disturbance. Figure X s The training of the adaptive multi-granularity animation classification network in the main branch is supervised by real fine-grained labels and coarse-grained labels annotated manually. In order to generate more accurate fine-grained pseudo-labels for the coarse-grained annotated animations, the fine-grained pseudo-label generation branch will automatically synchronize the network parameters of the main branch after each update of the network parameters of the main branch, so as to recalculate the fine-grained pseudo-labels.

[0106] Next, we will focus on how to utilize the slightly perturbed version of the motion. Figure X w Fine-grained prediction score g wf Generate reliable fine-grained pseudo-labels for coarse-grained annotated animation x:

[0107] First of all Perform softmax normalization to obtain fine-grained prediction probabilities. The probability of belonging to a certain fine-grained category Formal representation:

[0108]

[0109] Where τ∈(0,1] is the temperature parameter.

[0110] Then p wf Probability distribution of conversion to one-hot encoding This yields fine-grained pseudo-labels for the coarse-grained animated image x. However, these pseudo-labels cannot be used directly and need to be filtered through two conditions to ensure that the fine-grained pseudo-labels generated by the fine-grained pseudo-label generation branch are correct.

[0111] Condition 1: To ensure that the generated fine-grained pseudo-labels have sufficiently high confidence, the model sets a confidence threshold H (hyperparameter), requiring the fine-grained prediction probability p to be... wf The probability of the largest class must be higher than the predefined threshold H.

[0112] Condition two: To prevent the model from incorrectly labeling a fine-grained category as a very similar fine-grained category within another coarse-grained category (e.g., incorrectly labeling a fine-grained category like chinchilla (rodents) as a rabbit (a coarse-grained category), the CRAFT method also uses the true coarse-grained labels to assist in generating reliable fine-grained pseudo-labels based on the hierarchical relationship between coarse and fine-grained labels in the dataset. Specifically: first, based on the one-to-many hierarchical relationship between coarse and fine-grained labels, the fine-grained prediction probability p is... wfThe fine-grained category corresponding to the maximum value in the middle is mapped to the corresponding coarse-grained category, and then converted into a one-hot encoded coarse-grained label vector, denoted as p. wc (For example: map the fine-grained category of Husky with the highest probability value to the coarse-grained category of Dog); then ask p wc Compared to the animated image with coarse-grained annotations, the actual coarse-grained labels are shown below. c They are consistent.

[0113] Only when both of the above conditions are met will the pseudo-tag generation branch generate fine-grained pseudo-tags. Only then can it be adopted as a version of violent perturbation. Figure X s The true fine-grained labeling is formally represented as: Among them, I(p) wc ) is the indicator function, when the probability p is predicted by fine-grained methods. wf The calculated coarse-grained predicted label p wc Compared to true coarse-grained labels y c If the values ​​are the same, this indicator function takes the value 1; otherwise, it takes the value 0.

[0114] The formula is expressed as follows:

[0115]

[0116] I(max(p wf H) is also an indicator function, when the fine-grained prediction probability p wf When the probability of the largest class is greater than the predefined confidence threshold H, this indicator function takes the value of 1; otherwise, it takes the value of 0.

[0117] The formula is expressed as follows:

[0118]

[0119] Only when p wc =y c and max(p wf When both conditions are met, the fine-grained pseudo-tag generation branch generates fine-grained pseudo-tags. Only then can it be adopted.

[0120] 4. Model Training

[0121] Let S coarse ={(x b ,y bc ), b=1,...,B},S fine ={(x d ,y df ,y dc), d=1,...,D}, where B is the number of animated samples in the coarse-grained labeled subset, D is the number of animated samples in the fine-grained labeled subset, B+D=N, y bc This is a coarse-grained annotation animation x b coarse-grained tags, y df y dc These are fine-grained annotated animations x d Both fine-grained and coarse-grained tags are used, and both tags are represented using one-hot encoded vectors.

[0122] The key idea of ​​CRAFT is to use slightly perturbed animated graphs to generate one-hot encoded fine-grained pseudo-labels for supervision on a subset of coarse-grained labels, and use heavily perturbed animated graphs to generate both fine-grained and coarse-grained classification predictions.

[0123] The loss calculation formula for the coarse-grained labeled animations in the training set is as follows:

[0124]

[0125] Where, λ f and λ c It is a hyperparameter that balances the fine-grained classification loss and the coarse-grained classification loss of the coarse-grained labeled animation.

[0126] For finely labeled animated images in the training set, perform normal fine-grained classification. The cross-entropy classification loss is formalized as follows:

[0127]

[0128] Finally, the total loss function used for model training is:

[0129] L = L coarse +L fine

[0130] 5. Analysis of Experimental Results

[0131] (1) A detailed description of the model during the experiment:

[0132] In all experiments on the WGIF animated image dataset, a ResNet50 model pre-trained on ImageNet (a large visualization database used for research on visual object recognition software) was used as the backbone network. For fair comparison, the CRAFT method on the CIFAR-100 dataset used the same VGG and ResNet networks as the backbone network as the comparison method.

[0133] For all experiments on WGIF, in order to enable batch training, this embodiment of the application extracts 8 frames for each animated sample in the WGIF training set, wherein the experiment uses the loss coefficient λ by default. f =λc =0.5. Since different confidence thresholds on the WGIF dataset have varying impacts on the model's fine-grained animation classification performance, the confidence threshold, predefined based on previous experimental data, is set to H = 0.7, and the temperature parameter τ = 0.5. For the WGIF dataset, the total number of training iterations is set to 51200, the batch size is fixed at 12, and CRAFT uses batch SGD with a weight decay of 0.001 and a momentum parameter of 0.9 as the optimizer. The initial learning rate is 0.001, and as training progresses, the learning rate decays using a cosine annealing strategy. This embodiment uses the PyTorch deep learning framework and is trained on a server with three GTX 1080 Ti GPUs. For the CIFAR-100 dataset, for fairness, the same hyperparameter settings as the comparison method are used.

[0134] Among them, the comparison methods include: (1) a multi-stage learning method from coarse to fine. The core of this method is based on the idea of ​​curriculum learning, which uses the subordinate relationship information between category labels to construct a training curriculum from simple to complex. Specifically: the first stage learns a simple coarse-grained prediction model as a pre-trained model, and the second stage uses fine-grained labeling to fine-tune the model. (2) a method based on Min-pooling. Min-pooling uses the subordinate relationship between category labels to convert the fine-grained classification output into a coarse-grained output, thereby utilizing both types of labeling information at the same time. (3) a multi-scale and multi-level feature fusion method. Based on the Min-pooling method, more effective features are generated by introducing the Convolutional Block Attention Module (CBAM), Multi-scale Convolution Kernel Feature Fusion (MCKFF) feature fusion and Concatenating the Features from Different Levels (CFDL), which better utilizes the subordinate relationship between coarse and fine-grained labels.

[0135] The following specific example demonstrates the effectiveness of the proposed coarse-grained assisted fine-grained self-learning fine-grained animation classification method on the WGIF animation dataset, through comparative experiments. The method is as follows:

[0136] The ResNet50 architecture was selected as the backbone network for all comparative methods and the CRAFT method proposed in this chapter. Experiments were conducted to evaluate the methods at fine-grained annotation ratios of 20%, 30%, and 50% of the training set. Top-1 classification accuracy was used as the evaluation metric. The experimental results on the WGIF test set are as follows: Figure 4 As shown.

[0137] It is evident that the classification performance of the CRAFT method is superior to all the comparison methods in the experimental settings of three different scales of fine-grained labeled data. Specifically, in the experimental setting with a 50% fine-grained animated label ratio, the CRAFT model achieved a maximum Top-1 classification accuracy of 72.20%.

[0138] Compared to multi-stage learning methods, the CRAFT method greatly simplifies the training process and achieves significant improvements in classification performance (+2.56%).

[0139] Compared with the two-branch classification method that uses Min-pooling fixed rules for coarse-grained classification, the CRAFT method based on the adaptive multi-granularity animation classification network improves the classification performance by 1.28%. This shows that for animation datasets with imperfect hierarchical structures, the method provided in this application has better compatibility and stronger robustness.

[0140] The more complex multi-scale and multi-level feature fusion method based on the Min-pooling method only improves upon the Min-pooling method by 0.16 percentage points, and still lags behind the CRAFT method by 1.12%.

[0141] The above results strongly demonstrate the powerful performance of the proposed CRAFT method on fine-grained animated graph classification tasks based on coarse supervision information.

[0142] The following specific example demonstrates the generality and applicability of the coarse-grained assisted fine-grained self-learning framework to other fine-grained recognition tasks based on coarse-supervised information through experiments conducted on the commonly used fine-grained image classification benchmark dataset CIFAR-100. Details are as follows:

[0143] In the implementation, the keyframe attention pooling module designed for animated data was removed. This experiment was conducted based on a 50% fine-grained annotation ratio on the training set, and the Top-1 accuracy for fine-grained classification on the CIFAR-100 test set was as follows: Figure 5 As shown, it is clear that the CRAFT method based on both VGG and ResNet backbone networks outperforms the corresponding competing methods in classification performance, achieving the best fine-grained classification results (VGG: 74.43%, ResNet: 82.80%).

[0144] This experimental result shows that the CRAFT method is also applicable to fine-grained image classification tasks with insufficient fine-grained supervision, and the coarse-grained assisted fine-grained self-learning framework has a certain degree of universality.

[0145] 6. Ablation test

[0146] The following analysis will examine whether the model's performance will be affected from different perspectives, using ablation studies of different modules, fine-grained labeled data of different scales, and studies with different confidence thresholds:

[0147] (1) To verify the effectiveness of each module in the CRAFT method, this application embodiment conducted ablation experiments on different modules on the WGIF dataset to determine their roles in the overall model, such as... Figure 6 As shown, ablation experiments of different components in the CRAFT method are presented.

[0148] The experimental results show that:

[0149] 1) When the coarse-grained classification branch in the adaptive multi-granularity animation classification network is removed, the fine-grained classification performance of CRAFT on the animation classification dataset drops significantly (-1.92%), indicating that coarse-grained label information is helpful for model training. The adaptive multi-granularity animation classification network proposed in this chapter can effectively utilize coarse-grained supervision to improve fine-grained classification performance.

[0150] 2) More importantly, compared to removing coarse-grained branches, the performance degradation is more significant (-2.8%) when using Min-pooling as a coarse-grained classification method. In Min-pooling coarse-grained prediction, although the network training receives more supervision, the imperfect hierarchical structure of the WGIF animation dataset and the lack of obvious common features between fine-grained categories introduce unnecessary noise into this rule-based coarse-grained prediction and supervision, leading to severe performance degradation, even worse than not using coarse-grained supervision. Adaptive multi-granularity classification networks avoid this problem through a simple multi-branch design and can effectively utilize coarse-grained supervision.

[0151] 3) When generating fine-grained pseudo-labels for coarse-grained labeled data, removing the real coarse-grained label aids and using only a pseudo-label generation method based on a fixed threshold also resulted in a significant decrease in fine-grained classification performance (-1.96%). This demonstrates that introducing coarse-grained aids can greatly reduce the mislabeling of extremely similar fine classes within different coarse classes.

[0152] (2) The embodiments of this application evaluated the performance changes of the CRAFT method as the scale of fine-grained labeled data in the WGIF training set changed.

[0153] Using Top-1 classification accuracy as the evaluation metric, the results are as follows: Figure 7 As shown, the larger the proportion of fine-grained labeled data in the training set, the better the fine-grained classification performance of the CRAFT model. Specifically, when using 10% and 50% of the training data as fine-grained labeled data respectively, the Top-1 accuracy of fine-grained animation classification increases sequentially. With 100% fine-grained labeling as the upper limit (74.31%), the CRAFT method achieves 91% of the upper limit performance when using only 20% of the fine-grained labeled animations; when the proportion of fine-grained labeled data increases to 50%, the performance reaches 97% of the upper limit.

[0154] This clearly demonstrates that the CRAFT method can efficiently utilize coarse-grained labeled animation data in the training set, greatly reducing the classification model's dependence on fine-grained supervision and helping to improve the performance of fine-grained animation classification when fine-grained supervision is insufficient.

[0155] (3) Since the confidence threshold τ directly affects the number of available fine-grained pseudo-labels, this application embodiment studies the impact of different settings of τ on the performance of the classification model.

[0156] Experiments were conducted on the WGIF animation dataset using a 50% fine-grained animation annotation ratio, trying five different values ​​of τ∈{0.5, 0.6, 0.7, 0.8, 0.9}. The corresponding experimental results are as follows: Figure 8 As shown in the figure, the Top-1 classification accuracy on the test set initially increases and then decreases as the confidence threshold increases. The best classification performance (72.20%) is achieved when the confidence threshold is 0.7. This phenomenon is presumably caused by the following: when the confidence threshold is small, many erroneous fine-grained pseudo-labels are introduced, misleading the classification model and making the improvement in classification performance from the coarse-grained labeled data in the training set insignificant. As the confidence threshold increases, more and more erroneous pseudo-labels are filtered out, and the fine-grained classification performance gradually improves. However, when the confidence threshold is too large, almost all fine-grained pseudo-labels (including correct ones) generated by the model are filtered out, causing the fine-grained self-learning method assisted by coarse-grained data to fail, and the model performance to decline significantly.

[0157] In summary, the embodiments of this application directly solve the problem of insufficient fine-grained labeled animations by generating fine-grained pseudo-labels with coarse-grained label constraints for coarse-grained labeled animations in the training set. It uses a learnable neural network structure to replace the previous coarse-grained classification method based on the Minpooling rule to help train the fine-grained model, thereby improving the generalization and robustness of the model. Extensive experiments on multiple datasets show that the method provided by the embodiments of this application achieves superior performance compared with existing methods.

[0158] Next, referring to the accompanying drawings, a dynamic image classification apparatus according to an embodiment of this application is described.

[0159] Figure 9 This is a block diagram of a dynamic image classification device according to an embodiment of this application.

[0160] like Figure 9 As shown, the dynamic image classification device 10 includes an acquisition module 100 and a processing module 200.

[0161] The acquisition module 100 is used to acquire the dynamic image to be classified; the processing module 200 is used to identify the feature map of each frame in the dynamic image, input the feature map of each frame into the pre-trained classification model, and output the actual classification category of the dynamic image. The classification model includes a first branch network, which includes a fine-grained classifier and a coarse-grained classifier. The fine-grained classifier and the coarse-grained classifier are used to generate fine-grained prediction scores and coarse-grained prediction scores of the dynamic image, respectively, and the actual classification category of the dynamic image is matched according to the fine-grained prediction scores and / or coarse-grained prediction scores.

[0162] In this embodiment, the processing module 200 is further configured to: input the feature map of each frame image into the fine-grained classifier, the coarse-grained classifier, and the attention module respectively, and output the fine-grained frame prediction score, the coarse-grained frame prediction score, and the weight of each frame image; perform a weighted sum based on the fine-grained frame prediction score and the weight of each frame image to obtain the fine-grained prediction score of the dynamic image, and perform a weighted sum based on the coarse-grained frame prediction score and the weight to obtain the coarse-grained prediction score of the dynamic image; if the network is in the forward propagation calculation stage, match the actual classification category of the dynamic image according to the fine-grained prediction score; otherwise, match the actual classification category of the dynamic image according to the fine-grained prediction score and the coarse-grained prediction score.

[0163] In this embodiment of the application, the processing module 200 is further configured to: perform spatial pooling on the feature map of each frame image to obtain the dimensionality reduction features of each frame image.

[0164] In this embodiment, the processing module 200 is further configured to: acquire a training dataset, wherein the training dataset includes coarse-grained labeled dynamic images; perturb the data of the coarse-grained labeled dynamic images to obtain a first dynamic image and a second dynamic image, wherein the perturbation level of the first dynamic image is less than that of the second dynamic image; input the first dynamic image into a first branch network, output a fine-grained prediction score of the first dynamic image, generate a fine-grained pseudo-label based on the fine-grained prediction score of the first dynamic image, and use the fine-grained pseudo-label as the real fine-grained label of the second dynamic image; input the second dynamic image into a second branch network, Output the fine-grained prediction score and coarse-grained prediction score of the second dynamic image. Calculate the fine-grained cross-entropy classification loss based on the fine-grained prediction score and the true fine-grained label of the second dynamic image, and calculate the coarse-grained cross-entropy classification loss based on the coarse-grained prediction score and the coarse-grained labeled dynamic image. Calculate the total training loss based on the fine-grained cross-entropy classification loss and the coarse-grained cross-entropy classification loss. If the total loss is greater than a preset threshold, continue training the second branch network based on the trained dataset, and synchronously update the network parameters of the second branch network and the first branch network until the total loss is less than or equal to the preset threshold. Stop training and obtain the classification model.

[0165] In this embodiment, the processing module 200 is further configured to: normalize the fine-grained prediction score of the first dynamic image to obtain a fine-grained prediction probability matrix; convert the fine-grained prediction probability matrix into a one-hot encoded probability distribution matrix, map the fine-grained category corresponding to the maximum class probability in the probability distribution matrix to the corresponding coarse-grained category, and convert it into a one-hot encoded coarse-grained label; determine whether the maximum class probability in the probability distribution matrix is ​​higher than a preset confidence threshold, and whether the one-hot encoded coarse-grained label is consistent with the real coarse-grained label of the dynamic image; if the maximum class probability is higher than the preset confidence threshold, and the one-hot encoded coarse-grained label is consistent with the real coarse-grained label, then the probability distribution matrix is ​​used as a fine-grained pseudo-label; otherwise, the probability distribution matrix is ​​not used as a fine-grained pseudo-label.

[0166] It should be noted that the explanation of the above-described embodiments of the dynamic image classification method also applies to the dynamic image classification device of this embodiment, and will not be repeated here.

[0167] The dynamic image classification apparatus proposed in this application acquires the dynamic image to be classified and identifies the feature map of each frame. This feature map is then input into a pre-trained classification model for classification matching, and the actual classification category corresponding to the dynamic image is output. A learnable neural network structure replaces the previous coarse-grained classification method based on the Minpooling rule to assist in the training of the fine-grained model, improving the model's generalization and robustness. This solves the problem in related technologies where methods for fine-grained classification tasks cannot improve the performance of fine-grained image classification models, resulting in low generalization and robustness.

[0168] Figure 10 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:

[0169] The memory 1001, the processor 1002, and the computer program stored on the memory 1001 and capable of running on the processor 1002.

[0170] When the processor 1002 executes the program, it implements the dynamic image classification method provided in the above embodiments.

[0171] Furthermore, the vehicle also includes:

[0172] Communication interface 1003 is used for communication between memory 1001 and processor 1002.

[0173] The memory 1001 is used to store computer programs that can run on the processor 1002.

[0174] The memory 1001 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0175] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0176] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0177] The processor 1002 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0178] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for classifying dynamic images.

[0179] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0180] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0181] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0182] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0183] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0184] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for classifying dynamic images, characterized in that, Includes the following steps: Obtain the dynamic image to be classified; The feature map of each frame in the dynamic image is identified, and the feature map of each frame is input into a pre-trained classification model to output the actual classification category of the dynamic image. The classification model includes a first branch network, and the first branch network includes a fine-grained classifier and a coarse-grained classifier. The fine-grained classifier and the coarse-grained classifier are used to generate a fine-grained prediction score and a coarse-grained prediction score of the dynamic image, respectively, and the actual classification category of the dynamic image is matched according to the fine-grained prediction score and / or the coarse-grained prediction score. The classification model further includes a second branch network, wherein the first branch network and the second branch network have the same structure, and the training process of the classification model includes: Obtain a training dataset, wherein the training dataset includes coarse-grained labeled dynamic images; The coarse-grained labeled dynamic image is perturbed to obtain a first dynamic image and a second dynamic image, wherein the perturbation degree of the first dynamic image is less than that of the second dynamic image. The first dynamic image is input into the first branch network, and the fine-grained prediction score of the first dynamic image is output. A fine-grained pseudo-label is generated based on the fine-grained prediction score of the first dynamic image, and the fine-grained pseudo-label is used as the real fine-grained label of the second dynamic image. The second dynamic image is input into the second branch network, and the fine-grained prediction score and coarse-grained prediction score of the second dynamic image are output. The fine-grained cross-entropy classification loss is calculated based on the fine-grained prediction score of the second dynamic image and the real fine-grained label, and the coarse-grained cross-entropy classification loss is calculated based on the coarse-grained prediction score and the coarse-grained labeled dynamic image. The total training loss is calculated based on the granular cross-entropy classification loss and the coarse-grained cross-entropy classification loss. If the total loss is greater than a preset threshold, the second branch network is trained on the trained dataset, and the network parameters of the second branch network and the first branch network are updated synchronously until the total loss is less than or equal to the preset threshold. Then, training stops, and the classification model is obtained.

2. The method according to claim 1, characterized in that, The first branch network further includes an attention module, which inputs the feature map of each frame image into a pre-trained classification model and outputs the actual classification category of the dynamic image, including: The feature map of each frame image is input into the fine-grained classifier, the coarse-grained classifier, and the attention module, respectively, and the fine-grained frame prediction score, coarse-grained frame prediction score, and weight of each frame image are output. The fine-grained prediction score of the dynamic image is obtained by weighted summation based on the fine-grained frame prediction score of each frame and the weight, and the coarse-grained prediction score of the dynamic image is obtained by weighted summation based on the coarse-grained frame prediction score and the weight. If the network is in the forward propagation calculation stage, the actual classification category of the dynamic image is matched according to the fine-grained prediction score; otherwise, the actual classification category of the dynamic image is matched according to both the fine-grained prediction score and the coarse-grained prediction score.

3. The method according to claim 1, characterized in that, Before inputting the feature map of each frame image into the pre-trained classification model, the process includes: Spatial pooling is performed on the feature map of each frame to obtain the dimensionality-reduced features of each frame.

4. The method according to claim 1, characterized in that, The generation of fine-grained pseudo-labels based on the fine-grained prediction score of the first dynamic image includes: The fine-grained prediction score of the first dynamic image is normalized to obtain the fine-grained prediction probability matrix; The fine-grained prediction probability matrix is ​​converted into a one-hot encoded probability distribution matrix. The fine-grained category corresponding to the maximum class probability in the probability distribution matrix is ​​mapped to the corresponding coarse-grained category and converted into a one-hot encoded coarse-grained label. Determine whether the maximum class probability in the probability distribution matrix is ​​higher than a preset confidence threshold, and whether the coarse-grained label of the one-hot encoding is consistent with the real coarse-grained label of the dynamic image. If the maximum class probability is higher than the preset confidence threshold, and the coarse-grained label encoded by the one-hot encoding is consistent with the true coarse-grained label, then the probability distribution matrix is ​​used as the fine-grained pseudo-label; otherwise, the probability distribution matrix is ​​not used as the fine-grained pseudo-label.

5. A dynamic image classification device, characterized in that, For implementing the method as described in any one of claims 1-4, comprising: The acquisition module is used to acquire the dynamic images to be classified. The processing module is used to identify the feature map of each frame in the dynamic image, input the feature map of each frame into a pre-trained classification model, and output the actual classification category of the dynamic image. The classification model includes a first branch network, and the first branch network includes a fine-grained classifier and a coarse-grained classifier. The fine-grained classifier and the coarse-grained classifier are used to generate a fine-grained prediction score and a coarse-grained prediction score of the dynamic image, respectively, and the actual classification category of the dynamic image is matched according to the fine-grained prediction score and / or the coarse-grained prediction score.

6. The apparatus according to claim 5, characterized in that, The processing module is used for: The feature map of each frame image is input into the fine-grained classifier, the coarse-grained classifier, and the attention module, respectively, and the fine-grained frame prediction score, coarse-grained frame prediction score, and weight of each frame image are output. The fine-grained prediction score of the dynamic image is obtained by weighted summation based on the fine-grained frame prediction score of each frame and the weight, and the coarse-grained prediction score of the dynamic image is obtained by weighted summation based on the coarse-grained frame prediction score and the weight. If the network is in the forward propagation calculation stage, the actual classification category of the dynamic image is matched according to the fine-grained prediction score; otherwise, the actual classification category of the dynamic image is matched according to both the fine-grained prediction score and the coarse-grained prediction score.

7. The apparatus according to claim 5, characterized in that, The processing module is further used for: Spatial pooling is performed on the feature map of each frame to obtain the dimensionality-reduced features of each frame.

8. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method for classifying dynamic images as described in any one of claims 1-4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method for classifying dynamic images as described in any one of claims 1-4.

Images