Image classification model training method, image classification method, model, computer equipment and medium
Through the training method of image triplets, the image features are processed using multi-head cross attention and self-attention layers. Based on the comparison loss training model, the problem of insufficient overfitting and generalization ability caused by a small number of training samples is solved, and the image is improved Discrimination and generalization ability of classification models.
Patent Information
- Application Number
- CN202311558413.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-21
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-11-21
AI Technical Summary
In the field of computer vision, image classification models trained with a small number of training samples are prone to overfitting and perform poorly in unseen test categories, especially when differentiating similar visual concepts.
Using the training method of image triplets, image features are extracted through the multi-head cross-attention layer and the multi-head self-attention layer, feature cross-processing and enhancement are performed using the cross-attention layer and self-attention layer, and the model is trained based on the contrast loss to make the query The features of the image are close to the features of the target support image, and the query image is far away from the features of the other support image.
It improves the discriminant ability of small sample learning and the discriminating ability similar to visual concepts, and enhances the generalization ability and generalization performance of the model.
Smart Images

Figure CN117541868B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a training method for an image classification model, an image classification method, a model, a computer device and a medium. Background Art
[0002] In the field of computer vision, when the number of training sample data is small, supervised learning methods are used to train models, which is prone to overfitting. Small sample learning can use limited training samples when building models and generalizes well in unseen test categories.
[0003] In related technologies, various methods are used to simulate how humans quickly learn new visual concepts. For example, a meta-learning framework is used to perform small sample visual tasks. Or a metric-based method is used to measure the similarity between support and query samples. However, these methods are still prone to errors when using very few samples to simulate the distinction between similar visual concepts. Summary of the invention
[0004] The present application provides a training method for an image classification model, an image classification method, a model, a computer device and a medium, which can improve the discrimination ability of small sample learning.
[0005] The first aspect of the present application discloses a training method for an image classification model, which is applied to a client, and the method includes obtaining a training sample including an image triplet, wherein the image triplet includes a query image, a positive image belonging to the same category as the query image, and a negative image belonging to a different category from the query image; extracting vector features of the query image, the positive image, and the negative image; cross-attention processing of the vector features of the query image, the positive image, and the negative image using a multi-head cross-attention layer; enhancing the representation of the output of the cross-attention layer using a multi-head self-attention layer; calculating the loss based on the output of the self-attention layer; and training the image classification model based on the loss.
[0006] In a possible implementation of the first aspect above, the cross-attention processing includes performing four cross-attention operations on the three branches of the vector feature respectively.
[0007] In a possible implementation of the first aspect above, the cross-attention layer includes a first gating unit and a second gating unit, wherein the first gating unit is configured to adjust the attention matrix of the positive image and the reconstructed features of the query image, and the second gating unit is configured to adjust the attention matrix of the negative image and the reconstructed features of the query image.
[0008] In a possible implementation of the first aspect above, the self-attention layer includes a third gating unit and a fourth gating unit, wherein the third gating unit is configured to calculate the similarity between the positive image, the query image and the negative image, and the fourth gating unit is configured to calculate the similarity between the negative image, the query image and the positive image.
[0009] In a possible implementation of the first aspect above, the training sample also includes a second image triplet, which includes a second query image, a second negative image belonging to a different category from the second query image, and a third negative image belonging to a different category from the second query image, and the second image triplet is arranged in the training sample according to a predetermined ratio.
[0010] In a possible implementation of the first aspect above, the predetermined ratio is 0.2.
[0011] The second aspect of the present application discloses an image classification method, which is applied to a client, including obtaining a test image, a first supporting image, and a second supporting image, wherein the first supporting image and the second supporting image belong to different categories; inputting the test image, the first supporting image, and the second supporting image into a trained image classification model, wherein the trained image classification model is trained according to the first aspect of the present application; and outputting the category of the test image.
[0012] The third aspect of the present application discloses an image classification model, which includes a feature extraction layer for extracting high-order features of image samples, wherein the image samples are image triplets, including a query image, a third supporting image, and a fourth supporting image; a cross-attention layer for cross-attention processing of the high-order features; a self-attention layer for enhancing the representation of the output of the cross-attention layer; and a loss calculation layer for calculating the loss based on the output of the self-attention layer.
[0013] In a possible implementation of the third aspect above, the cross-attention layer includes a first gating unit and a second gating unit, wherein the first gating unit is configured to adjust the attention matrix of the third support image and the reconstructed features of the query image, and the second gating unit is configured to adjust the attention matrix of the fourth support image and the reconstructed features of the query image.
[0014] In a possible implementation of the third aspect above, the self-attention layer includes a third gating unit and a fourth gating unit, wherein the third gating unit is configured to calculate the similarity between the third support image, the query image and the fourth support image, and the fourth gating unit is configured to calculate the similarity between the fourth support image, the query image and the third support image.
[0015] The fourth aspect of the present application discloses a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the methods according to the first and second aspects of the present application.
[0016] The fifth aspect of the present application discloses a computer-readable medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the methods of the first and second aspects of the present application.
[0017] The sixth aspect of the present application discloses a computer program product, which implements the methods of the first and second aspects of the present application when the computer program is executed by a processor.
[0018] The present application provides a training method for an image classification model, an image classification method, a model, a computer device, and a medium, wherein the training samples include a query image, a positive image belonging to the same category as the query image, and a negative image belonging to a different category from the query image. In addition, in the multi-head cross-attention layer, the query image is used as an anchor point to perform a cross-attention operation with support images including positive and negative images; in the multi-head self-attention layer, each support image is self-attended to enhance the representation of each image. In addition, with contrast loss as a learning target, the features of the query image are made closer and closer to the features of the target support image, and the query image is made farther and farther from the features of another support image. The training method for the image classification model of the present application has a high learning discrimination ability on multiple small sample learning benchmarks, and has a high ability to distinguish similar visual concepts. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 A schematic diagram of a model for small sample learning according to an embodiment of the present application;
[0020] Figure 2 A flowchart of a method for training an image classification model according to an embodiment of the present application;
[0021] Figure 3 A schematic diagram of a cross-attention block according to an embodiment of the present application;
[0022] Figure 4A schematic diagram of a self-attention block according to an embodiment of the present application;
[0023] Figure 5 A training example of an image classification model for an embodiment of the present application;
[0024] Figure 6 A schematic diagram of a Transformer block of an embodiment of the present application;
[0025] Figure 7 A schematic diagram of a competition gate according to an embodiment of the present application;
[0026] Figure 8 A schematic diagram of a reference gate according to an embodiment of the present application;
[0027] Fig. 9 A schematic diagram for comparing self-attention, gated cross-attention and gated self-attention according to an embodiment of the present application;
[0028] Fig.10 A schematic diagram of feature visualization comparison of an image classification model according to an embodiment of the present application;
[0029] Fig.11 A schematic diagram of the structure of an image classification model according to an embodiment of the present application;
[0030] Fig.12 A schematic diagram of a flow chart of an image classification method according to an embodiment of the present application;
[0031] Fig.13 A schematic diagram of the structure of an electronic device according to an embodiment of the present application;
[0032] Fig.14a A schematic diagram of the structure of a computing device according to an embodiment of the present application;
[0033] Fig.14b A schematic diagram of the structure of a computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION
[0034] The present application is further described below in conjunction with specific embodiments and drawings. It is understood that the illustrative embodiments of the present disclosure include but are not limited to training methods for image classification models, image classification methods, models, computer devices and media, and the specific embodiments described herein are only for the purpose of explaining the present application, rather than limiting the present application. In addition, for ease of description, only some, but not all, structures or processes related to the present application are shown in the drawings.
[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0036] In the field of computer vision, supervised learning methods are usually used to train models to distinguish image types. However, in deep neural networks, when the training samples only include a small amount of labeled data, this method is prone to overfitting, and the trained model will show weak generalization ability for previously unseen classes.
[0037] In some implementations, more advanced algorithms, such as small sample learning methods, may be used to solve the above problems. However, these methods are still prone to errors when distinguishing similar visual concepts.
[0038] Few-shot learning refers to using limited training samples when building a model to learn previously unseen concepts. The trained model in few-shot learning needs to generalize well in unseen test categories. Figure 1 A schematic diagram of a model 100 for small sample learning according to an embodiment of the present application is shown.
[0039] The input of model 100 is support image 101 and query image 102. The set of support image 101 is Where N represents the image category, and K is the number of images contained in each category. The set of query images 102 is The goal of the model 100 is to learn a classifier to classify the query image and obtain an output 103. For a given test image, the corresponding classification result can be obtained using the trained model 100. Usually, there are only limited samples in each category, so related methods cannot distinguish different visual concepts with similar appearance.
[0040] In order to distinguish similar visual concepts in small sample learning, an embodiment of the present application provides a training method 200 for an image classification model, which not only focuses on the semantics of the image itself, but also uses other images as references to extract discriminative features of the image. The visual discrimination ability in small sample learning is improved through multi-head cross attention operation, multi-head self-attention operation and loss calculation.
[0041] Reference below Figure 2 The various steps S210-S260 in the image classification model training method 200 are explained in detail.
[0042] In S210 , a training sample including an image triplet is obtained, where the image triplet includes a query image, a positive image belonging to the same category as the query image, and a negative image belonging to a different category from the query image.
[0043] The training samples of model 100 are image triplets. The image triplets include the query image x a , and the query image x a Positive images x belonging to the same categoryp and the query image x a Negative images x belonging to different categories n That is, the set S of supporting images includes the positive image x p and the negative image x n .
[0044] When the number of training samples is limited, the small sample classification task can be decomposed into multiple conditional representation learning tasks. Specifically, given a query image x q , the support images in S can be ranked according to their semantic relevance to the query.
[0045]
[0046] where x k ∈S, and represent Compare Closer to x q The key here is how to determine the support image or Closer to x q In method 200, a conditional feature learning scheme is proposed to learn Simultaneously learn feature descriptions.
[0047]
[0048] Ψ is the conditional representation learning model, represents the query image z q The features of the image are dependent on the other two images. The conditional feature learning process avoids using a unified feature space for all images, so it can easily adapt to new object categories and improve the generalization ability of the model.
[0049] In S220, vector features of the query image, the positive image, and the negative image are extracted.
[0050] For an image triplet, vector features of the query image, positive image, and negative image can be extracted. In some examples, the high-level features (z p , z a , z n ), where z i =F(x i )∈R n ×d, n represents the number of tokens.
[0051] In S230, a cross-attention layer is used to perform cross-attention processing on the vector features of the query image, the positive image, and the negative image.
[0052] Although the extracted vector features, such as high-level features (z p , z a , z n ) contain rich semantic information, but they lack interaction, so the model training cannot benefit from their relative semantic relationship. Using multi-head cross attention layer to handle model dependencies between different images can enhance the interaction between different images.
[0053] In some examples, the multi-head cross-attention processing includes four cross-attention operations on the three branches of the vector features of the query image, the positive image, and the negative image for feature learning. The three branches are three linearly mapped token sets Q, K, and V. For a given input token set z1, z2, z3∈R n ×d, the multi-head cross attention (MHCA) processing can include linear projection and attention operations, see formula (3).
[0054]
[0055]
[0056]
[0057] Among them, W Q , W K , W V is the linear projection matrix. In the multi-head cross attention process, the token sets Q, K, and V come from different image samples.
[0058] In some embodiments, the cross-attention layer includes two gating units, namely a first gating unit and a second gating unit. The first gating unit is used to adjust the attention matrix of the positive image and the reconstruction features of the query image, and the second gating unit is used to adjust the attention matrix of the negative image and the reconstruction features of the query image. The two gating units are used to simulate the competition between the two support images, and can therefore be referred to as competition gates. The competition between the two support images is achieved by indicating a gating scheme that indicates the reconstruction process between each support query-image pair.
[0059] For a given image triplet feature (z p , z a , z n ), two competing gates G1 and G2∈R N They correspond to two supporting images respectively. Formula (4) lists the working principles of the two competitive gates.
[0060]
[0061]
[0062] z′ a =G1⊙MHCA(z a , z p , z p )+G2⊙MHCA(z a , z n , z n )+z a , (4)
[0063] where ⊙ represents element-wise multiplication, G′1, G′2∈R N×N is the result of G1 and G2 being repeated N times. In formula (4), the first competitive gate G1 corresponds to the positive image and works in the attention matrix of the positive image; the second competitive gate G2 corresponds to the negative image and works in the attention matrix of the negative image. In addition, the first competitive gate and the second competitive gate do not work on the attention matrix of the query image, but process the reconstructed features of the query image.
[0064] Figure 3 FIG. 3 is a schematic diagram of a cross-attention transformer block 310 according to an embodiment of the present application. p , z a , z n ) is input to the cross-attention transformer block 310, and the three branches of each feature enter the four multi-head cross-attention modules MHCA 314, MHCA 315, MHCA 316 and MHCA 317 for four cross-attention operations. Figure 3 The first competitive gate G1 in acts in the attention matrix of the positive image. The first competitive gate G1 acts directly on MHCA 314, where MHCA 314 can be expressed as MHCA / G 314 (with gating). The second competitive gate G2 acts in the attention matrix of the negative image. The second competitive gate G2 acts directly on MHCA 317, where MHCA 317 can be expressed as MHCA / G 317 (with gating). For the positive and negative images in the support set, the competition between the two support images is simulated by the competitive gate. Given two support images from different categories, the query image can only share the same label with one of the two images. The setting of the competitive gate can reduce the influence of each other on the features of the query image.
[0065] In addition, for the query image, a cross-attention operation is used to extract the salient features it shares with one of the two support images, and the first competitive gate G1 and the second competitive gate G2 process its reconstructed features (see the third formula in formula (4)).
[0066] Therefore, controlling the cross-attention transformer block 310 through two competitive gates can more accurately model the semantic relationship between image triplets.
[0067] Figure 3 The transformer block 310 in also includes a feed-forward layer FFN0 318. The features of the image triplet (z p , z a , z n After )(311-313) is input to the transformer block 310, the feature (z′) is processed. p , z′ a , z′ n )(311-313).
[0068] In S240, the self-attention layer is used to enhance the representation of the output of the cross-attention layer.
[0069] After feature reconstruction in the cross-attention block, multi-head self-attention is used to enhance the representation of each image. The processing of the self-attention layer can highlight important image regions and integrate the discriminative information into the [cls] token to generate the final representation.
[0070] In some examples, two gating units are included in the self-attention layer, which are configured to retrieve object details or background context shared between support-query image pairs. The two gating units here include a third gating unit and a fourth gating unit. The third gating unit is used to calculate the similarity between the positive image, the query image, and the negative image, that is, the similarity between the positive image Vs. (query image and negative image). The fourth gating unit is used to calculate the similarity between the negative image, the query image, and the positive image, that is, the similarity between the negative image Vs. (query image and positive image). The brackets here represent feature vector concatenation. These two gating units are used to refer to the two support images to identify important hidden semantics, and can therefore be called reference gates.
[0071] Figure 4 FIG. 3 shows a schematic diagram of a multi-head self-attention transformer block 320 of an embodiment of the present application. Figure 4 In the feature (z′ p , z′ a , z′ n ) is input to the multi-head self-attention transformer block 320, and the three branches of each feature enter the three multi-head self-attention modules MHSA 324, MHSA 325, and MHSA 326 for self-attention operation. Figure 4The third gating unit G3 in the MHSA 324 acts on the MHSA 324, where the MHSA 324 can be represented as MHSA / G 324 (with gating). The third gating unit is directly used to reweight the token in the MHSA 324. The fourth gating unit G4 acts on the MHSA 326, where the MHSA 326 can be represented as MHSA / G 326 (with gating). The fourth gating unit is directly used to reweight the token in the MHSA 326.
[0072] The gated self-attention operation can be expressed as:
[0073]
[0074]
[0075]
[0076] In some examples, the multi-head self-attention transformer block 320 may also include a fifth gating unit that acts directly on the MHSA 325 to improve the accuracy of the classification model.
[0077] Figure 4 The transformer block 320 in also includes a feed-forward layer FFN4 327. The features of the image triplet (z′ p , z′ a , z′ n After )(331-333) is input to transformer block 320, it is processed to obtain feature (z″ p , z″ a , z″ n )(321-323).
[0078] In S250, the loss is calculated based on the output of the self-attention layer.
[0079] The goal of the image classification model in this application is to maximize the similarity between images of the same category, while minimizing the similarity between images of different categories.
[0080] In some examples, calculating the contrastive loss includes calculating the similarity between the outputs of the self-attention layer; and calculating the mean squared error between the similarity and a target of the similarity.
[0081] For example, we can use cosine similarity to measure the similarity of triplets:
[0082]
[0083] Among them, τ is a learnable parameter initialized to 1. For a given image triplet (z p , za , z n ), can be based on the output of the image classification model (z″ p , z″ a , z″ n )’s [cls]token, and passes W∈R d×d’ Project it into the embedding space. According to formula (6), the similarities in the image triples are s ap =s(z a W, z p W), s an =s(z″ a W, z″ n W), s pn =s(z″ p W, z″ n W). Then calculate the mean squared error (MSE) between the similarity and its target:
[0084]
[0085] in, Represents similarity s ij goals, and
[0086]
[0087] According to formulas (6)-(8), Figure 4 In the example of , the calculated contrastive loss 340 can be used as a learning objective to force the query sample to be far away from the negative sample but close to the positive sample.
[0088] In S260, an image classification model is trained based on the loss.
[0089] The image classification model is trained using the loss calculated in S250 as a loss function. Figure 5 The training example of the image classification model 300 of one embodiment of the present application is shown. The training sample of the model 300 is an image triple, including a query image 302, a positive image 301 belonging to the same category as the query image 302, and a negative image 303 belonging to a different category from the query image 302.
[0090] The model 300 includes a multi-head cross-attention layer 304 and a multi-head self-attention layer 305. The multi-head cross-attention layer 304 performs cross-attention processing on the vector features extracted from the query image 302, the positive image 301, and the negative image 303. The multi-head self-attention layer 305 enhances the representation of the output of the multi-head cross-attention layer. The loss 306 is calculated based on the output of the multi-head attention layer 305, and the model 300 is trained based on the loss 306.
[0091] Figure 5 The left side of the figure also lists examples of query images, positive images, and negative images. These three images can belong to the same large category, such as large dogs. However, the query image, positive image, and negative image, although similar in appearance, belong to different breed subcategories.
[0092] In the training method of the image classification model of the present application, the training samples are image triplets, including a query image, a positive image belonging to the same category as the query image, and a negative image belonging to a different category from the query image. In addition, in the multi-head cross-attention layer, the query image is used as an anchor point to perform cross-attention operations with support images including positive and negative images; in the multi-head self-attention layer, each support image is self-attended to enhance the representation of each image. In addition, using contrast loss as a learning target, the features of the query image are made closer and closer to the features of the target support image, and the query image is made farther and farther from the features of the other support image. According to experimental results, the training method of the image classification model of the present application has a high learning discrimination ability on multiple small sample learning benchmarks, and has a high ability to distinguish similar visual concepts.
[0093] Figure 6 A schematic diagram of a Transformer block of the present application is shown, corresponding to Figure 3 310 or Figure 4 320 in . The Transformer block consists of two parts: a multi-head attention part and a FFN part. In some examples, the Transformer block is based on the visual Transformer as the backbone network. The Transformer block with gated attention can be used for conditional representation learning, and different gating units can be used to guide the feature learning process of different Transformers.
[0094] When the inputs Q, K, and V of the multi-head attention part come from different images, it corresponds to Figure 3 In the different branches MHCA 314, MHCA 315, MHCA 316 and MHCA 317, the MHCA parameters of different branches are shared. When there is a gating effect, it corresponds to MHCA / G 314 and MHCA / G 316.
[0095] In the cross-feature learning process, the competition gate is used to simulate the competition between two support images. For two support images from different categories, the query image can only share the same label with one of the two images. Therefore, during the training process, it is necessary to increase the influence of one support image on the query image and reduce the influence of the other support image on the query image at the same time. In the transformer block with cross-attention gate, the query image is used as an anchor to perform cross-attention operations with the two support images, and the two support images are used as competitors to learn discriminative features to determine which support class the query image belongs to. If the two images are from the same class, cross-attention operations are encouraged to extract common features. The gate in the cross-attention module can describe the salient object parts or feature points shared by the positive image-query image pair.
[0096] When the inputs Q, K, and V of the multi-head attention part come from the same image, it corresponds to Figure 4 Different branches MHSA324, MHSA 325 and MHSA 326 share MHSA parameters of different branches. When there is a gating effect, it corresponds to MHCA / G324 and MHCA / G326.
[0097] In the transformer block with self-attention gate, self-attention is performed on each support image separately, and the other two images are used as references to emphasize some important foreground details or background context in each image to help discover the object / scene semantics that are easily overlooked. Gating is used to reweight different patches in each image. The gate in the self-attention module can mine more foreground details and background context in each support query image pair.
[0098] In some embodiments, Figure 6 The Transformer blocks in include residual connections, which can optimize the self-attention operation. Figure 7 A schematic diagram of a competition gate of an embodiment of the present application is shown. For three input features: in and Broadcast through their [cls] token. Then, these features are projected and concatenated, expressed as:
[0099]
[0100] where X1,Y1∈R (1+n)×2d , [·,·] represents concatenation, and W1 is the MLP projection matrix. In order to combine these features, a feed-forward layer FFN1 can be used. In some examples, FFN1 includes two fully connected layers and one GELU activation layer.
[0101] X2=FFN1(X1), Y2=FNN1(Y1), M1=X2W2, M2=Y2W2, (10)
[0102] where X2,Y2∈R (1+n)×d , M1, M2∈R (1+n) M1 and M2 are the absolute similarities between the query token and the two support images.
[0103] If the negative support image is very similar to the query image, we need to further explore features that are unique only in one support image and do not appear in the other support image. We can again use relational learning to compare the relationship between the query image and the support image:
[0104] X3=FFN2(X2,Y2), Y3=FFN2(Y2,X2), M3=X3W3, M4=Y3W3 (11)
[0105] where X3,Y3∈R (1+n)×c , M3, M4∈R (1+n) Here M3 and M4 are relatively similar. Figure 7 The “C” in stands for concatenation. In some examples FFN2 includes two fully connected layers and one GELU activation layer.
[0106] Finally, the absolute similarity and relative similarity can be combined to generate the following two attention graphs G1 and G2:
[0107] G1=σ(M1⊙M3), G2=σ(M2⊙M4) (12)
[0108] Among them, ⊙ represents the multiplication between elements, σ is the sigmoid activation function, G1, G2∈R (1+n)×1 Respectively represent z p , z a and z n , z a Here, each gate function includes: (1) adjusting the weights of the query image tokens reconstructed from the support images to avoid negative knowledge transfer; (2) adjusting the attention map of the support branch to focus on the unique features it shares with the query.
[0109] During the competitive learning phase, many object parts and background contexts are ignored because they are not discriminative enough. Therefore, the reference gate can be designed to find the association between each support-query image pair.
[0110] Figure 8 A schematic diagram of a reference gate of an embodiment of the present application is shown. First, the support-query feature pair is connected and the attention matrix is calculated with another support feature to obtain a saliency score:
[0111]
[0112] In X4, the feature vectors of the query image and the negative image are concatenated, and then the attention matrix is calculated with the features of the positive image; in Y4, the feature vectors of the query image and the positive image are concatenated, and then the attention matrix is calculated with the features of the negative image. According to formula (13), for a support image, the saliency image of each patch is calculated by summarizing the similarity of each patch with the query image and another support image. The query image and another support image are concatenated together using [,].
[0113] Then, similar to the competitive gate, FFN3 is used to integrate these features to generate a saliency score:
[0114] X5=FNN3(X4)W4,Y5=FNN3(Y4)W4 (14)
[0115] Where W4∈R n×1 The reference gates G3 and G4 are calculated using the sigmoid activation function:
[0116] G3=σ(X5), G4=σ(Y5), (15)
[0117] According to formulas (13)-(15), for each support branch, a reference gate is designed to identify and highlight the informative patch, as well as integrate the discriminative information between the support query-image pairs. The obtained reference gates G3 and G4 can be directly used to reweight the tokens during the self-attention operation. The setting of the reference gate can increase the discriminative power of the [cls] token.
[0118] Reference now Fig. 9 , which shows a comparative schematic diagram of self-attention, gated criss-cross attention, and gated self-attention in image triplets. These attention maps show the relative similarity between the [cls]token and the patch token of the next layer. According to b)-c), the gated criss-cross attention scheme can extract more important foreground parts. The competition gate in the criss-cross attention block in c) helps to identify more distinctive regions. According to b) and d), the reference gate in the self-attention block helps to highlight the object parts in the support image that are more similar to the query image. The reference gate has the ability to identify shared object parts or prominent features. Therefore, the competition gate can focus on distinguishing objects / scenes, while the reference gate focuses on the scene or background context shared by the foreground object and the support-query image pair.
[0119] In some embodiments, the training sample also includes a second image triplet, which includes a second query image, a second negative image belonging to a different category from the second query image, and a third negative image belonging to a different category from the second query image, and the second image triplet is arranged in a predetermined ratio in the training sample.
[0120] The standard image triplet may only include the query image and the positive and negative images, that is, only the positive-anchor-negative (PAN) relationship in the triplet is considered. Therefore, if the negative-anchor-negative triplet (NAN) relationship is considered and the ratio of the two different image triplets is adjusted according to a predetermined ratio, the inductive bias can be avoided and the generalization ability of the model can be improved. In some examples, the performance of the model is optimal when the ratio of negative-anchor-negative triplets is 0.2.
[0121] Table 1 Ablation experiment results
[0122]
[0123] Table 1 shows the comparison results of the ablation experiment of the image classification model of the present application in the test phase. The performance of the image classification model of the present application is compared with that of the baseline HCTransformer model on the datasets CIFAR-FS and miniImageNet. The image resolution in the sample is set to 224×224. For each query sample, an N-way-k-shot method can be used, where N is 5 and k is 1. In the test, two images of the same category are not regarded as positive and negative pairs in the image triplet, so this will result in a total of nk(N-1)K image triplets for each query image, that is, 20 triplets. In the 1st case, the predictions of the 20 triplets are averaged to make the final decision.
[0124] DA in Table 1 is data augmentation, and the second image triplet can be used for data augmentation in the training sample. CA is cross attention, CG represents cross attention including competition gate, SA is self attention, and RG is self attention including reference gate.
[0125] Compared with the method of the baseline HCTransformer model (row 1), after adding CA, the performance of the method of the present application can be improved by 4.62% and 3.32% (see row 2). By further adding SA, an average improvement of 0.38% was obtained on these two datasets (see row 3). In addition, when the model includes CG and RG, the results are further enhanced by 0.36%-0.84% (see rows 4-6), indicating that conditional gating increases the representation of the model. In addition, even without data augmentation, the method of the present application can still achieve better performance.
[0126] Fig.10FIG. 1 is a schematic diagram showing a feature visualization comparison of an image classification model according to an embodiment of the present application. Fig.10 In the example of (a), 1000 images are randomly sampled from 10 randomly selected validation classes of the CIFAR-FS dataset and their features are visualized. In this example, each sampled image is treated as a query and randomly constructed with other images to form image triplets. Compared with the related art (iBOT model) in (a), the function of (b) of this application is more compact and the class boundaries are clearer.
[0127] Fig.11 The schematic diagram of the structure of an image classification model 400 of an embodiment of the present application is shown. The model 400 includes:
[0128] A feature extraction layer 410 is used to extract high-level features of an image sample, wherein the image sample is an image triplet, including a query image and two supporting images, namely a third supporting image and a fourth supporting image;
[0129] A cross-attention layer 420 is used to perform cross-attention processing on high-order features;
[0130] A self-attention layer 430 is used to enhance the representation of the output of the cross-attention layer; and
[0131] The loss calculation layer 440 is used to calculate the loss based on the output of the self-attention layer.
[0132] In some embodiments, the cross-attention layer 420 includes a first gating unit and a second gating unit, wherein the first gating unit is configured to adjust the attention matrix of the three support images and the reconstructed features of the query image, and the second gating unit is configured to adjust the attention matrix of the fourth support image and the reconstructed features of the query image.
[0133] In some embodiments, the self-attention layer 430 includes a third gating unit and a fourth gating unit, wherein the third gating unit is configured to calculate the similarity between the third support image, the query image, and the fourth support image, i.e., the similarity between the third support image Vs. (query image and fourth support image). The fourth gating unit is configured to calculate the similarity between the fourth support image, the query image, and the third support image, i.e., the similarity between the fourth support image Vs. (query image and third support image). The brackets here represent feature vector concatenation.
[0134] It can be understood that the functions of each sub-unit in model 400 correspond to the various steps in the above-mentioned model training method, which will not be repeated here.
[0135] Fig.12 An image classification method 500 of an embodiment of the present application is applied to a client. The method 500 includes:
[0136] In S510 , a test image, a first supporting image, and a second supporting image are acquired, wherein the first supporting image and the second supporting image belong to different categories.
[0137] In S520, the test image, the first supporting image and the second supporting image are input into a trained image classification model, wherein the trained image classification model is trained according to the training method of the image classification model of the present application.
[0138] In S530 , the category of the test image is output.
[0139] See also Fig.13 , Fig.13 The electronic device 600 may include one or more processors 602, a system control logic 608 connected to at least one of the processors 602, a system memory 604 connected to the system control logic 608, a non-volatile memory (NVM) 606 connected to the system control logic 608, and a network interface 610 connected to the system control logic 608.
[0140] The processor 602 may include one or more single-core or multi-core processors. The processor 602 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In the embodiments of the present invention, the processor 602 may be configured to execute the following operations: Figure 2-8 , one or more embodiments of the various embodiments shown in 12.
[0141] In some embodiments, system control logic 608 may include any suitable interface controller to provide any suitable interface to at least one of processors 602 and / or any suitable device or component in communication with system control logic 608 .
[0142] In some embodiments, the system control logic 608 may include one or more memory controllers to provide an interface to the system memory 604. The system memory 604 may be used to load and store data and / or instructions. In some embodiments, the memory 604 of the device 600 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).
[0143] NVM / memory 606 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM / memory 606 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.
[0144] NVM / storage 606 may include a portion of storage resources installed on a device of apparatus 600, or it may be accessible to the apparatus but not necessarily a part of the apparatus. For example, NVM / storage 606 may be accessed over a network via network interface 610.
[0145] In particular, system memory 604 and NVM / storage 606 may include, respectively, a temporary copy and a permanent copy of instructions 620. Instructions 620 may include instructions that, when executed by at least one of processors 602, cause device 600 to perform the following operations: Figure 2 , 12 In some embodiments, instructions 620, hardware, firmware and / or software components thereof may additionally / alternatively be placed in system control logic 608, network interface 610 and / or processor 602.
[0146] In one embodiment, at least one of the processors 602 may be packaged together with logic for one or more controllers of the system control logic 608 to form a system in package (SiP). In one embodiment, at least one of the processors 602 may be integrated on the same die with logic for one or more controllers of the system control logic 608 to form a system on chip (SoC).
[0147] The present application also provides a computing device 70. Fig.14a As shown, the computing device 70 includes: a bus 702, a processor 704, and a memory 706. The processor 704 and the memory 706 communicate with each other via the bus 702. The computing device 70 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 70.
[0148] like Fig.14a As shown, the memory 706 stores executable program codes, and the processor 704 executes the executable program codes to respectively implement the functions of the aforementioned data set generation device, thereby implementing the training method of the image classification model. That is, the memory 706 stores instructions for executing the training method of the image classification model.
[0149] Alternatively, the memory 706 stores executable codes, and the processor 704 executes the executable codes to respectively implement the functions of the aforementioned semantic matching device, thereby implementing the image classification method. That is, the memory 706 stores instructions for executing the image classification method.
[0150] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.
[0151] like Fig.14b As shown, the computing device cluster includes at least one computing device 70. The memory 706 in one or more computing devices 70 in the computing device cluster may store the same instructions for executing the training method of the image classification model.
[0152] In some possible implementations, the memory 706 of one or more computing devices 70 in the computing device cluster may also store partial instructions for executing the training method of the image classification model. In other words, the combination of one or more computing devices 70 may jointly execute the instructions for executing the training method of the image classification model.
[0153] The memory 706 in different computing devices 70 in the computing device cluster can store different instructions, each of which is used to execute part of the functions of the image classification model.
[0154] Alternatively, the computing device cluster includes at least one computing device 70. The memory 706 in one or more computing devices 70 in the computing device cluster may store the same instructions for executing the image classification method.
[0155] In some possible implementations, the memory 706 of one or more computing devices 70 in the computing device cluster may also store partial instructions for executing the image classification method. In other words, the combination of one or more computing devices 70 may jointly execute instructions for executing the image classification method.
[0156] One embodiment of the present application discloses a computer-readable medium. The computer-readable storage medium may be any available medium that a computing device can store or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute a training method for an image classification model or an image classification method.
[0157] One embodiment of the present application discloses a computer program product. The computer program product may be software or a program product that includes instructions and can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to execute a training method for an image classification model or an image classification method.
[0158] The above specific embodiments illustrate the implementation of the present application, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to the implementation. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0159] Furthermore, various operations will be described as multiple discrete operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations do not need to be performed in the order presented.
[0160] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A / B" means "A or B." The phrase "A and / or B" means "(A and B) or (A or B)."
[0161] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or by other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information through the Internet via electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.
[0162] It should be understood that although the terms "first", "second", etc. may be used here to describe various units or data, these units or data should not be limited by these terms. These terms are used only to distinguish one feature from another. For example, without departing from the scope of the exemplary embodiment, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.
[0163] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0164] While the present invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.
Claims
1. A training method for an image classification model, applied to a client, characterized in that: include, Acquire a training sample including an image triplet, the image triplet including a query image, a positive image belonging to the same category as the query image, and a negative image belonging to a different category than the query image; Extracting vector features of the query image, the positive image, and the negative image; Using a multi-head cross-attention layer to cross-attention the vector features of the query image, the positive image, and the negative image; the cross-attention layer includes a first gating unit and a second gating unit, wherein the first gating unit is configured to adjust the attention matrix of the positive image and the reconstructed features of the query image, and the second gating unit is configured to adjust the attention matrix of the negative image and the reconstructed features of the query image; enhancing the representation of the output of the cross-attention layer using a multi-head self-attention layer, the self-attention layer comprising a third gating unit and a fourth gating unit, wherein the third gating unit is configured to calculate the similarity between the positive image, the query image, and the negative image, and the fourth gating unit is configured to calculate the similarity between the negative image, the query image, and the positive image; Calculating a loss based on the output of the self-attention layer, including calculating a similarity between the outputs of the self-attention layer, and calculating a mean square error between the similarity and a target of the similarity; as well as The image classification model is trained based on the loss.
2. The method according to claim 1, characterized in that The cross-attention processing includes performing four cross-attention operations on the three branches of the vector features respectively.
3. The method according to claim 1, characterized in that: The training sample also includes a second image triplet, which includes a second query image, a second negative image belonging to a different category from the second query image, and a third negative image belonging to a different category from the second query image, and the second image triplet is arranged in a predetermined ratio in the training sample.
4. The method according to claim 3, characterized in that The predetermined ratio is 0.
2.
5. An image classification method, applied to a client, characterized in that: include, Acquire a test image, a first supporting image, and a second supporting image, wherein the first supporting image and the second supporting image belong to different categories; Inputting the test image, the first support image and the second support image into a trained image classification model, wherein the trained image classification model is trained according to any one of claims 1 to 4; and Output the category of the test image.
6. A computing device cluster, characterized in that: The method comprises at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the training method of the image classification model according to any one of claims 1 to 4; Or the image classification method as described in claim 5.
7. A computer-readable medium, characterized in that The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the training method of the image classification model described in any one of claims 1 to 4; or the image classification method described in claim 5.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the image classification model described in any one of claims 1 to 4; or the image classification method described in claim 5.
