Image model training method, image processing method, device, medium and equipment

Abstract

Embodiments of the present specification disclose an image model training method, an image processing method, an apparatus, a storage medium and equipment. The method comprises: dividing a sample image into a plurality of image patches, determining a patch vector corresponding to each image patch, and forming a patch vector sequence from the patch vectors corresponding to the plurality of image patches; performing connection processing on the patch vector sequence and a class identifier vector sequence to generate a connection vector sequence; inputting the connection vector sequence into an image feature extraction model to obtain N output class identifier vectors corresponding to N input class identifier vectors; determining a model loss of the image feature extraction model based on the output class identifier vectors and a label of the sample image, wherein the model loss comprises a class identifier constraint loss, and the class identifier constraint loss is used to learn multi-class image features of the sample image by constraining the output class identifier vectors; and adjusting parameters of the input class identifier vectors and parameters of the image feature extraction model based on the model loss.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to an image model training method, image processing method, apparatus, storage medium and device. Background Technology

[0002] With the development of machine learning technology, image recognition technology is being applied more and more widely. For example, identity recognition technology can be applied to security checks, access control, and payment.

[0003] One technical solution involves training multiple deep learning models to learn image representations. Since the image features extracted by different models differ, these differing features are integrated to represent the image. However, this approach is inefficient and costly to train multiple models.

[0004] Therefore, how to efficiently and cost-effectively acquire diverse image features has become a pressing technical challenge. Summary of the Invention

[0005] This specification provides an image model training method, image processing method, apparatus, storage medium, and device that can efficiently and cost-effectively acquire diverse image features and improve the robustness of the model.

[0006] Firstly, embodiments of this specification provide an image model training method, including:

[0007] The sample image is divided into multiple image blocks, and the block vector corresponding to each image block is determined. The block vectors corresponding to the multiple image blocks form a block vector sequence.

[0008] The block vector sequence is concatenated with the class identifier vector sequence to generate a concatenated vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The class identifier vectors are used to represent the sample image.

[0009] The connection vector sequence is input into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors;

[0010] Based on the output class identifier vector and the label of the sample image, the model loss of the image feature extraction model is determined. The model loss includes class identifier constraint loss, which is used to learn multi-class image features of the sample image by constraining the output class identifier vector.

[0011] The parameters of the input class identifier vector and the parameters of the image feature extraction model are adjusted based on the model loss.

[0012] Secondly, embodiments of this specification provide an image processing method, including:

[0013] The image to be identified is divided into multiple image blocks, and the block vector corresponding to each image block is determined. The block vectors corresponding to the image blocks in the multiple image blocks form a block vector sequence.

[0014] The block vector sequence is concatenated with the class identifier vector sequence to generate a concatenated vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The input class identifier vectors are used to learn the image features of the sample image. The N input class identifier vectors are trained using the image model training method described above.

[0015] The connection vector sequence is input into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors. The image feature extraction model is trained using the above image model training method.

[0016] The image to be identified is identified based on the N output class identifier vectors.

[0017] Thirdly, embodiments of this specification provide an image model training apparatus, comprising:

[0018] The block sequence determination module is used to divide the sample image into multiple image blocks, determine the block vector corresponding to each image block, and form a block vector sequence from the block vectors corresponding to the image blocks in the multiple image blocks.

[0019] A connection processing module is used to connect the block vector sequence with the class identifier vector sequence to generate a connection vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The class identifier vectors are used to represent the sample image.

[0020] The feature extraction module is used to input the connection vector sequence into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors;

[0021] The loss determination module is used to determine the model loss of the image feature extraction model based on the output class identifier vector and the label of the sample image. The model loss includes class identifier constraint loss, which is used to learn multi-class image features of the sample image by constraining the output class identifier vector.

[0022] The parameter adjustment module is used to adjust the parameters of the input class identifier vector and the parameters of the image feature extraction model based on the model loss.

[0023] Fourthly, this specification provides an image processing apparatus, comprising:

[0024] The block vector determination module is used to divide the image to be identified into multiple image blocks, determine the block vector corresponding to each image block, and form a block vector sequence from the multiple image blocks.

[0025] The connection processing module is used to connect the block vector sequence and the class identifier vector sequence to generate a connection vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The input class identifier vectors are used to learn the image features of the sample image. The N input class identifier vectors are trained using the image model training method described above.

[0026] The feature extraction module is used to input the connection vector sequence into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors. The image feature extraction model is trained by the above-described image model training method.

[0027] The image recognition module is used to recognize the image to be recognized based on the N output class identifier vectors.

[0028] Fifthly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method described above.

[0029] Sixthly, embodiments of this specification provide a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to perform the steps of the method described above.

[0030] In a seventh aspect, embodiments of this specification provide an electronic device, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method described above.

[0031] According to the technical solution of the embodiments of this specification, on the one hand, multiple learnable class tokens are used to learn multi-class representations of images. By constraining multiple class tokens through class token constraint loss, the image feature extraction model can autonomously perform differentiated representation learning, ensuring that the acquired image features have significant differences. On the other hand, by implicitly constraining the acquisition of different feature representations of images, diverse image features can be acquired efficiently and at low cost. Furthermore, integrating diverse image features can improve the robustness of the model. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this specification;

[0034] Figure 2 A flowchart illustrating an image model training method provided in an embodiment of this specification;

[0035] Figure 3 This is a flowchart illustrating the process of determining the class identifier constraint loss according to the embodiments provided in this specification;

[0036] Figure 4 This is a flowchart illustrating another image model training method provided according to an embodiment of this specification;

[0037] Figure 5 This is a schematic diagram of the network structure for dynamically adjusting the loss weights according to the embodiments provided in this specification;

[0038] Figure 6 A flowchart illustrating another method for determining class identifier constraint loss according to an embodiment of this specification is shown;

[0039] Figure 7 This is a schematic flowchart of an image processing method provided in an embodiment of this specification;

[0040] Figure 8 This is a schematic diagram of the structure of an image model training device provided in the embodiments of this specification;

[0041] Figure 9 This is a schematic diagram of another image processing apparatus provided in the embodiments of this specification;

[0042] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0043] To make the features and advantages of this specification more apparent and understandable, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this specification without creative effort are within the scope of protection of this specification.

[0044] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0045] Class token: It is used to learn the image features of a category, such as the image features of an image block of a category. If the class token represents a certain category and the image block represents a certain position in the image, then the learned class token can indicate where the features of a certain category may be located in the image.

[0046] Class token constraint loss: used to constrain the relationship between output class token vectors, for example, by using SDC constraint loss to make the output class token vectors orthogonal to each other.

[0047] Class label pair loss weights: These are used to identify the relative importance of two output class label vectors. For example, the larger the distance between these two output class label vectors, such as the cosine distance, the larger the class label pair loss weights corresponding to these two output class label vectors will be; conversely, the smaller the distance between these two output class label vectors, the smaller the class label pair loss weights corresponding to these two output class label vectors will be.

[0048] Classification loss: It is used to constrain the relationship between the output class token vector and the label (class) of the sample image, so that there is a strong correlation between each output class token vector and the class of the sample image.

[0049] One approach involves extracting image features from local regions of an image using a Transformer model and then integrating these local features from different regions. Another approach integrates features from different layers of a deep learning model to fuse features of varying granularity. These approaches share the common goal of extracting and integrating multiple distinct image features to obtain a more robust representation of the image. However, these approaches, which focus only on local regions or features of varying granularity, fail to capture the truly necessary representation of the image.

[0050] Based on the above, embodiments of this specification provide an image model training method. This method simultaneously represents an image using multiple class tokens. Implicit constraints, such as SDC (Self-Diverse Constraint Loss), are applied to the multiple image features extracted by the image feature extraction model (i.e., the output class tokens). This ensures that the multiple image features are orthogonal to each other, enabling the image feature extraction model to autonomously learn differentiated representations. This allows for the efficient and low-cost acquisition of diverse image features, improving the model's robustness. Therefore, according to the technical solution of this specification, different image representations from the same target task not only contain the same identity information but also contain differentiated features from different aspects, such as coarse / fine granularity, global / local regions, and other unknown complementary features, thereby significantly improving the model's robustness.

[0051] The technical solutions of the embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0052] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this specification.

[0053] like Figure 1 As shown, image 105 is a sample image in an image recognition task, which can be a ReID (Re-Identification) task, an image classification task, or an image retrieval task, etc. If it's for an image model training application scenario, the image recognition model includes a class label vector sequence 120, a transformer model 130, and a class label constraint layer 140. If it's for an image recognition application scenario, the image recognition model includes a class label vector sequence 120, a transformer model 130, and an image recognition task layer 150.

[0054] Taking an image model training application scenario as an example, image 105 is a sample image 105. The sample image 105 is divided into multiple image blocks 1, and the block vector 110 corresponding to each image block is determined. The block vector 110 forms a block vector sequence. The block vector sequence is concatenated with the class identifier vector sequence 120 to generate a connection vector sequence. The class identifier vector sequence 120 includes N input class identifier vectors 122, i.e., class tokens. The connection vector sequence is input into the transformer model 130 to obtain N output class identifier vectors 135 corresponding to the N input class identifier vectors 122. Based on the output class identifier vectors 135 and the labels of the sample image 105, the model loss of the image feature extraction model is determined through the class identifier constraint layer 140. The model loss includes the class identifier constraint loss, which is used to learn the diverse image features of the sample image 105 by constraining the output class identifier vectors 135. The parameters of the input class identifier vectors 122 and the parameters of the transformer model 130 are adjusted based on the model loss.

[0055] Taking an image recognition scenario as an example, image 105 is the image to be recognized. The image to be recognized is divided into multiple image blocks, and the block vector 110 corresponding to each image block is determined. The block vectors 110 corresponding to the image blocks in the multiple image blocks form a block vector sequence. The block vector sequence is connected with the class identifier vector sequence 120 to generate a connection vector sequence. The class identifier vector sequence 120 includes N input class identifier vectors 122, where N is a positive integer greater than or equal to 2. The N input class identifier vectors are trained using the image model training method described above. The connection vector sequence is input into the transformer model 130 to obtain N output class identifier vectors 135 corresponding to the N input class identifier vectors 122. The transformer model 130 is trained using the image model training method described above. Based on the N output class identifier vectors, the image to be recognized is recognized by the image recognition task layer 150.

[0056] It should be noted that the steps in the image processing method in the example embodiments of this specification may be partially executed by the client, partially executed by the server, or entirely executed by the server or entirely by the client. This specification does not impose any special limitations on this.

[0057] based on Figure 1 The model structure shown below will be combined with... Figures 2-5 This specification provides a detailed description of the image model training method provided in the embodiments. It should be noted that the model structure described above is only shown to facilitate understanding of the spirit and principles of this specification, and the embodiments are not limited in any way. Rather, the embodiments can be applied to any applicable scenario.

[0058] Figure 2This is a flowchart illustrating an image model training method provided in an embodiment of this specification. This image model training method can be executed by a device with computing capabilities, such as a terminal device or a server. Figure 2 As shown, the image training method in the embodiments of this specification may include the following steps S210 to S250.

[0059] In step S210, the sample image is divided into multiple image blocks, and the block vector corresponding to each image block is determined. The block vectors corresponding to the image blocks in the multiple image blocks form a block vector sequence.

[0060] In the example embodiment, the sample images are sample images in an image recognition task. The image recognition task can be a ReID task, an image classification task, or an image retrieval task, etc. Taking an image classification task as an example, multiple sample images corresponding to the image classification task are obtained by pre-labeling multiple images with classification tags. Based on the size of the sample images and the task processing requirements, the sample images are divided into multiple image blocks. For example, if the size of the sample image is 224*224 and the size of the image block is 16*16, the sample image is divided into 196 image blocks.

[0061] Furthermore, the pixel matrix corresponding to each image block is linearly projected to obtain the block vector corresponding to each image block. For example, if the size of the image block is 16*16, the pixel matrix corresponding to the image block is linearly projected to obtain a block vector with a dimension of 256.

[0062] In step S220, the block vector sequence and the class identifier vector sequence are concatenated to generate a concatenated vector sequence, which includes N input class identifier vectors.

[0063] In the example embodiment, the class token vector sequence includes N input class token vectors, where N is a positive integer greater than or equal to 2. The dimension of the input class token vectors is the same as the dimension of the image block vectors. Each input class token vector can be used to learn image features of a category, such as image features of an image block of a category. N can be the number of types of image features, and its size can be determined according to the image recognition task. The block vector sequence and the class token vector sequence are concatenated using a concatenation function to generate a concatenation vector sequence, which includes the block vectors and the input class token vectors.

[0064] For example, if the dimension of the block vector is 256, then the dimension of the input class identifier vector is also 256. The concatenation function is the concat function. The concat function concatenates the N input class identifier vector block vector sequences in the class identifier vector sequence to obtain the concatenation vector sequence. The concatenation vector sequence includes the block vectors of multiple image blocks and the N input class identifier vectors.

[0065] In step S230, the connection vector sequence is input into the image feature extraction model to obtain N output class identifier vectors corresponding to N input class identifier vectors.

[0066] In the example embodiment, the image recognition task can be a ReID task, an image classification task, or an image retrieval task, etc. The image feature extraction model can be a Transformer model, which inputs the sequence of connection vectors into the Transformer model and obtains N output class label vectors corresponding to N input class label vectors.

[0067] For example, a Transformer model includes at least one Transformer encoder, each Transformer encoder including multiple encoding layers, which are used to extract block vectors of image blocks and features of input class identifier vectors from the sequence of connection vectors, and obtain N output class identifier vectors corresponding to N input class identifier vectors from the extracted features.

[0068] It should be noted that although the Transformer is used as an example to illustrate the image feature extraction model, those skilled in the art should understand that image features can also be other suitable models, such as other self-attention models, which are also within the scope of the embodiments in this specification.

[0069] In step S240, the model loss of the image feature extraction model is determined based on the output class identifier vector and the label of the sample image. The model loss includes the class identifier constraint loss.

[0070] In the example embodiment, the model loss includes class label constraint loss and classification loss. The class label constraint loss is used to learn multi-class image features of the sample image by constraining the output class label vector. For example, the class label constraint loss is used to make the output class label vectors mutually orthogonal, so that the multiple output class label vectors are distributed in different feature spaces, thereby enabling different class label vectors to learn image features of different classes.

[0071] It should be noted that although the example of orthogonality between output class label vectors is used to illustrate the class label constraint loss, those skilled in the art should understand that the class label constraint loss can also be other appropriate class label constraint losses, as long as they can make the output class label vectors have sufficient differences so that different class label vectors can learn image features of different classes.

[0072] Furthermore, in the example embodiment, the classification loss is used to determine the classification loss by constraining the relationship between the output class token vector and the label category of the sample image, so that there is a strong correlation between each output class token vector, i.e., the output classtoken, and the label of the sample image.

[0073] For example, in the example embodiment, for the class labeling constraint loss, the distance between every two output class labeling vectors in the N output class labeling vectors is determined; based on the distance between every two output class labeling vectors, the class labeling constraint loss of the image feature extraction model is determined. For the classification loss, the cross-entropy between the output class labeling vectors and the labels of the sample images is determined; based on the cross-entropy between the output class labeling vectors and the labels of the sample images, the classification loss of the image feature extraction model is determined.

[0074] It should be noted that although the classification loss is described using cross-entropy loss, those skilled in the art should understand that the classification loss can also be other appropriate loss functions, such as the KL divergence loss function, which is also within the scope of the embodiments in this specification.

[0075] In step S250, the parameters of the input class identifier vector and the parameters of the image feature extraction model are adjusted based on the model loss.

[0076] In the example embodiment, the classification loss and class labeling constraint loss of the image feature extraction model are determined, and the parameters of the input class labeling vector and the parameters of the image feature extraction model are adjusted by backpropagation based on the classification loss and class labeling constraint loss.

[0077] For example, the sum of the classification loss and class label constraint loss of the image feature extraction model is determined, with the goal of the sum of the classification loss and class label constraint loss being less than a predetermined threshold. The minimum value of the classification loss and class label constraint loss is determined by methods such as gradient descent. The parameter adjustment value of the next layer network is determined based on the minimum value of the classification loss and class label constraint loss. The parameters of the input class label vector and the parameters of the image feature extraction model are adjusted based on the parameter adjustment value.

[0078] according to Figure 2The technical solution in the example embodiment, on the one hand, uses multiple learnable class tokens to learn multi-class representations of images. By constraining multiple class tokens through class token constraint loss, the image feature extraction model can autonomously learn differentiated representations, ensuring that the acquired image features have significant differences. On the other hand, by implicitly constraining the acquisition of different feature representations of images, it is possible to efficiently and cost-effectively acquire diverse image features. Furthermore, integrating diverse image features can improve the robustness of the model.

[0079] Furthermore, in the example embodiment, before inputting the connection vector sequence into the image feature extraction model, the image model training method further includes: obtaining the position embedding vector corresponding to the position information of each image block in multiple image blocks; adding the position embedding vector to the corresponding connection vector in the connection vector sequence to obtain the added connection vector sequence.

[0080] According to the technical method in the above example embodiments, by embedding location information into the connection vector, the location of image blocks can be represented more accurately, thereby enabling efficient and accurate training of the model and improving the robustness of the model.

[0081] Further, in the example embodiment, the query vector (QUERY), key vector (KEY), and value vector (VALUE) corresponding to each connection vector in the connection vector sequence are determined; the query vector and key vector corresponding to each connection vector are subjected to dot product processing to obtain the attention value between the query vector and the key vector; the attention value is used as a weight to perform a weighted operation on the value vector to obtain the output class identifier vector. The output class identifier vector represents the image features of the corresponding image block, that is, it represents the interaction between the class token and each image block.

[0082] According to the technical solution in the above example embodiment, by learning the interaction between different class identifier vectors (i.e., class tokens) and each image block, the image features of a specific class of image blocks can be obtained.

[0083] Figure 3 A schematic flowchart illustrating the process of determining class identifier constraint loss according to an embodiment of this specification is shown.

[0084] Reference Figure 3 As shown, in step S310, the distance between every two output class identifier vectors in the N output class identifier vectors is determined.

[0085] In the example embodiment, two input class identifier vectors are randomly selected from N output class identifier vectors, and the distance between each pair of output class identifier vectors is determined. For example, assuming the distance between two output class identifier vectors is the Euclidean distance, the Euclidean distance between each pair of output class identifier vectors in the N output class identifier vectors is determined. Refer to the following equation (1):

[0086] ν ij =|cos(f i ,f j )|,i <j,i,j=1,...,N (1)

[0087] Where, ν ij f is the distance between two class identifier vectors. i f j It is any two output class identifier vectors, i.e., class tokens, out of N output class identifier vectors.

[0088] It should be noted that although Euclidean distance has been used as an example, those skilled in the art should understand that the distance between two output class identifier vectors can also be other appropriate distances, such as Hamming distance or Manhattan distance, which is also within the scope of the embodiments in this specification.

[0089] In step S320, the class labeling constraint loss of the image feature extraction model is determined based on the distance between every two output class labeling vectors.

[0090] In the example embodiment, the distance between every two output class identifier vectors in the N output class identifier vectors is summed to obtain the distance sum; the distance sum is averaged, and the class identifier constraint loss of the image feature extraction model is determined based on the result of the averaging process.

[0091] For example, the distances between any two output class identifier vectors in the N output class identifier vectors are summed to obtain the distance sum; the number of combinations of two output class identifier vectors selected from the N output class identifier vectors is determined; the class identifier constraint loss is obtained by averaging the distance sum based on the number of combinations. See equation (2) below:

[0092]

[0093] Among them, L sdc For class identification constraint loss, ν ij The distance between two class identifier vectors is N, where N is the number of class identifier vectors. It represents the number of combinations of two output class identifier vectors selected from N output class identifier vectors.

[0094] according to Figure 3The technical solution in the example embodiment, on the one hand, uses SDC to constrain the output class tokens to ensure that the extracted image features are sufficiently diverse, which helps the model to automatically learn diverse representation spaces; on the other hand, integrating these constrained class tokens can improve the robustness and perturbation invariance of the model, and achieve better performance improvement in downstream tasks such as ReID tasks, image retrieval tasks and image classification tasks.

[0095] Figure 4 A schematic flowchart of another image model training method provided according to an embodiment of this specification is shown.

[0096] Reference Figure 4 As shown, the image model training method includes the following steps (1) to (8):

[0097] (1) Divide the sample image into multiple image blocks.

[0098] In the example embodiment, the size of the image block is determined according to the image recognition task and computing resource requirements, and the sample image is divided into H×W image blocks according to the size of the sample image and the size of the image block.

[0099] Reference Figure 4 As shown, if the size of image block 410 is 16*16 and the size of sample image 405 is 32*48, then the sample image is divided into 2*3, or 6 image blocks 410.

[0100] (2) Each image block is mapped to an image block vector of dimension D through a trainable linear projection.

[0101] In an example embodiment, a trainable linear projection layer is used to linearly map the pixel matrix of the image blocks to generate an image block vector of dimension D.

[0102] Reference Figure 4 As shown, the pixel matrix of the image block is linearly projected through a trainable linear projection layer 420 to obtain the block vector corresponding to the image block. The output of the linear projection layer 420 can be represented as a block embedding vector sequence P. C ×D Where C is the number of image blocks and D is the dimension of the block vector of the image blocks.

[0103] (3) Connect multiple learnable class identifier vectors with dimension D into a class identifier vector sequence.

[0104] In the example embodiment, multiple learnable embeddings of dimension D are concatenated into a sequence, represented as the input class token vector sequence, i.e., class tokens f.N×D , where N is the number of input class tokens, and the size of N can be determined according to the image recognition task.

[0105] Reference Figure 4 As shown, N is 5, and the 5 input class tokens are concatenated into a class token sequence 425.

[0106] (4) Connect the class identifier vector sequence with the block vector sequence to obtain the connection vector sequence.

[0107] In the example embodiment, the class tokens sequence and the block vector sequence are concatenated using the concat function to obtain a concatenated vector sequence, X. (N+C)×D =[f N×D ;P C×D ].

[0108] (5) Add learnable position embedding vectors to the connection vector sequence.

[0109] In the example embodiment, the position embedding vector corresponding to the position information of each image block in the multiple image blocks is obtained; the position embedding vector is added to the corresponding connection vector in the connection vector sequence to obtain the added connection vector sequence.

[0110] Reference Figure 4 As shown, the position embedding vector 430 is added to the corresponding connection vector of the connection vector sequence. For example, position embedding vectors 0-4 are added to the input class identifier vector, and position embedding vectors 5-9 are added to the block vector of the image block.

[0111] (6) Input the connection vector with added position embedding vector into the image feature extraction model to obtain multiple output class identifier vectors corresponding to the input class identifier vector.

[0112] In the example implementation, after adding learnable position embedding vectors to the connection vector sequence, the connection vector sequence is input into the Transformer model to obtain multiple output class token vectors, i.e., multiple output class tokens.

[0113] Reference Figure 4 As shown, the Transformer model includes multiple Transformer layers 435, through which multiple output class identifier vectors 440 are obtained.

[0114] (7) Determine the class label constraint loss of multiple output class label vectors and the classification loss of the model.

[0115] In the example embodiment, the relationship between the output class tokens is constrained by SDC (Self-Diverse Constraint Loss), i.e., class token constraint loss. The SDC constraint loss ensures that the output class tokens are orthogonal to each other. The class token constraint loss of multiple output class tokens is determined by the above equation (2).

[0116] Furthermore, the multiple output class label vectors are constrained by the class label constraint loss, so that the multiple output class label vectors are distributed in different feature spaces, referring to... Figure 4 As shown, let features fa, fb, and fc be output class identifier vectors. Through the class identifier constraint layer 445, features fa, fb, and fc that are in the same original feature space are distributed to different feature spaces. For example, feature fa is distributed in feature space A, feature fb is distributed in feature space B, and feature fc is distributed in feature space C.

[0117] Using SDC to constrain class label vectors ensures that the extracted image features are sufficiently diverse, which helps the model automatically learn diverse representation spaces.

[0118] Furthermore, in the example embodiment, the model's classification loss is used to represent the relationship between the output class label vector and the image category. The cross-entropy between the output class label vector and the label of the sample image is determined; based on the cross-entropy between the output class label vector and the label of the sample image, the classification loss of the image feature extraction model is determined.

[0119] (8) Based on classification loss and class label constraint loss, the parameters of the input class label vector and the parameters of the image feature extraction model are adjusted by backpropagation.

[0120] In the example embodiment, the sum of the classification loss and class label constraint loss of the image feature extraction model is determined. With the goal of the sum of the classification loss and class label constraint loss being less than a predetermined threshold, the minimum value of the classification loss and class label constraint loss is determined by methods such as gradient descent. The parameter adjustment value of the next layer network is determined based on the minimum value of the classification loss and class label constraint loss. The parameters of the input class label vector and the parameters of the image feature extraction model are adjusted based on the parameter adjustment value.

[0121] according to Figure 4The technical solution in the example embodiment, on the one hand, treats the class token as an information collector, receiving the embedding vectors of image blocks and summarizing them to obtain the image's feature representation. Multiple class tokens are added to the ViT (VisionTransformer) structure, which is equivalent to representing the image simultaneously through multiple information collectors. The introduced SDC constrains these class tokens to ensure that the final features have significant differences. On the other hand, compared to multi-model ensemble methods, the technical solution in the above embodiment operates in parallel on multiple class tokens, and only introduces a small number of parameters, without significantly increasing the time consumption. Furthermore, compared to integrating local features or features of different granularities, the technical solution in the above embodiment does not impose explicit constraints to obtain different feature representations, but rather allows the model to autonomously learn different feature representations through implicit constraints. This method yields more feature differences, and the integrated features are more robust.

[0122] Furthermore, to obtain more compact and robust features, more class labels are needed to acquire more diverse feature spaces. However, since each pair of class label vectors needs to be orthogonal, the class label constraint loss becomes difficult to optimize as the number of class labels increases. For example, some class label vector pairs have low cosine similarity, while others have high cosine similarity. In other words, class label vector pairs with high cosine similarity are not well optimized during training and cannot learn the desired number of feature spaces. This problem arises because the randomness during training causes the distance between some class label vector pairs to easily decrease, while the distance between others easily increases. Therefore, it is necessary to dynamically change the relative importance of each pair of output class label vectors during training.

[0123] Based on the above, in some example embodiments, a corresponding class label pair loss weight is set for every two output class label vectors. This class label pair loss weight is used to identify the relative importance of the two output class label vectors. For example, if the distance between the two output class label vectors, such as the cosine distance, is larger, then the value of the corresponding class label pair loss weight is larger; conversely, if the distance between the two output class label vectors is smaller, then the value of the corresponding class label pair loss weight is smaller. The class label pair loss weight for every two output class label vectors is dynamically changed during training.

[0124] For example, refer to Figure 5As shown, w1, w2, ..., wk are the loss weights of the class label pairs of two output class label vectors. The dynamic weight controller 510 dynamically changes the loss weights of the class label pairs of every two output class label vectors, i.e., w1, w2, ..., wk, during the training process.

[0125] according to Figure 5 The technical solution in the example embodiment sets corresponding class label pair loss weights for every two output class label vectors, which can dynamically change the relative importance of every two output class label vectors during training, so that every two output class label vectors can be orthogonal to each other, ensuring that the final obtained features have greater differences, and thus learning more feature space.

[0126] Figure 6 A flowchart illustrating another determination of class identifier constraint loss according to an embodiment of this specification is shown.

[0127] Reference Figure 6 As shown, in step S610, the loss weight of the class identifier pair corresponding to each pair of output class identifier vectors is determined based on the distance between each pair of output class identifier vectors.

[0128] In the example embodiment, two input class identifier vectors are randomly selected from N output class identifier vectors, the distance between each pair of output class identifier vectors is determined, and the loss weight of the class identifier pair corresponding to each pair of output class identifier vectors is determined based on the distance.

[0129] For example, let the distance between two output class label vectors be the Euclidean distance. Determine the Euclidean distance between every two output class label vectors among the N output class label vectors. If the value of the Euclidean distance between these two output class label vectors is larger, then the value of the loss weight of the class label pair corresponding to these two output class label vectors is larger; if the value of the Euclidean distance between these two output class label vectors is smaller, then the value of the loss weight of the class label pair corresponding to these two output class label vectors is smaller.

[0130] Furthermore, in some example embodiments, the normalized exponential function value corresponding to each pair of output class identifier vectors is determined based on the distance between each pair of output class identifier vectors; and the loss weight of the class identifier pair corresponding to each pair of output class identifier vectors is determined based on the normalized exponential function value. In some example embodiments, refer to the following equation (3):

[0131]

[0132] Among them, v ij v mn w is the distance between two class identifier vectors. ij For v ij The corresponding class identifier corresponds to the loss weight.

[0133] In step S620, based on the class identifier pair loss weight, the distance between every two output class identifier vectors in the N output class identifier vectors is weighted and summed to determine the class identifier constraint loss of the image feature extraction model.

[0134] In an example embodiment, based on the loss weights of the class identifier pairs corresponding to every two output class identifier vectors, a weighted sum is performed on the distances between every two output class identifier vectors in the N output class identifier vectors. Based on the result of the weighted summation, the class identifier constraint loss of the image feature extraction model is determined. In some example embodiments, refer to the following equation (4):

[0135] L SDC =∑ i ∑ j w ij v ij i <j,i,j=1,...,N (4)

[0136] Among them, L SDC For class identification constraint loss, v ij v mn w is the distance between two class identifier vectors. ij For v ij The corresponding class identifier corresponds to the loss weight.

[0137] according to Figure 6 The technical solution in the example embodiment sets corresponding class label pair loss weights for every two output class label vectors, and performs a weighted summation of the distance between every two output class label vectors based on the class label pair loss weights to determine the class label constraint loss of the image feature extraction model. This not only dynamically changes the relative importance of every two output class label vectors during training, but also better optimizes the class label constraint loss, so that every two output class label vectors can be orthogonal to each other, ensuring that the final features have greater differences, and thus learning more feature space.

[0138] Figure 7 A schematic flowchart of an image processing method according to an embodiment of this specification is shown. This image processing method can be executed by a device with computing capabilities, such as a terminal device. Figure 7 As shown, the image processing method in the embodiments of this specification may include the following steps S710 to S750.

[0139] Reference Figure 7 As shown, in step S710, the image to be identified is divided into multiple image blocks, and the block vector corresponding to each image block is determined. The block vectors corresponding to the image blocks in the multiple image blocks form a block vector sequence.

[0140] In the example embodiment, the image recognition task can be a ReID task, an image classification task, or an image retrieval task, etc. The image to be recognized is the image corresponding to the image recognition task. For example, if the image recognition task is a pedestrian re-identification (ReID) task, the image to be recognized is an image of pedestrians on the road obtained through a camera.

[0141] In step S720, the block vector sequence and the class identifier vector sequence are concatenated to generate a concatenated vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2.

[0142] In the example embodiment, N input class token vectors are obtained through the image model training method described above. For example, assuming the image recognition task is a pedestrian ReID task, the image feature extraction model is trained using the image model training method described above on sample images of multiple pedestrians on roads, resulting in N input class token vectors. The input class token vector is used to extract image features from image blocks of a certain category. The class token represents a category, and the image block represents a location in the image. The learned input class token vector represents the probability of a feature of a certain category at a certain location in the image.

[0143] Furthermore, in the example embodiment, a concatenation function is used to connect the block vector sequence and the class identifier vector sequence to generate a concatenation vector sequence, which includes block vectors and input class identifier vectors. For example, if the dimension of the block vectors is 256, then the dimension of the input class identifier vectors is also 256. The concatenation function is the `concat` function, which concatenates the N input class identifier vector block vector sequences in the class identifier vector sequence to obtain the concatenation vector sequence. The concatenation vector sequence includes block vectors from multiple image blocks and N input class identifier vectors.

[0144] In step S730, the connection vector sequence is input into the image feature extraction model to obtain N output class identifier vectors corresponding to N input class identifier vectors.

[0145] In the example embodiment, the image feature extraction model is trained using the image model training method described above. The image feature extraction model is a Transformer model, which includes at least one Transformer encoder. Each Transformer encoder includes multiple encoding layers. The encoding layers are used to extract features of image block vectors and input class identifier vectors from the connection vector sequence, and to obtain N output class identifier vectors corresponding to N input class identifier vectors from the extracted features.

[0146] In step S740, the image to be identified is identified based on N output class identifier vectors.

[0147] In the example embodiment, the output class token represents the probability that a feature of a certain category appears at a certain location in the image. The N output class tokens are subjected to average pooling, and then the image to be recognized is processed by the image recognition task layer. The image recognition task layer can be a ReID task, an image classification task, or an image retrieval task, etc.

[0148] For example, refer to Figure 7 As shown, the image recognition task is set as the ReID task, and the image to be recognized is a pedestrian image. The N output class identifier vectors extracted from the pedestrian image are subjected to average pooling through the image recognition task layer 450. The N output class identifier vectors are integrated, and the similarity between the integrated class identifier vector and the feature vector of the target person in the database is calculated. Pedestrian re-identification is performed based on the calculated similarity.

[0149] according to Figure 7 The technical solution in the example embodiment, on the one hand, can ensure that the extracted image features are sufficiently diverse by performing multiple types of image representations on the image through multiple class tokens; on the other hand, integrating the image representations of multiple extracted class tokens can improve the robustness and perturbation invariance of the model, and can achieve better performance improvement in downstream tasks such as re-identification tasks, image retrieval tasks and image classification tasks.

[0150] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] Below, we will combine Figure 8 as well as Figure 1 The system architecture shown in this specification provides a detailed description of the image model training device provided in the embodiments. It should be noted that... Figure 8 The image model training device described herein is used to execute the instructions. Figures 2-7 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 2-5 The example shown.

[0152] Please see Figure 8 This is a schematic diagram of the structure of an image model training device provided in an embodiment of this specification. Figure 8 As shown, the image model training apparatus 800 of this embodiment may include: a block sequence determination module 810, a connection processing module 820, a feature extraction module 830, a loss determination module 840, and a parameter adjustment module 850. Among them,

[0153] The block sequence determination module 810 is used to divide the sample image into multiple image blocks, determine the block vector corresponding to each image block, and form a block vector sequence from the block vectors corresponding to the image blocks in the multiple image blocks.

[0154] The connection processing module 820 is used to connect the block vector sequence with the class identifier vector sequence to generate a connection vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2.

[0155] Feature extraction module 830 is used to input the connection vector sequence into the image feature extraction model to obtain N output class identifier vectors corresponding to N input class identifier vectors;

[0156] The loss determination module 840 is used to determine the model loss of the image feature extraction model based on the output class identifier vector and the label of the sample image. The model loss includes class identifier constraint loss, which is used to learn multi-class image features of the sample image by constraining the output class identifier vector.

[0157] The parameter adjustment module 850 is used to adjust the parameters of the input class identifier vector and the parameters of the image feature extraction model based on the model loss.

[0158] In some example embodiments, based on the above scheme, the loss determination module 840 includes:

[0159] A distance determination unit is used to determine the distance between any two output class identifier vectors in the N output class identifier vectors;

[0160] A class identifier constraint loss determination unit is used to determine the class identifier constraint loss of the image feature extraction model based on the distance between every two output class identifier vectors, wherein the class identifier constraint loss is used to make the output class identifier vectors mutually orthogonal.

[0161] In some example embodiments, based on the above scheme, the class identifier constraint loss determination unit is configured as follows:

[0162] The distances between any two output class identifier vectors in the N output class identifier vectors are summed to obtain the distance sum;

[0163] Determine the number of combinations of taking two output class identifier vectors from the N output class identifier vectors;

[0164] The class identification constraint loss is obtained by averaging the distances based on the number of combinations.

[0165] In some example embodiments, based on the above scheme, the class identification constraint loss determination unit includes:

[0166] The weight determination unit is used to determine the loss weight of the class identifier pair corresponding to each pair of output class identifier vectors based on the distance between each pair of output class identifier vectors;

[0167] The weighted summation unit is used to perform a weighted summation of the distances between every two output class identifier vectors in the N output class identifier vectors based on the class identifier pair loss weights, thereby determining the class identifier constraint loss of the image feature extraction model.

[0168] In some example embodiments, based on the above scheme, the weight determination unit is configured as follows:

[0169] Based on the distance between every two output class identifier vectors, determine the normalized exponential function value corresponding to every two output class identifier vectors;

[0170] Based on the normalized exponential function value, the loss weights for each pair of output class identifier vectors are determined.

[0171] In some example embodiments, based on the above scheme, the model loss further includes a classification loss, and the loss determination module 840 is configured as follows:

[0172] Determine the cross-entropy between the output class identifier vector and the label of the sample image;

[0173] The classification loss of the image feature extraction model is determined based on the cross-entropy.

[0174] In some example embodiments, based on the above scheme, the image model training device 800 further includes:

[0175] The location vector acquisition module is used to acquire a location embedding vector corresponding to the location information of each of the multiple image blocks before inputting the connection vector sequence into the image feature extraction model.

[0176] The position embedding module is used to add the position embedding vector to the corresponding connection vector in the connection vector sequence to obtain the connection vector sequence after addition.

[0177] In some example embodiments, the block sequence determination module 810 is configured to:

[0178] Linear projection is performed on the pixel matrix corresponding to each image block to obtain the block vector corresponding to each image block;

[0179] The block vector sequence is formed based on the block vector corresponding to each of the image blocks.

[0180] In some example embodiments, the feature extraction module 830 is configured to:

[0181] Determine the query vector, key vector, and value vector corresponding to each join vector in the join vector sequence;

[0182] Perform a dot product between the query vector and the key vector corresponding to each of the connection vectors to obtain the attention value between the query vector and the key vector;

[0183] The attention value is used as a weight to perform a weighted operation on the value vector to obtain the output class identifier vector, which represents the image features of a corresponding image block.

[0184] In some example embodiments, based on the above scheme, the image feature extraction model is a Transformer model.

[0185] According to this instruction manual Figure 8 The technical solution of the embodiment, on the one hand, uses multiple learnable class tokens to learn multi-class representations of images. By constraining multiple class tokens through class token constraint loss, the image feature extraction model can autonomously perform differentiated representation learning, ensuring that the acquired image features have significant differences. On the other hand, by implicitly constraining the acquisition of different feature expressions of images, it is possible to efficiently and cost-effectively acquire diverse image features, and the robustness of the model can be improved by integrating diverse feature characteristics.

[0186] The above is an illustrative scheme of an image model training device according to an embodiment of this specification. It should be noted that the technical solution of this image model training device and the technical solution of the image model training method described above belong to the same concept. Details not described in detail in the technical solution of the image model training device can be found in the description of the technical solution of the image model training method described above.

[0187] Please see Figure 9 , Figure 9 This is a schematic diagram of another image processing apparatus provided in an embodiment of this specification. Figure 9 As shown, the image processing apparatus 900 in the embodiments of this specification may include: a block sequence determination module 910, a connection processing module 920, a feature extraction module 930, and an image recognition module 940.

[0188] in,

[0189] The block vector determination module 910 is used to divide the image to be identified into multiple image blocks, determine the block vector corresponding to each image block, and form a block vector sequence from the multiple image blocks.

[0190] The connection processing module 920 is used to connect the block vector sequence and the class identifier vector sequence to generate a connection vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The N input class identifier vectors are obtained by training using the above-mentioned image model training method.

[0191] Feature extraction module 930 is used to input the connection vector sequence into the image feature extraction model to obtain N output class identifier vectors corresponding to N input class identifier vectors. The image feature extraction model is trained by the above image model training method.

[0192] The image recognition module 940 is used to recognize the image to be recognized based on the N output class identifier vectors.

[0193] The above is an illustrative embodiment of an image processing apparatus according to this specification. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. Details not described in detail in the technical solution of the image processing apparatus can be found in the description of the technical solution of the image processing method described above.

[0194] This specification also provides a computer storage medium that can store multiple program instructions adapted to be loaded and executed by a processor as described above. Figures 2-7 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figures 2-7 The specific details of the illustrated embodiments will not be elaborated here.

[0195] This specification also provides a computer program product that stores at least one instruction, which is loaded and executed by a processor as described above. Figures 2-7 The image processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 2-7The specific details of the illustrated embodiments will not be elaborated here.

[0196] Please refer to Figure 10 This diagram illustrates the structure of an electronic device provided in an exemplary embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 1010, a memory 1020, an input device 1030, an output device 1040, and a bus 1050. The processor 1010, memory 1020, input device 1030, and output device 1040 may be connected via the bus 1050.

[0197] Processor 1010 may include one or more processing cores. Processor 1010 connects to various parts of the electronic device through various interfaces and lines, and performs various functions and processes data of electronic device 1000 by running or executing instructions, programs, code sets, or instruction sets stored in memory 1020, and by calling data stored in memory 1020. Optionally, processor 1010 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 1010 may integrate one or more of a central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU mainly handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem is used for wireless communication. It is understood that the modem may also not be integrated into processor 1010 and may be implemented separately through a communication chip.

[0198] The memory 1020 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1020 may include non-transitory computer-readable storage medium. The memory 1020 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (e.g., touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be an Android system, including systems deeply developed based on the Android system, an iOS system, including systems deeply developed based on the iOS system, or other systems.

[0199] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0200] The input device 1030 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 1040 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 1030 and the output device 1040 can be combined, and both the input device 1030 and the output device 1040 can be a touch display screen.

[0201] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, Wireless Fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0202] exist Figure 10 In the illustrated electronic device, the processor 1010 can be used to call the image model training application stored in the memory 1020, and specifically perform the following operations:

[0203] The sample image is divided into multiple image blocks, and the block vector corresponding to each image block is determined. The block vectors corresponding to the multiple image blocks form a block vector sequence.

[0204] The block vector sequence is concatenated with the class identifier vector sequence to generate a concatenated vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The class identifier vectors are used to represent the sample image.

[0205] The connection vector sequence is input into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors;

[0206] Based on the output class identifier vector and the label of the sample image, the model loss of the image feature extraction model is determined. The model loss includes class identifier constraint loss, which is used to learn multi-class image features of the sample image by constraining the output class identifier vector.

[0207] The parameters of the input class identifier vector and the parameters of the image feature extraction model are adjusted based on the model loss.

[0208] In some example embodiments, when the processor 1010 executes the class labeling constraint loss for determining the image feature extraction model, it specifically performs the following operations:

[0209] Determine the distance between any two output class identifier vectors among the N output class identifier vectors;

[0210] The class labeling constraint loss of the image feature extraction model is determined based on the distance between any two output class labeling vectors, wherein the class labeling constraint loss is used to make the output class labeling vectors mutually orthogonal.

[0211] In some example embodiments, when the processor 1010 performs the operation of determining the class identifier constraint loss based on the distance between every two output class identifier vectors, it specifically performs the following operations:

[0212] The distances between any two output class identifier vectors in the N output class identifier vectors are summed to obtain the distance sum;

[0213] Determine the number of combinations of taking two output class identifier vectors from the N output class identifier vectors;

[0214] The class identification constraint loss is obtained by averaging the distances based on the number of combinations.

[0215] In some example embodiments, when the processor 1010 performs the operation of determining the class identifier constraint loss based on the distance between every two output class identifier vectors, it specifically performs the following operations:

[0216] Based on the distance between every two output class identifier vectors, determine the loss weight of the class identifier pair corresponding to every two output class identifier vectors;

[0217] Based on the class identifier pair loss weights, the distances between every two output class identifier vectors in the N output class identifier vectors are weighted and summed to determine the class identifier constraint loss of the image feature extraction model.

[0218] In some example embodiments, when the processor 1010 performs the operation of determining the loss weights for class identifier pairs corresponding to each pair of output class identifier vectors based on the distance between each pair of output class identifier vectors, it specifically performs the following operations:

[0219] Based on the distance between every two output class identifier vectors, determine the normalized exponential function value corresponding to every two output class identifier vectors;

[0220] Based on the normalized exponential function value, the loss weights for each pair of output class identifier vectors are determined.

[0221] In some example embodiments, the model loss also includes a classification loss. When the processor 1010 performs the operation of determining the classification loss of the image feature extraction model based on the output class identifier vector and the label of the sample image, it specifically performs the following operations:

[0222] Determine the cross-entropy between the output class identifier vector and the label of the sample image;

[0223] The classification loss of the image feature extraction model is determined based on the cross-entropy.

[0224] In some example embodiments, the processor 1010 also performs the following operations:

[0225] Before inputting the connection vector sequence into the image feature extraction model, a position embedding vector corresponding to the position information of each of the multiple image blocks is obtained;

[0226] The position embedding vector is added to the corresponding connection vector in the connection vector sequence to obtain the added connection vector sequence.

[0227] In some example embodiments, when the processor 1010 performs the step of determining the block vector corresponding to each of the image blocks, it specifically performs the following operations:

[0228] By linearly projecting the pixel matrix corresponding to each image block, the block vector corresponding to each image block is obtained.

[0229] The method further includes:

[0230] The block vector sequence is formed based on the block vector corresponding to each of the image blocks.

[0231] In some example embodiments, when the processor 1010 performs the step of inputting the connection vector sequence into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors, it specifically performs the following operations:

[0232] Determine the query vector, key vector, and value vector corresponding to each join vector in the join vector sequence;

[0233] Perform a dot product between the query vector and the key vector corresponding to each of the connection vectors to obtain the attention value between the query vector and the key vector;

[0234] The value vector is weighted using the attention value as a weight to obtain the output class identifier vector.

[0235] In some example embodiments, the image feature extraction model is a Transformer model.

[0236] According to this instruction manual Figure 10 The technical solution of the embodiment, on the one hand, uses multiple learnable class tokens to learn multi-class representations of images. By constraining multiple class tokens through class token constraint loss, the image feature extraction model can autonomously perform differentiated representation learning, ensuring that the acquired image features have significant differences. On the other hand, by implicitly constraining the acquisition of different feature representations of images, it is possible to efficiently and cost-effectively acquire diverse image features, and the robustness of the model can be improved by integrating diverse image features.

[0237] The above is an illustrative embodiment of an electronic device according to this specification. It should be noted that the technical solution of this electronic device and the technical solution of the image processing method described above belong to the same concept. Details not described in detail in the technical solution of the electronic device can be found in the description of the technical solution of the image processing method described above.

[0238] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium for the computer program can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0239] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

[0240] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. An image model training method, comprising: The sample image is divided into multiple image blocks, and the block vector corresponding to each image block is determined. The block vectors corresponding to the multiple image blocks form a block vector sequence. The block vector sequence is concatenated with the class identifier vector sequence to generate a concatenated vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The class identifier vectors are used to learn the image features of the sample image. The connection vector sequence is input into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors; Based on the output class identifier vector and the label of the sample image, the model loss of the image feature extraction model is determined. The model loss includes class identifier constraint loss, which is used to learn multi-class image features of the sample image by constraining the output class identifier vector. The parameters of the input class identifier vector and the parameters of the image feature extraction model are adjusted based on the model loss.

2. The method according to claim 1, wherein determining the class labeling constraint loss of the image feature extraction model comprises: Determine the distance between any two output class identifier vectors among the N output class identifier vectors; The class labeling constraint loss of the image feature extraction model is determined based on the distance between any two output class labeling vectors, wherein the class labeling constraint loss is used to make the output class labeling vectors mutually orthogonal.

3. The method according to claim 2, wherein determining the class identifier constraint loss based on the distance between every two output class identifier vectors includes: The distances between any two output class identifier vectors in the N output class identifier vectors are summed to obtain the distance sum; Determine the number of combinations of taking two output class identifier vectors from the N output class identifier vectors; The class identification constraint loss is obtained by averaging the distances based on the number of combinations.

4. The method according to claim 2, wherein determining the class label constraint loss of the image feature extraction model based on the distance between every two output class label vectors includes: Based on the distance between every two output class identifier vectors, determine the loss weight of the class identifier pair corresponding to every two output class identifier vectors; Based on the class identifier pair loss weights, the distances between every two output class identifier vectors in the N output class identifier vectors are weighted and summed to determine the class identifier constraint loss of the image feature extraction model.

5. The method according to claim 4, wherein determining the loss weight of the class identifier pair corresponding to each pair of output class identifier vectors based on the distance between each pair of output class identifier vectors includes: Based on the distance between every two output class identifier vectors, determine the normalized exponential function value corresponding to every two output class identifier vectors; Based on the normalized exponential function value, the loss weights for each pair of output class identifier vectors are determined.

6. The method according to claim 1, wherein the model loss further includes a classification loss, and determining the classification loss of the image feature extraction model based on the output class identifier vector and the label of the sample image includes: Determine the cross-entropy between the output class identifier vector and the label of the sample image; The classification loss of the image feature extraction model is determined based on the cross-entropy.

7. The method according to claim 1, wherein before inputting the connection vector sequence into the image feature extraction model, the method further comprises: Obtain the position embedding vector corresponding to the position information of each of the multiple image blocks; The position embedding vector is added to the corresponding connection vector in the connection vector sequence to obtain the added connection vector sequence.

8. The method according to claim 1, wherein determining the block vector corresponding to each of the image blocks includes: By linearly projecting the pixel matrix corresponding to each image block, the block vector corresponding to each image block is obtained. The method further includes: The block vector sequence is formed based on the block vector corresponding to each of the image blocks.

9. The method according to claim 1, wherein inputting the connection vector sequence into an image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors comprises: Determine the query vector, key vector, and value vector corresponding to each join vector in the join vector sequence; Perform a dot product between the query vector and the key vector corresponding to each of the connection vectors to obtain the attention value between the query vector and the key vector; The value vector is weighted using the attention value as a weight to obtain the output class identifier vector.

10. The method according to any one of claims 1 to 9, wherein the image feature extraction model is a Transformer model.

11. An image processing method, comprising: The image to be identified is divided into multiple image blocks, and the block vector corresponding to each image block is determined. The block vectors corresponding to the image blocks in the multiple image blocks form a block vector sequence. The block vector sequence is concatenated with the class identifier vector sequence to generate a concatenated vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The input class identifier vectors are used to learn the image features of the sample image. The N input class identifier vectors are trained by the image model training method described in any one of claims 1 to 10. The connection vector sequence is input into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors. The image feature extraction model is trained by the image model training method according to any one of claims 1 to 10. The image to be identified is identified based on the N output class identifier vectors.

12. An image model training device, comprising: The block sequence determination module is used to divide the sample image into multiple image blocks, determine the block vector corresponding to each image block, and form a block vector sequence from the block vectors corresponding to the image blocks in the multiple image blocks. The connection processing module is used to connect the block vector sequence and the class identifier vector sequence to generate a connection vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The class identifier vectors are used to represent the sample image. The feature extraction module is used to input the connection vector sequence into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors; The loss determination module is used to determine the model loss of the image feature extraction model based on the output class identifier vector and the label of the sample image. The model loss includes class identifier constraint loss, which is used to learn multi-class image features of the sample image by constraining the output class identifier vector. The parameter adjustment module is used to adjust the parameters of the input class identifier vector and the parameters of the image feature extraction model based on the model loss.

13. An image processing apparatus, comprising: The block vector determination module is used to divide the image to be identified into multiple image blocks, determine the block vector corresponding to each image block, and form a block vector sequence from the multiple image blocks. A connection processing module is used to connect the block vector sequence and the class identifier vector sequence to generate a connection vector sequence. The class identifier vector sequence includes N input class identifier vectors, where N is a positive integer greater than or equal to 2. The input class identifier vectors are used to learn the image features of the sample image. The N input class identifier vectors are trained by the image model training method according to any one of claims 1 to 10. The feature extraction module is used to input the connection vector sequence into the image feature extraction model to obtain N output class identifier vectors corresponding to the N input class identifier vectors. The image feature extraction model is trained by the image model training method of any one of claims 1 to 10. The image recognition module is used to recognize the image to be recognized based on the N output class identifier vectors.

14. A computer storage medium storing a plurality of instructions adapted for loading by a processor and performing the steps of the method as claimed in any one of claims 1 to 11.

15. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 11.

16. A computer program product comprising instructions that, when run on a computer or processor, cause the computer or processor to perform the steps of the method as described in any one of claims 1-11.

Images