A multi-label image classification method, device, terminal and storage medium

Abstract

The application discloses a multi-label image classification method and device, a terminal and a storage medium. The method comprises the following steps: obtaining an image to be classified, determining a plurality of local features and a global feature corresponding to the image to be classified according to the image to be classified; obtaining a plurality of category labels, determining a plurality of target text embeddings according to initial text embeddings corresponding to each category label and each local feature; and determining an image classification result according to each initial text embedding, each target text embedding and the global feature. Since the application reconstructs the target text embedding according to the local feature and the initial text embedding of the image, dynamically determines the corresponding relationship between the local area of the image and the category label, effectively captures the complete feature of the object, and solves the problem that the existing multi-label classification method is difficult to capture the complete feature of the object area, thereby limiting the processing of new labels that have not been seen in the training process, and affecting the accuracy and robustness of multi-label classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and particularly relates to a multi-label image classification method and device, a terminal and a storage medium. BACKGROUND

[0002] For technologies such as automatic driving, virtual reality and robots, a picture can contain multiple objects, and these application scenarios require a classifier to accurately identify multiple labels in the image. Therefore, the multi-label classification task is becoming more and more important.

[0003] The existing multi-label classification method divides a picture into small blocks with a fixed grid, and calculates the similarity between each small block and a label, which is difficult to capture the complete features of the object region, resulting in limitations when processing new labels that have not been seen during training, affecting the accuracy and robustness of multi-label classification.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a multi-label image classification method, device, terminal and storage medium to solve the problem that the existing multi-label classification method is difficult to capture the complete features of the object region, resulting in limitations when processing new labels that have not been seen during training, affecting the accuracy and robustness of multi-label classification.

[0006] The technical solution adopted by the present application to solve the problem is as follows:

[0007] In a first aspect, the present application provides a multi-label image classification method, wherein the method comprises:

[0008] Obtaining a to-be-classified image, determining a plurality of local features and a global feature corresponding to the to-be-classified image according to the to-be-classified image;

[0009] Obtaining a plurality of category labels, determining a plurality of target text embeddings according to the initial text embeddings corresponding to each of the category labels and each of the local features;

[0010] Determining an image classification result according to each of the initial text embeddings, each of the target text embeddings and the global feature.

[0011] In an implementation method, the determining a plurality of local features and a global feature corresponding to the to-be-classified image according to the to-be-classified image comprises:

[0012] Segmenting, linearly mapping and position encoding the to-be-classified image to determine a plurality of feature vectors corresponding to the to-be-classified image;

[0013] learning each feature vector through an encoding layer to obtain the local feature and the global feature corresponding to the classification image, wherein the encoding layer comprises N encoding blocks, and each encoding block comprises a multi-head self-attention mechanism, a normalization layer and a multi-layer perceptron.

[0014] In an implementation method, learning each feature vector through an encoding layer to obtain the local feature and the global feature corresponding to the classification image comprises:

[0015] inputting each feature vector into the image encoder for learning to obtain encoding block data output by each encoding block in the image encoder, wherein the image encoder comprises N encoding blocks, the input data of the first encoding block is each feature vector, the input data of the i-th encoding block is the encoding block data output by the (i-1)-th encoding block, and 0

[0016] obtaining the encoding block data output by the (N-1)-th encoding block as the local feature;

[0017] obtaining the encoding block data output by the N-th encoding block as the global feature.

[0018] In an implementation method, determining a plurality of target text embeddings according to each initial text embedding corresponding to each category label and each local feature comprises:

[0019] calculating a first cosine similarity of each initial text embedding and each local feature;

[0020] determining a plurality of target text embeddings according to each first cosine similarity and each local feature.

[0021] In an implementation method, determining a plurality of target text embeddings according to each first cosine similarity and each local feature comprises:

[0022] summing each first cosine similarity to determine a first cosine similarity sum;

[0023] determining a plurality of similarity weights corresponding to each local feature according to each first cosine similarity corresponding to the local feature and the first cosine similarity sum;

[0024] reconstructing a target text embedding corresponding to the local feature according to the local feature and each similarity weight.

[0025] In an implementation method, determining an image classification result according to each initial text embedding, each target text embedding and the global feature comprises:

[0026] compute a second cosine similarity of the initial text embedding and the global feature;

[0027] compute a third cosine similarity of the initial text embedding and the target text embedding corresponding to the initial text embedding;

[0028] determine the image classification result according to the second cosine similarity and the third cosine similarity.

[0029] In an implementation method, the determining the image classification result according to the second cosine similarity and the third cosine similarity comprises:

[0030] normalizing the second cosine similarity and the third cosine similarity respectively to determine a first confidence and a second confidence;

[0031] obtaining a preset weight, linearly combining the first confidence and the second confidence based on the preset weight, and determining the image classification result.

[0032] In a second aspect, an embodiment of the present application further provides a multi-label image classification device, and the multi-label image classification device comprises:

[0033] a feature extraction module configured to obtain an image to be classified, and determine a plurality of local features and a global feature corresponding to the image to be classified;

[0034] a label reconstruction module configured to obtain a plurality of category labels, and determine a plurality of target text embeddings according to initial text embeddings corresponding to each of the category labels and each of the local features;

[0035] an image classification module configured to determine an image classification result according to each of the initial text embeddings, each of the target text embeddings, and the global feature.

[0036] In a third aspect, an embodiment of the present application further provides a terminal, which comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the multi-label image classification method described in any of the above aspects; and the processor is configured to execute the programs.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement the multi-label image classification method described in any of the above aspects.

[0038] The beneficial effects of the present application: the embodiment of the present application obtains a to-be-classified image, determines a plurality of local features and global features corresponding to the to-be-classified image according to the to-be-classified image; obtains a plurality of category labels, determines a plurality of target text embeddings according to the initial text embedding corresponding to each category label and each local feature; determines an image classification result according to each initial text embedding, each target text embedding and the global feature. Since the present application reconstructs the target text embedding according to the local feature of the image and the initial text embedding, dynamically determines the corresponding relationship between the local area of the image and the category label, can effectively capture the complete feature of the object, solves the problem that the existing multi-label classification method is difficult to capture the complete feature of the object area, and has limitations when processing new labels that have not been seen before in the training process, affects the accuracy and robustness of multi-label classification. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 is a flowchart of the multi-label image classification method provided by the embodiment of the present application.

[0041] Figure 2 is a multi-label image classification method using a CLIP (language-image pre-training) model provided by the embodiment of the present application.

[0042] Figure 3 is an internal module diagram of the multi-label image classification device provided by the embodiment of the present application.

[0043] Figure 4 is a principle block diagram of the terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application discloses a multi-label image classification method, device, terminal and storage medium. In order to make the purpose, technical scheme and effect of the present application more clear and definite, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0045] As will be understood by persons skilled in the art of the technology to which this application pertains, the singular forms "a," "an," "said," and "the" include plural referents unless the context clearly dictates otherwise. It should be further understood that the word "comprise" and variations of the word, such as "comprising," "comprises," and "comprised of," as used herein, do not exclude the presence of other elements or steps than those listed. It is to be understood that when we refer to an element being "connected" or "coupled" to another element by a medium, it can be directly connected or coupled or additional intervening elements can also be present. In addition, "connected" or "coupled" as used herein can include wirelessly connected or wirelessly coupled. As used herein, the term "and / or" includes all combinations of one or more of the associated listed items and all combinations of the associated listed items.

[0046] As will be understood by persons skilled in the art of the technology to which this application pertains, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0047] For technologies such as autonomous driving, virtual reality, and robots, a picture can contain multiple items, and these application scenarios require a classifier to accurately identify multiple labels in an image. Therefore, the multi-label classification task is becoming more and more important.

[0048] Existing multi-label classification methods divide a picture into small blocks with a fixed grid, and calculate the similarity between each small block and a label, which is difficult to capture the complete features of the object region, resulting in limitations when processing new labels that have not been seen during training, affecting the accuracy and robustness of multi-label classification.

[0049] In view of the above defects of the prior art, the present application provides a multi-label image classification method, which comprises the following steps: obtaining an image to be classified, determining a plurality of local features and a global feature corresponding to the image to be classified according to the image to be classified; obtaining a plurality of category labels, determining a plurality of target text embeddings according to the initial text embeddings corresponding to each of the category labels and each of the local features; and determining an image classification result according to each of the initial text embeddings, each of the target text embeddings and the global feature. Since the target text embedding is reconstructed according to the local features of the image and the initial text embedding, the corresponding relationship between the local region of the image and the category label is dynamically determined, the complete features of the object can be effectively captured, and the problem that the existing multi-label classification method is difficult to capture the complete features of the object region, resulting in limitations when processing new labels not seen in the training process, affecting the accuracy and robustness of multi-label classification, is solved.

[0050] Exemplary method

[0051] As shown in Figure 1 the method comprises the following steps:

[0052] In step S100, an image to be classified is obtained, and a plurality of local features and a global feature corresponding to the image to be classified are determined according to the image to be classified.

[0053] The image to be classified includes a plurality of objects to be classified. In this embodiment, the plurality of local features and the global feature corresponding to the image to be classified are obtained according to the image to be classified, so as to realize classification of each object to be classified in the image to be classified, solve the problem that when the local feature is used for classification alone, the error of classification is high when the local feature does not contain the target object to be classified, and the problem that when the global feature is used for classification alone, the details of the local region in the image cannot be effectively captured.

[0054] In an implementation manner, the plurality of local features and the global feature corresponding to the image to be classified are determined according to the image to be classified, which comprises the following steps:

[0055] In step S101, the image to be classified is segmented, linearly mapped and positionally encoded, and a plurality of feature vectors corresponding to the image to be classified are determined.

[0056] In step S102, each of the feature vectors is learned by an encoding layer to obtain the local features and the global feature corresponding to the image to be classified, wherein the encoding layer comprises N encoding blocks, each of the encoding blocks comprises a multi-head self-attention mechanism, a normalization layer and a multi-layer perceptron.

[0057] Specifically, the input image to be classified is segmented into a plurality of 2D image blocks. Assuming that the size of the input image to be classified is HxW, the image to be classified is segmented into image blocks with a size of PxP. The specific process is as follows:

[0058] #Assume the input image to be classified is x_q, with a size of H×W, where P is the image patch size, x_patch is the image patch, and split_image_into_patches is the image segmentation function.

[0059] x_patch=split_image_into_patches(x_q,P).

[0060] After segmentation, image patches are linearly mapped onto a feature space via a trainable linear projection, resulting in feature vectors. Each image patch corresponds to one feature vector. The parameters of the feature vectors are continuously adjusted during training to optimize the feature representation.

[0061] # Linear mapping, where linear_projection is the linear mapping function.

[0062] x_patch=linear_projection(x_patch).

[0063] The feature vectors obtained after linear mapping are positionally encoded using a positional encoding layer to obtain several feature vectors corresponding to the image to be classified. The encoding layer then learns from these feature vectors to obtain the local and global features of the image to be classified.

[0064] like Figure 2 As shown, this embodiment employs a visual transformer (ViT) in a pre-trained vision-language model (such as the CLIP model). The visual transformer includes 12 coding blocks, each of which includes a multi-head self-attention mechanism (MSA), a normalization layer (NORM), and a multilayer perceptron (MLP). The visual transformer is used to segment, linearly map, encode the position of the image to be classified, and learn each feature vector to obtain the local and global features corresponding to the image to be classified.

[0065] Specifically, by learning each of the feature vectors through an encoding layer, the local features and global features corresponding to the classified image are obtained, including:

[0066] The feature vectors are input into the image encoder for learning to obtain the coded block data output by each coded block in the image encoder. The image encoder includes N coded blocks, the input data of the first coded block is the feature vectors, the input data of the i-th coded block is the coded block data output by the (i-1)-th coded block, and 0... <i<N;

[0067] The coded block data output by the (N-1)th coded block is obtained as the local feature;

[0068] obtain the encoding block data of the Nth encoding block output as the global feature.

[0069] Specifically, taking the CLIP model as an example, the CLIP model is good at extracting global features of the whole picture. If a local image block is input, the representation ability of the CLIP will be poor. In the feature extraction process of the CLIP model, with the increase of the number of layers, the output features will change from describing local features to tending to describe global features. Global features are more suitable for single-label classification tasks, but not suitable for multi-label classification tasks. The embodiment is aimed at the above characteristics of the CLIP model. The output features of the N-1th encoding block of the visual transformer of the CLIP model are used as local features, and the output features of the Nth encoding block are used as global features. The local features and global features can be obtained simply and quickly, so as to effectively capture the local information and global information of the image to be classified, avoid the problem that the output features of the image encoder are not suitable for multi-label classification tasks, and improve the accuracy and robustness of the multi-label classification result in the embodiment.

[0070] The specific process of obtaining the local features and global features based on the feature vector learning by the visual transformer includes:

[0071] x_0=[E_cls,x_patch]+E_pos

[0072] for k in range(K):

[0073] y=x_k-1+MSA(NORM(x_k-1))

[0074] x_k=y_k+MLP(NORM(y_k))

[0075] #obtain global features and local features

[0076] m_cls,m_patch=x_K

[0077] Wherein, E_cls represents a class token embedding, E_pos represents a position embedding, [·,·] represents a splicing operation, m_cls is a global feature, and m_patch is a local feature.

[0078] Step S200, obtain a plurality of category labels, and determine a plurality of target text embeddings according to the initial text embedding corresponding to each category label and each local feature.

[0079] Since the segmented image block may not contain the target to be classified or only contains a small part, the complete features of the target to be classified cannot be preserved, and the similarity calculation of the local features corresponding to the image block and the category label will bring errors. In order to improve the classification performance and capture the complete features of the target to be classified as much as possible, the initial text embedding corresponding to each local feature and each category label is reconstructed to dynamically determine the corresponding relationship between the local region of the image and the category label, thereby solving the problem of being difficult to capture the complete features of the target to be classified.

[0080] In an implementation manner, the determining of the target text embedding according to the initial text embedding corresponding to each category label and each local feature comprises:

[0081] In step S201, the first cosine similarity of each initial text embedding and each local feature is calculated.

[0082] In step S202, a plurality of target text embeddings are determined according to each first cosine similarity and each local feature.

[0083] Specifically, for each category label, the text editor of the CLIP model is used to convert the category label into a text embedding to obtain the initial text embedding corresponding to each category label. For example, the text editor accepts a sentence containing a category label and generates a corresponding text embedding:

[0084] # Assuming that the category label is d, and the CLIP_text_encoder is a text encoder,

[0085] t_i=CLIP_text_encoder("There is a"+d+"in the scene.")。

[0086] The cosine similarity between the initial text embedding corresponding to each category label and each local feature is calculated as the first cosine similarity:

[0087] s_ij=cosine_similarity(t_i,m_patch_j)。

[0088] cosine_similarity is a cosine similarity function, t_i is the i-th initial text embedding, and m_patch_j is the j-th local feature.

[0089] For the reconstruction of the target text embedding, it needs to be considered which image features are combined to obtain the target text embedding. In this embodiment, the first cosine similarity is used to measure the similarity between the initial text embedding and the local feature, and the weight of each local feature for reconstructing the target text feature is determined, and the local feature is dynamically selected according to the weight to reconstruct the target text embedding, so as to ensure that the reconstructed target text embedding can accurately reflect the local features of the image. The target text embedding corresponding to each local feature is determined according to the first cosine similarity and the local feature. Specifically, the first cosine similarity is summed to determine the first cosine similarity sum, the first cosine similarity corresponding to the local feature is used to determine the similarity weight corresponding to the local feature, and the target text embedding corresponding to the local feature is reconstructed according to the local feature and the similarity weight. The implementation is as follows:

[0090] t_i* = sum_j (s_ij / sum(s_ij)*m_patch_j),

[0091] t_i* is the i-th target text embedding.

[0092] In one implementation, a learnable correlation coefficient matrix A is introduced, which can dynamically calculate the cosine similarity between each local feature of the image and each initial text embedding. Based on the calculated cosine similarity, when reconstructing the target text embedding, the local features of the image can be dynamically selected.

[0093] Step S300, determining an image classification result according to each initial text embedding, each target text embedding, and the global feature.

[0094] In short, this embodiment combines the reconstructed target text embedding and the initial text embedding-image similarity to generate the final image classification result.

[0095] In one implementation, the determination of the image classification result according to each initial text embedding, each target text embedding, and the global feature includes:

[0096] Step S301, calculating a second cosine similarity between the initial text embedding and the global feature;

[0097] Step S302, calculating a third cosine similarity between the initial text embedding and the target text embedding corresponding to the initial text embedding;

[0098] Step S303, determining the image classification result according to the second cosine similarity and the third cosine similarity.

[0099] Specifically, the embodiment adopts a text encoder and an image encoder of a CLIP model, calculates a cosine similarity between an initial text embedding corresponding to a category label and a global feature of the image to be classified as a second cosine similarity S_CLIP, and captures global information of the image:

[0100] S_CLIP = cosine_similarity (t_i, m_cls).

[0101] A cosine similarity between the reconstructed target text embedding and the initial text embedding corresponding to the category label is calculated as a third cosine similarity S R to quantify the reconstruction error:

[0102] S R = cosine_similarity (t_i, t_i*).

[0103] The second cosine similarity and the third cosine similarity are further combined to generate a final image classification result.

[0104] In an implementation manner, the determining the image classification result according to the second cosine similarity and the third cosine similarity comprises:

[0105] Step S3031, normalizing the second cosine similarity and the third cosine similarity respectively to determine a first confidence and a second confidence;

[0106] Step S3032, obtaining a preset weight, and linearly combining the first confidence and the second confidence based on the preset weight to determine the image classification result.

[0107] Specifically, the second cosine similarity and the third cosine similarity are normalized respectively to obtain a first confidence corresponding to the second cosine similarity and a second confidence corresponding to the third cosine similarity. The first confidence and the second confidence represent the confidence of the image on the corresponding category label. A preset weight is obtained, and the first confidence and the second confidence are linearly combined based on the preset weight, so as to obtain a final confidence as the image classification result.

[0108] Specifically, the linear combination of the first confidence and the second confidence based on the preset weight is represented as:

[0109] S_total = (1-η)*S_CLIP + η*S_R,

[0110] S_CLIP represents the second cosine similarity or the first confidence, S R represents the third cosine similarity or the second confidence, and η is the preset weight. The preset weight can be obtained by pre-setting or by model optimization.

[0111] In one implementation, the preset weight η is set to 0.9, which can make the multi-label image classification method in this embodiment achieve better accuracy and robustness.

[0112] For the proposed multi-label image classification method, this embodiment enhances the learning ability of the model by using supervised contrastive loss, ensuring that the model can effectively capture class-specific features. The specific steps are as follows:

[0113] (1) Supervised contrastive loss calculation. Use supervised contrastive loss (Supervised Contrastive Loss) to enhance the diversity and relevance of attention masks. This loss function converts ranking loss into classification loss by integrating non-normalized matching scores and contrastive loss:

[0114] L = -sum(log(sigmoid(s_p)))-sum(log(1-sigmoid(s_n))).

[0115] (2) Model training. Use AdamW optimizer to train the model, adjust the learning rate and weight decay parameters to optimize the model performance. During the training process, gradually adjust the model parameters to better capture the local and global features of the image.

[0116] optimizer = AdamW(model.parameters(), lr = 0.001, weight_decay = 0.005)

[0117] for epoch in range(num_epochs):

[0118] train_step(model, optimizer, loss_function, train_loader).

[0119] This embodiment conducts experiments on NUS-WIDE and Open Images datasets to evaluate the performance of the multi-label image classification method. The experimental results show that the multi-label image classification method outperforms existing methods in general zero-shot tasks, with an average precision (mAP) improvement of 2.2% and 2.1%, respectively.

[0120] Experimental steps:

[0121] (1) Data preprocessing: Convert NUS-WIDE and Open Images datasets into data formats suitable for model training and evaluation.

[0122] (2) Model training: Train the model on the training set using the above optimization and training method.

[0123] (3) Model evaluation: Evaluate the performance of the model on the test set, calculate the mean average precision (mAP), precision, recall, and F1 score.

[0124] # Calculate mAP

[0125] mAP = calculate_mAP(predictions, ground_truth).

[0126] # Calculate Precision, Recall and F1-Score

[0127] precision, recall, f1 = calculate_precision_recall_f1(predictions, ground_truth).

[0128] calculate_precision_recall_f1(predictions, ground_truth).

[0129] Based on the above embodiments, the application also provides a multi-label image classification device, as shown in Figure 3 The device comprises:

[0130] a feature extraction module 01 for obtaining an image to be classified, determining a plurality of local features and global features corresponding to the image to be classified according to the image to be classified;

[0131] a label reconstruction module 02 for obtaining a plurality of category labels, determining a plurality of target text embeddings according to the initial text embeddings corresponding to each of the category labels and each of the local features;

[0132] an image classification module 03 for determining an image classification result according to each of the initial text embeddings, each of the target text embeddings, and the global feature.

[0133] Based on the above embodiments, the application also provides a terminal, and its principle block diagram can be as shown in Figure 4 The terminal comprises a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the multi-label image classification method. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0134] Those skilled in the art can understand that,Figure 4 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the solution of the present application, and does not constitute a limitation on the terminal to which the solution of the present application is applied. The specific terminal can include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In an implementation, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, which contains instructions for performing the multi-label image classification method.

[0136] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the method can be included. Any reference to memory, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0137] To sum up, the application discloses a multi-label image classification method and device, a terminal and a storage medium, the method obtains an image to be classified, determines a plurality of local features and a global feature corresponding to the image to be classified according to the image to be classified, obtains a plurality of category labels, determines a plurality of target text embeddings according to initial text embeddings corresponding to each of the category labels and each of the local features, and determines an image classification result according to each of the initial text embeddings, each of the target text embeddings and the global feature. Since the application reconstructs the target text embedding according to the local feature and the initial text embedding of the image, dynamically determines the corresponding relationship between the local area of the image and the category label, can effectively capture the complete feature of the object, solves the problem that the existing multi-label classification method is difficult to capture the complete feature of the object area, and has limitations when processing new labels that have not been seen in the training process, and affects the accuracy and robustness of multi-label classification.

[0138] It should be understood that the application is not limited to the examples described above, and that all modifications and variations that can be made by a person of ordinary skill in the art based on the above description should be within the scope of protection of the appended claims of the application.

Claims

1. A multi-label image classification method, characterized by, The method comprises: acquiring a to-be-classified image, and determining a plurality of local features and a global feature corresponding to the to-be-classified image according to the to-be-classified image; acquiring a plurality of category labels, and determining a plurality of target text embeddings according to an initial text embedding corresponding to each of the category labels and each of the local features; determining an image classification result according to each of the initial text embeddings, each of the target text embeddings, and the global feature; the determining of the plurality of local features and the global feature corresponding to the to-be-classified image according to the to-be-classified image comprises: segmenting, linearly mapping, and positionally encoding the to-be-classified image to determine a plurality of feature vectors corresponding to the to-be-classified image; learning each of the feature vectors through an encoding layer to obtain the local features and the global feature corresponding to the to-be-classified image, wherein the encoding layer comprises N encoding blocks, each of the encoding blocks comprises a multi-head self-attention mechanism, a normalization layer, and a multi-layer perceptron; the learning of each of the feature vectors through the encoding layer to obtain the local features and the global feature corresponding to the to-be-classified image comprises: inputting each of the feature vectors into the encoding layer for learning to obtain encoding block data output by each of the encoding blocks in the encoding layer, wherein the input data of the first encoding block is each of the feature vectors, the input data of the i-th encoding block is the encoding block data output by the (i-1)-th encoding block, and 0 < i < N; obtaining the encoding block data output by the (N-1)-th encoding block as the local features; obtaining the encoding block data output by the N-th encoding block as the global feature; the determining of a plurality of target text embeddings according to an initial text embedding corresponding to each of the category labels and each of the local features comprises: calculating a first cosine similarity of each of the initial text embeddings and each of the local features; determining a plurality of target text embeddings according to each of the first cosine similarities and each of the local features.

2. The multi-label image classification method of claim 1, wherein, the determining of a plurality of target text embeddings according to each of the first cosine similarities and each of the local features comprises: summing each of the first cosine similarities to determine a first cosine similarity sum; determining a plurality of similarity weights corresponding to the local features according to each of the first cosine similarities corresponding to the local features and the first cosine similarity sum; reconstructing a target text embedding corresponding to the local features according to the local features and each of the similarity weights.

3. The multi-label image classification method of claim 1, wherein, the determining of an image classification result according to each of the initial text embeddings, each of the target text embeddings, and the global feature comprises: calculating a second cosine similarity of the initial text embedding and the global feature; calculating a third cosine similarity of the initial text embedding and the target text embedding corresponding to the initial text embedding; determining the image classification result according to the second cosine similarity and the third cosine similarity.

4. The multi-label image classification method of claim 3, wherein, the determining of the image classification result according to the second cosine similarity and the third cosine similarity comprises: respectively normalizing the second cosine similarity and the third cosine similarity to determine a first confidence and a second confidence; Obtaining a preset weight, linearly combining the first confidence and the second confidence based on the preset weight, and determining the image classification result.

5. A multi-label image classification apparatus characterized by comprising: The device comprises: The feature extraction module is configured to obtain a to-be-classified image, and determine a plurality of local features and a global feature corresponding to the to-be-classified image according to the to-be-classified image; The label reconstruction module is configured to obtain a plurality of category labels, and determine a plurality of target text embeddings according to initial text embeddings corresponding to each of the category labels and each of the local features; The image classification module is configured to determine an image classification result according to each of the initial text embeddings, each of the target text embeddings, and the global feature; The determination of the plurality of local features and the global feature corresponding to the to-be-classified image according to the to-be-classified image comprises: segmenting, linearly mapping, and positionally encoding the to-be-classified image to determine a plurality of feature vectors corresponding to the to-be-classified image; learning each of the feature vectors through an encoding layer to obtain the local features and the global feature corresponding to the to-be-classified image, wherein the encoding layer comprises N encoding blocks, each of the encoding blocks comprises a multi-head self-attention mechanism, a normalization layer, and a multi-layer perceptron; The learning of each of the feature vectors through the encoding layer to obtain the local features and the global feature corresponding to the to-be-classified image comprises: inputting each of the feature vectors into the encoding layer for learning to obtain encoding block data output by each of the encoding blocks in the encoding layer, wherein the input data of the first encoding block is each of the feature vectors, the input data of the i-th encoding block is the encoding block data output by the (i-1)-th encoding block, and 0 < i < N; obtaining the encoding block data output by the (N-1)-th encoding block as the local features; obtaining the encoding block data output by the N-th encoding block as the global feature; The determination of a plurality of target text embeddings according to initial text embeddings corresponding to each of the category labels and each of the local features comprises: calculating a first cosine similarity of each of the initial text embeddings and each of the local features; determining a plurality of target text embeddings according to each of the first cosine similarities and each of the local features.

6. A terminal, characterized by comprising: The terminal comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the multi-label image classification method of any one of claims 1-4; and the processor is configured to execute the programs.

7. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are suitable for being loaded and executed by the processor to implement the steps of the multi-label image classification method of any one of claims 1-4.

Images