Multi-label Image Classification Method, Apparatus, Device, Storage Medium and Program Product

Abstract

The present invention provides a multi-label image classification method, apparatus, device, storage medium and program product, which are applied to the field of image processing technology. The method includes: obtaining a high-resolution feature map of the image to be classified, where the image to be classified includes at least two object categories; determining pixel masks for at least two object categories of the image to be classified, mapping the pixel masks onto the high-resolution feature map for mask averaging operation to obtain target image features; performing interactive processing on text features including category attribute descriptions and category names with the high-resolution feature map to obtain target text features including visual information; and matching the target image features with the target text features to determine the category label of the image to be classified.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a multi-label image classification method, apparatus, device, storage medium, and program product. Background Art

[0002] Label image classification is a very important task in the field of computer vision and occupies a very important part in applications such as smart cities and intelligent management.

[0003] In the prior art, one or more labels can be given to an image according to the objects appearing in the image. If there is only one object in the image, then it corresponds to a single-label classification task; if the image includes multiple objects, then it corresponds to a multi-label classification task. The multi-label classification task needs to classify an image containing multiple targets to accurately identify multiple categories.

[0004] However, although there are numerous multi-label image classification methods in the prior art, they still have deficiencies in terms of recognition accuracy. Summary of the Invention

[0005] The present invention provides a multi-label image classification method, apparatus, device, storage medium, and program product to solve the problem that the multi-label image classification methods in the prior art have deficiencies in terms of recognition accuracy.

[0006] The present invention provides a multi-label image classification method, including: obtaining a high-resolution feature map of an image to be classified, where at least two object categories are included in the image to be classified; determining pixel masks of at least two object categories of the image to be classified, mapping the pixel masks to the high-resolution feature map for mask averaging operation to obtain a target image feature; performing interactive processing on a text feature including category attribute description and category name with the high-resolution feature map to obtain a target text feature including visual information; and matching the target image feature with the target text feature to determine a category label of the image to be classified.

[0007] According to the multi-label image classification method provided by the present invention, the obtaining of the high-resolution feature map of the image to be classified includes: obtaining the image to be classified; extracting an image feature of the image to be classified; and inputting the image feature into a pixel encoder to obtain the high-resolution feature map.

[0008] A multi-label image classification method provided according to the present invention, determining pixel masks of at least two object categories of the image to be classified, and mapping the pixel masks onto the high-resolution feature map for mask averaging operation to obtain target image features, includes: randomly initializing a set of token vectors in the image to be classified; performing self-attention operation and normalization operation on the token vectors to obtain a first output result; performing token-to-image cross-attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; after the second output result passes through mapping processing of a linear layer and activation function processing in sequence, obtaining activation values; performing image-to-token cross-attention operation on the activation values and the high-resolution feature map to obtain a third output result from dense input to sparse input; performing token-to-image attention operation on the third output result and the activation values to obtain a fourth output result; using the fourth output result as the input of a linear layer for mapping processing to obtain a prediction result; after the fourth output result undergoes two transposed convolutional upsamplings, performing a dot product operation with the prediction result to obtain the target image features.

[0009] A multi-label image classification method provided according to the present invention, before the step of performing interaction processing on the text feature containing category attribute description and category name with the high-resolution feature map to obtain a target text feature containing visual information, the method further includes: generating a discriminative category attribute description for the object category to be recognized through a large language model; combining the category attribute description, category name, and adaptively learned text to form a text description; inputting the text description into a text encoder to obtain the text feature.

[0010] A multi-label image classification method provided according to the present invention, performing interaction processing on the text feature containing category attribute description and category name with the high-resolution feature map to obtain a target text feature containing visual information, includes: performing interaction processing on the text feature and the image feature through a cross-attention module, and then obtaining the target text feature through a linear layer.

[0011] A multi-label image classification method provided according to the present invention, after the step of matching the target image features and the target text features to determine the category label of the image to be classified, the method further includes: updating the model parameters of the image classification model by calculating the classification loss, where the image classification model is used to implement multi-label image classification.

[0012] The present invention also provides a multi-label image classification device, including the following modules: an acquisition module and a processing module; the acquisition module is used to acquire a high-resolution feature map of the image to be classified, and at least two object categories are included in the image to be classified; the processing module is used to determine pixel masks of at least two object categories of the image to be classified, map the pixel masks onto the high-resolution feature map for mask averaging operation to obtain a target image feature; perform interactive processing on the text feature including category attribute description and category name and the high-resolution feature map to obtain a target text feature including visual information; match the target image feature and the target text feature to determine the category label of the image to be classified.

[0013] According to a multi-label image classification device provided by the present invention, the acquisition module is used to acquire an image to be classified; extract the image feature of the image to be classified; input the image feature into a pixel encoder to obtain the high-resolution feature map.

[0014] According to a multi-label image classification device provided by the present invention, the processing module is used to randomly initialize a group of token vectors in the image to be classified; perform self-attention operation and normalization operation on the token vectors to obtain a first output result; perform token-to-image cross-attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; after the second output result passes through mapping processing and activation function processing of a linear layer in sequence, obtain an activation value; perform image-to-token cross-attention operation on the activation value and the high-resolution feature map to obtain a third output result from dense input to sparse input; perform token-to-image attention operation on the third output result and the activation value to obtain a fourth output result; use the fourth output result as the input of a linear layer for mapping processing to obtain a prediction result; after the fourth output result passes through two transposed convolutional upsamplings, perform dot product operation with the prediction result to obtain the target image feature.

[0015] According to a multi-label image classification device provided by the present invention, the processing module is used to generate a discriminative category attribute description for the object category to be recognized through a large language model; combine the category attribute description, category name, and adaptively learned text to form a text description; input the text description into a text encoder to obtain the text feature.

[0016] According to a multi-label image classification device provided by the present invention, the processing module is used to perform interactive processing on the text feature and the image feature through a cross-attention module, and obtain the target text feature through a linear layer.

[0017] A multi-label image classification device provided by the present invention, the processing module is used to update the model parameters of the image classification model by calculating the classification loss, and the image classification model is used to implement multi-label image classification.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the multi-label image classification method as described in any one of the above.

[0019] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the multi-label image classification method as described in any one of the above.

[0020] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the multi-label image classification method as described in any one of the above.

[0021] The multi-label image classification method, device, equipment, storage medium and program product provided by the present invention can obtain a high-resolution feature map of the image to be classified, where the image to be classified includes at least two object categories; determine pixel masks of at least two object categories of the image to be classified, map the pixel masks to the high-resolution feature map for mask averaging operation to obtain target image features; perform interactive processing on the text features including category attribute descriptions and category names with the high-resolution feature map to obtain target text features including visual information; match the target image features with the target text features to determine the category labels of the image to be classified. Through this solution, on the one hand, since pixel masks of at least two object categories of the image to be classified can be determined, the model can learn image features of different targets according to different masks. By decoupling the features of each target, the feature expression ability of each target can be enhanced, thereby improving the recognition accuracy; on the other hand, text features including category attribute descriptions and category names can be interactively processed with the high-resolution feature map to obtain target text features including visual information. Since the category attribute descriptions with distinctiveness between categories can be combined with the corresponding category names and texts, more discriminative semantic content for each category can be obtained, thereby improving the recognition accuracy; on the other hand, the target image features can be matched with the target text features. Since the regions corresponding to each target in the image to be classified can be matched with multiple text features, the text feature closest to the target to be recognized can be found, thereby improving the recognition accuracy. Description of the Drawings

[0022] To more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 is a schematic flowchart of the multi-label image classification method provided by the present invention;

[0024] Figure 2 is a schematic structural diagram of the multi-label image classification device provided by the present invention;

[0025] Figure 3 is a schematic structural diagram of the electronic device provided by the present invention. Specific Embodiments

[0026] To make the objectives, technical solutions, and advantages of the present application clearer, the following clearly and completely describes the technical solutions in the present application with reference to the accompanying drawings in the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts fall within the scope of protection of the present application.

[0027] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0028] It should be noted that in this text, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article, or device including that element. In addition, it should be pointed out that the methods and devices in the embodiments of the present application are not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0029] For the convenience of clearly describing the technical solutions of the embodiments of the present application, in the embodiments of the present application, terms such as "first", "second", etc. are used to distinguish identical or similar items with basically the same functions and roles. Those skilled in the art can understand that terms such as "first", "second", etc. are not used to limit the quantity and execution order.

[0030] Some exemplary embodiments are described for purposes of illustration in the embodiments of the present application. It should be understood that the present application may be implemented in other ways not specifically shown in the drawings.

[0031] As Figure 1 shown, the embodiments of the present application provide a multi-label image classification method, and this multi-label image classification method can be applied to a multi-label image classification device. This multi-label image classification method may include S101 - S104:

[0032] S101. The multi-label image classification device acquires a high-resolution feature map of the image to be classified.

[0033] Among them, at least two object categories are included in the image to be classified.

[0034] Optionally, the multi-label image classification device may acquire the image to be classified; extract the image features of the image to be classified; and input the image features into a pixel encoder to obtain the high-resolution feature map.

[0035] Specifically, the multi-label image classification device may first input the image to be classified into an image encoder to extract the image features of the image to be classified, and then input the extracted image features into a pixel encoder to obtain a high-resolution feature map.

[0036] Optionally, the above image encoder can be ViT-Base / 16.

[0037] It should be noted that inputting the image to be classified into the image encoder to extract image features can provide a basic feature representation for subsequent steps; sending the extracted image features into the pixel decoder to obtain a high-resolution feature map with a higher resolution is beneficial to improving the recognition rate of the model for objects, especially for small targets.

[0038] S102. The multi-label image classification device determines pixel masks for at least two object categories of the image to be classified, maps the pixel masks onto the high-resolution feature map for mask averaging operation, and obtains target image features.

[0039] Optionally, the multi-label image classification device can randomly initialize a set of token vectors in the image to be classified; perform self-attention operation and normalization operation on the token vectors to obtain a first output result; perform cross-attention operation from token to image on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; after the second output result passes through the mapping process of the linear layer and the activation function process in sequence, obtain activation values; perform cross-attention operation from image to token on the activation values and the high-resolution feature map to obtain a third output result from dense input to sparse input; perform token-to-image attention operation on the third output result and the activation values to obtain a fourth output result; use the fourth output result as the input of the linear layer for mapping process to obtain a prediction result; after the fourth output result undergoes two transposed convolutional upsamplings, perform dot product operation with the prediction result to obtain the target image features.

[0040] Specifically, the multi-label image classification device can execute steps 1021-1026:

[0041] Step 1021. Randomly initialize a set of token vectors, first perform self-attention operation through the self-attention module, and then perform normalization operation using the normalization layer to obtain a first output result.

[0042] Step 1022. Perform cross-attention operation from token to image on the first output result output in step 1021 and the high-resolution feature map to obtain a cross-attention result from sparse input to dense input, that is, a second output result.

[0043] Step 1023. Send the second output result into the linear layer, map it to a higher dimension, and then obtain activation values through the ReLU function.

[0044] Step 1024: Perform an image-to-token cross-attention operation on the activation value and the high-resolution feature map to obtain the cross-attention result from the dense input to the sparse input, which is the third output result.

[0045] Step 1025: Perform a token-to-image attention operation on the third output result and the "activation value" of the result of Step 1023, then send it to a linear layer with the same number of elements as the total number of categories in the image dataset to predict the output result and obtain the prediction result.

[0046] Step 1026: Upsample the third output result by a factor of 4 through two transposed convolutional layers, and perform a dot product operation with the prediction result obtained in Step 1025 to predict the mask of the target in the image to be recognized and obtain the target image feature.

[0047] It should be noted that the high-resolution feature map is sent into the mask encoder for further processing to predict the pixel-level mask related to the object categories contained in the image to be classified, and then the mask is mapped onto the high-resolution feature map, enabling the image classification model to learn the image features of different targets according to different masks, thereby decoupling the features of each target and making the target areas that the model focuses on more accurate when recognizing the objects contained in the image.

[0048] S103: The multi-label image classification device performs an interaction process on the text feature containing the category attribute description and the category name and the high-resolution feature map to obtain the target text feature containing visual information.

[0049] Optionally, before performing the interaction process on the text feature containing the category attribute description and the category name and the high-resolution feature map to obtain the target text feature containing visual information, the multi-label image classification device can generate a discriminative category attribute description for the object category to be recognized through a large language model; combine the category attribute description, the category name, and the adaptively learned text to form a text description; and input the text description into the text encoder to obtain the text feature.

[0050] Optionally, the multi-label image classification device can perform an interaction process on the text feature and the image feature through a cross-attention module and then obtain the target text feature through a linear layer.

[0051] Specifically, the multi-label image classification device can utilize a large language model to generate an attribute description that is most discriminative compared to all other object categories for the object category to be recognized, obtain the category attribute description, and combine the category attribute description and the object category name to be recognized with adaptively learned text, and send them into the text encoder to obtain text features. Then, the cross-attention module is used to interact the image features extracted in S101 above with the text features. Among them, the text features are used as K and V in the cross-attention module, and the image features are used as Q. Through the attention formula calculation, the features after cross-attention of the text and image modalities are obtained and normalized. Finally, a linear layer is used to output the text features containing visual information, that is, the target text features.

[0052] S104. The multi-label image classification device matches the target image features with the target text features to determine the category label of the image to be classified.

[0053] Optionally, after matching the target image features with the target text features to determine the category label of the image to be classified, the multi-label image classification device can update the model parameters of the image classification model by calculating the classification loss. The image classification model is used to implement multi-label image classification.

[0054] Specifically, in order to obtain the final classification result of the multi-label image, the multi-label image classification device can use the multi-label alignment module to match the target image features and target text features of different targets, and calculate the classification loss respectively.

[0055] It should be noted that this application uses text descriptions with discriminative attributes related to the target to make different targets easier to distinguish. In addition, the targets at different positions on the image will be automatically located by the model, so that the targets will not interfere with each other during recognition. Since the model has rich multi-modal information and can model the localization of the target, it has good image recognition ability, strong robustness and generalization ability.

[0056] In the embodiments of the present application, on the one hand, since at least two object category pixel masks of the image to be classified can be determined, the model can learn image features of different targets according to different masks. By decoupling the features of each target, the feature expression ability of each target can be enhanced, thereby improving the recognition accuracy. On the other hand, text features including category attribute descriptions and category names can be interacted with the high-resolution feature map to obtain target text features containing visual information. Since the category attribute descriptions with distinctiveness between categories can be combined with the corresponding category names and texts, more discriminative semantic content for each category can be obtained, thereby improving the recognition accuracy. On the other hand, the target image features and the target text features can be matched. Since the regions corresponding to each target in the image to be classified can be matched with multiple text features, the text features closest to the target to be recognized can be found, thereby improving the recognition accuracy.

[0057] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of the method. To implement the above functions, it includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0058] For the multi-label image classification method provided by the embodiments of the present application, the execution subject can be a multi-label image classification device, or a control module for multi-label image classification in the multi-label image classification device. In the embodiments of the present application, taking the multi-label image classification device executing the multi-label image classification method as an example, the multi-label image classification device provided by the embodiments of the present application is described.

[0059] It should be noted that the embodiments of the present application can divide the functional modules of the multi-label image classification device according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. Optionally, the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0060] Such as Figure 2As shown in the figure, an embodiment of the present application provides a multi-label image classification device 200. The multi-label image classification device 200 includes an acquisition module 201 and a processing module 202. The acquisition module 201 can be used to acquire a high-resolution feature map of the image to be classified, and at least two object categories are included in the image to be classified; the processing module 202 is used to determine pixel masks for at least two object categories of the image to be classified, map the pixel masks onto the high-resolution feature map for mask averaging operation to obtain target image features; perform interactive processing on the text features including category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; and match the target image features with the target text features to determine the category labels of the image to be classified.

[0061] Optionally, the acquisition module 201 is used to acquire the image to be classified; extract the image features of the image to be classified; and input the image features into a pixel encoder to obtain the high-resolution feature map.

[0062] Optionally, the processing module 202 is used to randomly initialize a set of token vectors in the image to be classified; perform self-attention operation and normalization operation on the token vectors to obtain a first output result; perform token-to-image cross-attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; after the second output result passes through the mapping processing of a linear layer and the activation function processing in sequence, obtain activation values; perform image-to-token cross-attention operation on the activation values and the high-resolution feature map to obtain a third output result from dense input to sparse input; perform token-to-image attention operation on the third output result and the activation values to obtain a fourth output result; use the fourth output result as the input of a linear layer for mapping processing to obtain a prediction result; and after the fourth output result undergoes two transposed convolutional upsamplings, perform a dot product operation with the prediction result to obtain the target image features.

[0063] Optionally, the processing module 202 is used to generate a discriminative category attribute description for the object category to be recognized through a large language model; combine the category attribute description, the category name, and adaptively learned text to form a text description; and input the text description into a text encoder to obtain the text features.

[0064] Optionally, the processing module 202 is used to perform interactive processing on the text features and the image features through a cross-attention module, and obtain the target text features through a linear layer.

[0065] Optionally, the processing module 202 is configured to update the model parameters of the image classification model by calculating the classification loss. The image classification model is used to implement multi-label image classification.

[0066] In the embodiments of the present application, on the one hand, since at least two object category pixel masks of the image to be classified can be determined, the model can learn image features of different targets according to different masks. By decoupling the features of each target, the feature expression ability of each target can be enhanced, thereby improving the recognition accuracy. On the other hand, the text features including category attribute descriptions and category names can be interacted with the high-resolution feature map to obtain target text features containing visual information. Since the category attribute descriptions with discrimination between categories can be combined with the corresponding category names and texts, more discriminative semantic content for each category can be obtained, thereby improving the recognition accuracy. On the third hand, the target image features and the target text features can be matched. Since the regions corresponding to each target in the image to be classified can be matched with multiple text features, the text features closest to the target to be recognized can be found, thereby improving the recognition accuracy.

[0067] Figure 3 An entity structure diagram of an electronic device is exemplified, as Figure 3 shown. The electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. Among them, the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the logical instructions in the memory 330 to execute the multi-label image classification method, which includes: obtaining a high-resolution feature map of the image to be classified, where the image to be classified includes at least two object categories; determining at least two object category pixel masks of the image to be classified, mapping the pixel masks to the high-resolution feature map for mask averaging operation to obtain target image features; performing interactive processing on the text features including category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; and matching the target image features with the target text features to determine the category label of the image to be classified.

[0068] In addition, when the logical instructions in the above-mentioned memory 330 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0069] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-label image classification method provided by the above-mentioned various methods. The method includes: obtaining a high-resolution feature map of the image to be classified, where the image to be classified includes at least two object categories; determining pixel masks for at least two object categories of the image to be classified, mapping the pixel masks onto the high-resolution feature map for mask averaging operation to obtain target image features; performing interactive processing on the text features including category attribute descriptions and category names with the high-resolution feature map to obtain target text features including visual information; and matching the target image features with the target text features to determine the category label of the image to be classified.

[0070] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the multi-label image classification method provided by the above-mentioned various methods. The method includes: obtaining a high-resolution feature map of the image to be classified, where the image to be classified includes at least two object categories; determining pixel masks for at least two object categories of the image to be classified, mapping the pixel masks onto the high-resolution feature map for mask averaging operation to obtain target image features; performing interactive processing on the text features including category attribute descriptions and category names with the high-resolution feature map to obtain target text features including visual information; and matching the target image features with the target text features to determine the category label of the image to be classified.

[0071] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0072] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-label image classification method, characterized in that including: Obtain a high-resolution feature map of the image to be classified, where the image to be classified includes at least two object categories; Determine the pixel masks of at least two object categories of the image to be classified, map the pixel masks onto the high-resolution feature map for mask averaging operation to obtain the target image feature; Perform interactive processing on the text feature containing category attribute description and category name with the high-resolution feature map to obtain the target text feature containing visual information; Match the target image feature with the target text feature to determine the category label of the image to be classified; The determining the pixel masks of at least two object categories of the image to be classified, mapping the pixel masks onto the high-resolution feature map for mask averaging operation to obtain the target image feature, includes: Randomly initialize a group of token vectors in the image to be classified; Perform self-attention operation and normalization operation on the token vectors to obtain the first output result; Perform cross-attention operation from token to image on the first output result and the high-resolution feature map to obtain the second output result from sparse input to dense input; After the second output result passes through the mapping processing of the linear layer and the activation function processing in sequence, obtain the activation value; Perform cross-attention operation from image to token on the activation value and the high-resolution feature map to obtain the third output result from dense input to sparse input; Perform token-to-image attention operation on the third output result and the activation value to obtain the fourth output result; Use the fourth output result as the input of the linear layer for mapping processing to obtain the prediction result; After the fourth output result undergoes two transposed convolutional upsamplings, perform dot product operation with the prediction result to obtain the target image feature.

2. The multi-label image classification method according to claim 1, wherein The obtaining the high-resolution feature map of the image to be classified, includes: Obtain the image to be classified; Extract the image feature of the image to be classified; Input the image feature into a pixel encoder to obtain the high-resolution feature map.

3. The multi-label image classification method according to claim 1, wherein Before the performing interactive processing on the text feature containing category attribute description and category name with the high-resolution feature map to obtain the target text feature containing visual information, the method further includes: Generate a discriminative category attribute description for the object category to be recognized through a large language model; Combine the category attribute description, category name, and adaptively learned text to form a text description; Input the text description into a text encoder to obtain the text feature.

4. The multi-label image classification method according to claim 3, characterized in that, The performing interactive processing on the text feature containing category attribute description and category name with the high-resolution feature map to obtain the target text feature containing visual information, includes: Perform interactive processing on the text feature and the image feature through a cross-attention module, and then obtain the target text feature through a linear layer.

5. The multi-label image classification method according to claim 1, wherein After the matching the target image feature with the target text feature to determine the category label of the image to be classified, the method further includes: Update the model parameters of the image classification model by calculating the classification loss, where the image classification model is used to implement multi-label image classification.

6. A multi-label image classification device, characterized in that including: An acquisition module and a processing module; The acquisition module is used to acquire a high-resolution feature map of the image to be classified, where the image to be classified includes at least two object categories; The processing module is used to determine pixel masks for at least two object categories of the image to be classified, map the pixel masks onto the high-resolution feature map for mask averaging operation to obtain target image features; Perform interactive processing on the text features including category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; Match the target image features with the target text features to determine the category label of the image to be classified; The processing module is used to randomly initialize a set of token vectors in the image to be classified; Perform self-attention operation and normalization operation on the token vectors to obtain a first output result; Perform token-to-image cross-attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; After the second output result passes through the mapping processing of the linear layer and the activation function processing in sequence, obtain activation values; Perform image-to-token cross-attention operation on the activation values and the high-resolution feature map to obtain a third output result from dense input to sparse input; Perform token-to-image attention operation on the third output result and the activation values to obtain a fourth output result; Use the fourth output result as the input of the linear layer for mapping processing to obtain a prediction result; After the fourth output result undergoes two transposed convolutional upsamplings, perform a dot product operation with the prediction result to obtain the target image features.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-label image classification method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-label image classification method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-label image classification method according to any one of claims 1 to 5.

Images