Multi-label image classification method and device, equipment, storage medium and program product

Abstract

The invention provides a multi-label image classification method and device, equipment, a storage medium and a program product, and is applied to the technical field of image processing. The method comprises the steps of obtaining a high-resolution feature map of a to-be-classified image, wherein the to-be-classified image comprises at least two object categories; determining pixel masks of at least two object categories of the to-be-classified image, and mapping the pixel masks to the high-resolution feature map for mask averaging operation to obtain a target image feature; performing interactive processing on text features containing category attribute description and category names and the high-resolution feature map to obtain target text features containing visual information; and matching the target image features with the target text features to determine category labels of the to-be-classified images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-label image classification method, device, equipment, storage medium and program product. Background Art

[0002] Label image classification is a very important task in the field of computer vision. It plays a very important role in applications such as smart cities and smart management.

[0003] In the prior art, one or more labels can be given to an image according to the objects that appear in the image. If there is only one object in the image, it corresponds to a single-label classification task; if the image includes multiple objects, it corresponds to a multi-label classification task. The multi-label classification task requires classifying images containing multiple objects in order to accurately identify multiple categories.

[0004] However, although there are many multi-label image classification methods in the prior art, they still have shortcomings in recognition accuracy. Summary of the invention

[0005] The present invention provides a multi-label image classification method, device, equipment, storage medium and program product, which are used to solve the problem that the multi-label image classification method in the prior art is insufficient in recognition accuracy.

[0006] The present invention provides a multi-label image classification method, comprising: obtaining a high-resolution feature map of an image to be classified, wherein the image to be classified includes at least two object categories; determining pixel masks of at least two object categories of the image to be classified, mapping the pixel masks to the high-resolution feature map to perform mask averaging operations to obtain target image features; interactively processing text features containing category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; matching the target image features with the target text features to determine the category label of the image to be classified.

[0007] According to a multi-label image classification method provided by the present invention, obtaining a high-resolution feature map of an image to be classified includes: obtaining an image to be classified; extracting image features of the image to be classified; and inputting the image features into a pixel encoder to obtain the high-resolution feature map.

[0008] According to a multi-label image classification method provided by the present invention, the pixel masks of at least two object categories of the image to be classified are determined, and the pixel masks are mapped to the high-resolution feature map for mask averaging operation to obtain target image features, including: randomly initializing a group of word element vectors in the image to be classified; performing self-attention operation and normalization operation on the word element vector to obtain a first output result; performing a word element to image cross attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; after the second output result is sequentially subjected to linear layer mapping processing and activation function processing, an activation value is obtained; the activation value and the high-resolution feature map are subjected to image to word element cross attention operation to obtain a third output result from dense input to sparse input; the third output result and the activation value are subjected to word element to image attention operation to obtain a fourth output result; the fourth output result is used as the input of the linear layer for mapping processing to obtain a prediction result; the fourth output result is subjected to two transposed convolution upsampling and then subjected to a dot multiplication operation with the prediction result to obtain the target image features.

[0009] According to a multi-label image classification method provided by the present invention, before the text features containing category attribute descriptions and category names are interactively processed with the high-resolution feature map to obtain target text features containing visual information, the method also includes: generating a discriminative category attribute description for the object category to be identified through a large language model; combining the category attribute description, the category name and the adaptively learned text to form a text description; and inputting the text description into a text encoder to obtain the text features.

[0010] According to a multi-label image classification method provided by the present invention, the text features containing category attribute descriptions and category names are interactively processed with the high-resolution feature map to obtain target text features containing visual information, including: after interactively processing the text features with the image features through a cross-attention module, the target text features are obtained through a linear layer.

[0011] According to a multi-label image classification method provided by the present invention, after matching the target image features with the target text features to determine the category label of the image to be classified, the method also includes: updating the model parameters of the image classification model by calculating the classification loss, and the image classification model is used to realize multi-label image classification.

[0012] The present invention also provides a multi-label image classification device, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to acquire a high-resolution feature map of an image to be classified, wherein the image to be classified includes at least two object categories; the processing module is used to determine pixel masks of at least two object categories of the image to be classified, map the pixel masks to the high-resolution feature map, perform mask averaging operations, and obtain target image features; interactively process text features containing category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; match the target image features with the target text features to determine the category label of the image to be classified.

[0013] According to a multi-label image classification device provided by the present invention, the acquisition module is used to acquire the image to be classified; extract the image features of the image to be classified; input the image features into a pixel encoder to obtain the high-resolution feature map.

[0014] According to a multi-label image classification device provided by the present invention, the processing module is used to randomly initialize a group of word element vectors in the image to be classified; perform self-attention operations and normalization operations on the word element vectors to obtain a first output result; perform a word-to-image cross-attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; the second output result is sequentially subjected to mapping processing of the linear layer and activation function processing to obtain an activation value; the activation value and the high-resolution feature map are subjected to a cross-attention operation from image to word element to obtain a third output result from dense input to sparse input; the third output result and the activation value are subjected to a word-to-image attention operation to obtain a fourth output result; the fourth output result is used as the input of the linear layer for mapping processing to obtain a prediction result; the fourth output result is subjected to two transposed convolution upsamplings and then subjected to a dot multiplication operation with the prediction result to obtain the target image feature.

[0015] According to a multi-label image classification device provided by the present invention, the processing module is used to generate a discriminative category attribute description for the object category to be identified through a large language model; combine the category attribute description, the category name and the adaptively learned text to form a text description; and input the text description into a text encoder to obtain the text feature.

[0016] According to a multi-label image classification device provided by the present invention, the processing module is used to obtain the target text feature through a linear layer after interactively processing the text feature and the image feature through a cross-attention module.

[0017] According to a multi-label image classification device provided by the present invention, the processing module is used to update the model parameters of the image classification model by calculating the classification loss, and the image classification model is used to implement multi-label image classification.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-label image classification method described above is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-label image classification methods described above.

[0020] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multi-label image classification method as described above is implemented.

[0021] The multi-label image classification method, device, equipment, storage medium and program product provided by the present invention can obtain a high-resolution feature map of an image to be classified, wherein the image to be classified includes at least two object categories; determine the pixel masks of at least two object categories of the image to be classified, map the pixel masks to the high-resolution feature map for mask averaging operation, and obtain target image features; interactively process text features containing category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; match the target image features with the target text features to determine the category label of the image to be classified. Through this scheme, on the one hand, since the pixel masks of at least two object categories of the image to be classified can be determined, the model can learn the image features of different targets according to different masks. By decoupling the features of each target, the feature expression ability of each target can be enhanced, thereby improving the recognition accuracy; on the other hand, the text features containing category attribute descriptions and category names can be interactively processed with the high-resolution feature map to obtain target text features containing visual information. Since the category attribute descriptions that are discriminative between categories can be combined with the corresponding category names and texts, more discriminative semantic content of each category can be obtained, thereby improving the recognition accuracy; on the other hand, the target image features can be matched with the target text features. Since the area corresponding to each target in the image to be classified can be matched with multiple text features, the text features closest to the target to be identified can be found, thereby improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a flowchart of the multi-label image classification method provided by the present invention; Figure 2 It is a structural schematic diagram of the multi-label image classification device provided by the present invention; Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0026] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0027] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and order of execution.

[0028] The embodiments of the present application describe some exemplary embodiments for the purpose of explanation. It should be understood that the present application can be implemented in other ways that are not specifically shown in the drawings.

[0029] like Figure 1 As shown, the embodiment of the present application provides a multi-label image classification method, which can be applied to a multi-label image classification device. The multi-label image classification method may include S101-S104: S101. A multi-label image classification device obtains a high-resolution feature map of an image to be classified.

[0030] Wherein, the above-mentioned image to be classified includes at least two object categories.

[0031] Optionally, the multi-label image classification device can obtain an image to be classified; extract image features of the image to be classified; and input the image features into a pixel encoder to obtain the high-resolution feature map.

[0032] Specifically, the multi-label image classification device can first input the image to be classified into an image encoder to extract image features of the image to be classified, and then input the extracted image features into a pixel encoder to obtain a high-resolution feature map.

[0033] Optionally, the above-mentioned image encoder may be ViT-Base / 16.

[0034] It should be noted that inputting the image to be classified into the image encoder to extract image features can provide basic feature representation for subsequent steps; sending the extracted image features to the pixel decoder to obtain a high-resolution feature map with a larger resolution is conducive to improving the model's recognition rate of objects, especially for small targets.

[0035] S102. The multi-label image classification device determines pixel masks of at least two object categories of the image to be classified, maps the pixel masks to the high-resolution feature map, performs a mask averaging operation, and obtains target image features.

[0036] Optionally, the multi-label image classification device can randomly initialize a group of word element vectors in the image to be classified; perform self-attention and normalization operations on the word element vectors to obtain a first output result; perform a word-to-image cross-attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; subject the second output result to sequential mapping processing and activation function processing of the linear layer to obtain an activation value; perform an image-to-word element cross-attention operation on the activation value and the high-resolution feature map to obtain a third output result from dense input to sparse input; perform a word-to-image attention operation on the third output result and the activation value to obtain a fourth output result; use the fourth output result as the input of the linear layer to perform mapping processing to obtain a prediction result; subject the fourth output result to two transpose convolution upsamplings, and then perform a dot multiplication operation with the prediction result to obtain the target image feature.

[0037] Specifically, the multi-label image classification device can perform steps 1021-1026: Step 1021: randomly initialize a set of word element vectors, first perform self-attention operation through the self-attention module, and then perform normalization operation through the normalization layer to obtain a first output result; Step 1022: Perform a word-to-image cross attention operation on the first output result outputted in step 1021 and the high-resolution feature map to obtain a cross attention result from sparse input to dense input, i.e., a second output result; Step 1023: Send the second output result to the linear layer, map it to a higher dimension, and then obtain an activation value through the ReLU function; Step 1024: Perform a cross-attention operation from image to word on the activation value and the high-resolution feature map to obtain a cross-attention result from dense input to sparse input, i.e., a third output result; Step 1025: perform a word-to-image attention operation on the third output result and the result "activation value" of step 1023, and then send them to a linear layer with the same number of categories as the total number in the image data set to predict the output result and obtain a predicted result; Step 1026: upsample the third output result by 4 times through two transposed convolutions, and perform a dot multiplication operation with the prediction result obtained in step 1025 to predict the mask of the target in the image to be identified and obtain the target image feature.

[0038] It should be noted that the high-resolution feature map is sent to the mask encoder for further processing to predict the pixel-level mask related to the object category contained in the image to be classified, and then the mask is mapped to the high-resolution feature map, so that the image classification model can learn the image features of different targets according to different masks, thereby decoupling the features of each target, so that the model can identify the objects contained in the image more accurately.

[0039] S103, the multi-label image classification device interactively processes the text features containing the category attribute description and the category name with the high-resolution feature map to obtain target text features containing visual information.

[0040] Optionally, before interactively processing the text features containing category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information, the multi-label image classification device can generate a discriminative category attribute description for the object category to be identified through a large language model; combine the category attribute description, category name and adaptively learned text to form a text description; and input the text description into a text encoder to obtain the text features.

[0041] Optionally, the multi-label image classification device can interactively process the text features and the image features through a cross-attention module, and then obtain the target text features through a linear layer.

[0042] Specifically, the multi-label image classification device can use a large language model to generate the most distinguishing attribute description for the object category to be identified compared with all other object categories, obtain the category attribute description, and combine the category attribute description and the name of the object category to be identified with the adaptively learned text, and send it to the text encoder to obtain text features. Afterwards, the image features extracted in S101 and the text features are interacted using a cross-attention module, wherein the text features are regarded as K and V in the cross-attention module, and the image features are regarded as Q. The features after the cross-attention of the image-text modality are obtained and normalized through the attention formula, and finally the linear layer is used to output the text features containing visual information, that is, the target text features.

[0043] S104: The multi-label image classification device matches the target image feature with the target text feature to determine the category label of the image to be classified.

[0044] Optionally, after matching the target image features with the target text features to determine the category label of the image to be classified, the multi-label image classification device can update the model parameters of the image classification model by calculating the classification loss, and the image classification model is used to implement multi-label image classification.

[0045] Specifically, in order to obtain the final classification result of the multi-label image, the multi-label image classification device can use the multi-label alignment module to match the target image features and target text features of different targets, and calculate the classification losses respectively.

[0046] It should be noted that this application uses text descriptions with distinguishable attributes related to the target, making it easier to distinguish different targets. In addition, targets at different positions on the image will be automatically located by the model, so that the targets will not interfere with each other during recognition. Since the model has rich multimodal information and can model the location of targets, it has good image recognition capabilities and strong robustness and generalization.

[0047] In an embodiment of the present application, on the one hand, since the pixel masks of at least two object categories of the image to be classified can be determined, the model can learn the image features of different targets according to different masks. By decoupling the features of each target, the feature expression ability of each target can be enhanced, thereby improving the recognition accuracy; on the other hand, the text features containing the category attribute description and the category name can be interactively processed with the high-resolution feature map to obtain the target text features containing visual information. Since the category attribute description that is discriminative between categories can be combined with the corresponding category name and text, more discriminative semantic content of each category can be obtained, thereby improving the recognition accuracy; on the other hand, the target image features can be matched with the target text features. Since the area corresponding to each target in the image to be classified can be matched with multiple text features, the text features closest to the target to be identified can be found, thereby improving the recognition accuracy.

[0048] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0049] The multi-label image classification method provided in the embodiment of the present application can be executed by a multi-label image classification device or a control module for multi-label image classification in the multi-label image classification device. In the embodiment of the present application, the multi-label image classification device executing the multi-label image classification method is taken as an example to illustrate the multi-label image classification device provided in the embodiment of the present application.

[0050] It should be noted that the embodiment of the present application can divide the functional modules of the multi-label image classification device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0051] like Figure 2 As shown, an embodiment of the present application provides a multi-label image classification device 200. The multi-label image classification device 200 includes: an acquisition module 201 and a processing module 202. The acquisition module 201 can be used to acquire a high-resolution feature map of an image to be classified, wherein the image to be classified includes at least two object categories; the processing module 202 is used to determine the pixel masks of at least two object categories of the image to be classified, map the pixel masks to the high-resolution feature map, perform mask averaging operations, and obtain target image features; interactively process text features containing category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; match the target image features with the target text features to determine the category label of the image to be classified.

[0052] Optionally, the acquisition module 201 is used to acquire an image to be classified; extract image features of the image to be classified; and input the image features into a pixel encoder to obtain the high-resolution feature map.

[0053] Optionally, the processing module 202 is used to randomly initialize a group of word element vectors in the image to be classified; perform self-attention operations and normalization operations on the word element vectors to obtain a first output result; perform a word-to-image cross-attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; subject the second output result to sequential mapping processing and activation function processing of the linear layer to obtain an activation value; perform an image-to-word element cross-attention operation on the activation value and the high-resolution feature map to obtain a third output result from dense input to sparse input; perform a word-to-image attention operation on the third output result and the activation value to obtain a fourth output result; use the fourth output result as the input of the linear layer to perform mapping processing to obtain a prediction result; subject the fourth output result to two transpose convolution upsamplings, and then perform a dot multiplication operation with the prediction result to obtain the target image feature.

[0054] Optionally, the processing module 202 is used to generate a discriminative category attribute description for the object category to be identified through a large language model; combine the category attribute description, the category name and the adaptively learned text to form a text description; and input the text description into a text encoder to obtain the text feature.

[0055] Optionally, the processing module 202 is used to obtain the target text feature through a linear layer after interactively processing the text feature and the image feature through a cross-attention module.

[0056] Optionally, the processing module 202 is used to update model parameters of an image classification model by calculating classification loss, and the image classification model is used to implement multi-label image classification.

[0057] In an embodiment of the present application, on the one hand, since the pixel masks of at least two object categories of the image to be classified can be determined, the model can learn the image features of different targets according to different masks. By decoupling the features of each target, the feature expression ability of each target can be enhanced, thereby improving the recognition accuracy; on the other hand, the text features containing the category attribute description and the category name can be interactively processed with the high-resolution feature map to obtain the target text features containing visual information. Since the category attribute description that is discriminative between categories can be combined with the corresponding category name and text, more discriminative semantic content of each category can be obtained, thereby improving the recognition accuracy; on the other hand, the target image features can be matched with the target text features. Since the area corresponding to each target in the image to be classified can be matched with multiple text features, the text features closest to the target to be identified can be found, thereby improving the recognition accuracy.

[0058] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the multi-label image classification method, which includes: obtaining a high-resolution feature map of the image to be classified, wherein the image to be classified includes at least two object categories; determining pixel masks of at least two object categories of the image to be classified, mapping the pixel masks to the high-resolution feature map for mask averaging operation, and obtaining target image features; interactively processing the text features containing the category attribute description and the category name with the high-resolution feature map to obtain the target text features containing visual information; matching the target image features with the target text features to determine the category label of the image to be classified.

[0059] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0060] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-label image classification method provided by the above methods, and the method includes: obtaining a high-resolution feature map of the image to be classified, wherein the image to be classified includes at least two object categories; determining pixel masks of at least two object categories of the image to be classified, mapping the pixel masks to the high-resolution feature map for mask averaging operations to obtain target image features; interactively processing text features containing category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; matching the target image features with the target text features to determine the category label of the image to be classified.

[0061] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-label image classification method provided by the above-mentioned methods, the method comprising: obtaining a high-resolution feature map of an image to be classified, wherein the image to be classified includes at least two object categories; determining pixel masks of at least two object categories of the image to be classified, mapping the pixel masks to the high-resolution feature map for performing a mask averaging operation to obtain target image features; interactively processing text features containing category attribute descriptions and category names with the high-resolution feature map to obtain target text features containing visual information; matching the target image features with the target text features to determine the category label of the image to be classified.

[0062] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0063] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-label image classification method, characterized in that: include: Acquire a high-resolution feature map of an image to be classified, wherein the image to be classified includes at least two object categories; Determine pixel masks of at least two object categories of the image to be classified, map the pixel masks to the high-resolution feature map, perform mask averaging operation, and obtain target image features; Interactively processing the text features containing the category attribute description and the category name with the high-resolution feature map to obtain target text features containing visual information; The target image feature is matched with the target text feature to determine the category label of the image to be classified.

2. The multi-label image classification method according to claim 1, characterized in that: The step of obtaining a high-resolution feature map of an image to be classified comprises: Get the image to be classified; Extracting image features of the image to be classified; The image features are input into a pixel encoder to obtain the high-resolution feature map.

3. The multi-label image classification method according to claim 1, characterized in that: The step of determining pixel masks of at least two object categories of the image to be classified, mapping the pixel masks to the high-resolution feature map, performing a mask averaging operation, and obtaining target image features includes: Randomly initialize a set of word element vectors in the image to be classified; Performing a self-attention operation and a normalization operation on the word element vector to obtain a first output result; Performing a word-to-image cross attention operation on the first output result and the high-resolution feature map to obtain a second output result from sparse input to dense input; The second output result is sequentially processed by a linear layer and an activation function to obtain an activation value; Performing an image-to-word cross attention operation on the activation value and the high-resolution feature map to obtain a third output result from dense input to sparse input; Performing a word-to-image attention operation on the third output result and the activation value to obtain a fourth output result; Using the fourth output result as the input of the linear layer, performing mapping processing, and obtaining a prediction result; The fourth output result is up-sampled by two transposed convolutions and then subjected to a dot multiplication operation with the prediction result to obtain the target image feature.

4. The multi-label image classification method according to claim 1, characterized in that: Before interactively processing the text features containing the category attribute description and the category name with the high-resolution feature map to obtain the target text features containing visual information, the method further includes: Generate discriminative category attribute descriptions for the object categories to be identified through a large language model; Combining the category attribute description, the category name, and the adaptively learned text to form a text description; The text description is input into a text encoder to obtain the text features.

5. The multi-label image classification method according to claim 2, characterized in that: The interactive processing of the text features containing the category attribute description and the category name with the high-resolution feature map to obtain the target text features containing visual information includes: After interactively processing the text features and the image features through a cross-attention module, the target text features are obtained through a linear layer.

6. The multi-label image classification method according to claim 1, characterized in that: After matching the target image feature with the target text feature to determine the category label of the image to be classified, the method further includes: The model parameters of the image classification model are updated by calculating the classification loss, wherein the image classification model is used to implement multi-label image classification.

7. A multi-label image classification device, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire a high-resolution feature map of an image to be classified, wherein the image to be classified includes at least two object categories; The processing module is used to determine pixel masks of at least two object categories of the image to be classified, map the pixel masks to the high-resolution feature map, perform mask averaging operation, and obtain target image features; Interactively processing the text features containing the category attribute description and the category name with the high-resolution feature map to obtain target text features containing visual information; The target image feature is matched with the target text feature to determine the category label of the image to be classified.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multi-label image classification method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-label image classification method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-label image classification method according to any one of claims 1 to 6 is implemented.

Images