Classification model training method and device, classification method and device, equipment and storage medium

Abstract

The invention relates to a multi-label classification model training method, an image multi-label classification method and device, computer equipment and a computer readable storage medium. The multi-label classification model training method comprises the steps that image sample features and label text features obtained based on a preset label word bank are fused, and fused sample features are obtained; inputting the fusion sample features into a preset basic classification model; the basic classification model is used for generating and outputting corresponding classification probability information and image description information according to the fusion sample features; determining a first multi-label classification loss according to the classification probability information, and training a basic classification model based on the first multi-label classification loss; determining a second multi-label classification loss according to the image description information; based on the second multi-label classification loss, training of the basic classification model is assisted; and obtaining an image multi-label classification model. By adopting the method, the accuracy of multi-label classification can be improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a multi-label classification model training method and device, an image multi-label classification method, device, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] For multi-label classification of images, the same image is classified into multiple labels. With the development of artificial intelligence technology, using large models for multi-label classification is one of the directions of technological development, but the corresponding label classification accuracy needs to be improved. Summary of the invention

[0003] Based on this, it is necessary to provide a multi-label classification model training method, an image multi-label classification method, an apparatus, a computer device, a computer-readable storage medium and a computer program product to improve the accuracy of image multi-label classification in order to solve the above-mentioned technical problems.

[0004] In a first aspect, the present application provides a multi-label classification model training method, comprising:

[0005] Extract features from image samples to obtain image sample features;

[0006] The image sample features and the label text features obtained based on the preset label vocabulary are fused to obtain the fused sample features;

[0007] Input the fused sample features into a preset basic classification model; the basic classification model is used to generate classification probability information of each label in the label vocabulary of the image sample according to the fused sample features, as well as image description information of the image sample and output it;

[0008] Determine a first multi-label classification loss according to the classification probability information, and train a basic classification model based on the first multi-label classification loss;

[0009] And, determining a second multi-label classification loss according to the image description information; and assisting in training a basic classification model based on the second multi-label classification loss;

[0010] Based on the training of the basic classification model, an image multi-label classification model is obtained; the image multi-label classification model is used to perform multi-label classification on the images to be classified.

[0011] In one embodiment, the multi-label classification model training method further includes: determining long-tail class label information; inputting the long-tail class label information into a basic classification model; the basic classification model is further used to generate and output semantic description information for the long-tail class label based on the long-tail class label information;

[0012] Determining a second multi-label classification loss according to the image description information includes: determining the second multi-label classification loss according to the image description information and the semantic description information.

[0013] In one of the embodiments, the multi-label classification model training method further includes: expanding the label vocabulary according to the long-tail label information to increase the types and or quantity of the long-tail labels in the label vocabulary;

[0014] The image sample features and the label text features obtained based on a preset label vocabulary are fused, including: the image sample features and the label text features obtained based on an expanded label vocabulary are fused.

[0015] In one embodiment, a first multi-label classification loss is determined according to classification probability information, and a basic classification model is trained based on the first multi-label classification loss, including: determining the first multi-label classification loss according to the classification probability information and based on an asymmetric loss function, and training the basic classification model based on the first multi-label classification loss to increase the basic classification model's attention to long-tail class labels.

[0016] In one embodiment, the multi-label classification model training method further includes: constructing a text prompt, the text prompt including an overall information description task and a detailed information description task for the image;

[0017] Inputting the fused sample features into a preset basic classification model includes: inputting the fused sample features and text prompts into the basic classification model to guide the basic classification model to describe the overall and detailed aspects of the image sample, and obtain image description information of the image sample.

[0018] In a second aspect, the present application also provides an image multi-label classification method, comprising:

[0019] Perform feature extraction on the image to be classified to obtain image features;

[0020] The image features are fused with the label text features obtained based on a preset label vocabulary to obtain fused features;

[0021] The fused features are input into a pre-trained image multi-label classification model to obtain a classification result of the image to be classified relative to multiple labels; the multiple labels are labels in a label vocabulary; the image multi-label classification model is trained based on the multi-label classification model training method in the first aspect mentioned above.

[0022] In a third aspect, the present application also provides a multi-label classification model training device, comprising:

[0023] The feature module is used to extract features from image samples to obtain image sample features; fuse the image sample features with label text features obtained based on a preset label vocabulary to obtain fused sample features;

[0024] The input module is used to input the fused sample features into a preset basic classification model; the basic classification model is used to generate classification probability information of each label in the label vocabulary of the image sample according to the fused sample features, as well as image description information of the image sample and output it;

[0025] A training module is used to determine a first multi-label classification loss according to classification probability information, and train a basic classification model based on the first multi-label classification loss; and determine a second multi-label classification loss according to image description information; based on the second multi-label classification loss, assist in the training of the basic classification model; based on the training of the basic classification model, an image multi-label classification model is obtained; the image multi-label classification model is used to perform multi-label classification on the image to be classified.

[0026] In a fourth aspect, the present application also provides an image multi-label classification device, comprising:

[0027] The feature fusion module is used to extract features of the image to be classified to obtain image features; the image features are fused with the label text features obtained based on a preset label vocabulary to obtain fused features;

[0028] A classification module is used to input the fusion features into a pre-trained image multi-label classification model to obtain the classification results of the image to be classified relative to multiple labels; the multiple labels are labels in the label vocabulary; the image multi-label classification model is trained based on the multi-label classification model training method in the first aspect mentioned above.

[0029] In a fifth aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the multi-label classification model training method in the first aspect, or implements the image multi-label classification method in the second aspect.

[0030] In a sixth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-label classification model training method in the aforementioned first aspect, or implements the image multi-label classification method in the aforementioned second aspect.

[0031] In the seventh aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-label classification model training method in the aforementioned first aspect, or implements the image multi-label classification method in the aforementioned second aspect.

[0032] The above-mentioned multi-label classification model training method and device, image multi-label classification method, computer equipment, computer-readable storage medium and computer program product, by fusing the image sample features with the label text features corresponding to the label vocabulary, obtain the fused sample features and input them into the basic classification model to obtain the corresponding classification probability information and image description information. The basic classification model is trained by the first multi-label classification loss determined based on the classification probability information, and the training of the basic classification model is assisted by the second multi-label classification loss determined based on the image description information. In this way, the "primary-auxiliary" joint training of the basic classification model is realized, which enables the basic classification model to learn the relationship between different labels, so that the classification model obtained by training can improve the accuracy of multi-label classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of a process of a multi-label classification model training method in one embodiment;

[0035] Figure 2 Another schematic diagram of a multi-label classification model training method in one embodiment;

[0036] Figure 3 A schematic diagram of a system architecture corresponding to an image multi-label classification method in an embodiment;

[0037] Figure 4 is a flowchart of a multi-label image classification method in another embodiment;

[0038] Figure 5 It is a structural block diagram of a multi-label classification model training device in one embodiment;

[0039] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Unless otherwise specified, the "multiple" referred to herein may be two or more.

[0041] The image multi-label classification method provided in this application can be applied to all scenarios that require multi-label classification of images. For example, it can be applied to: 1. Public cloud multi-label image classification service: the multi-label recognition multimodal model trained by this method can be launched on the public cloud AI platform to provide multi-label recognition services for thousands of image categories; 2. Privatization scenarios: such as security, personal cloud disk, etc.

[0042] In one embodiment, Figure 1 As shown, a multi-label classification model training method is provided. This embodiment uses the method applied to a server as an example, for example, a cloud disk server. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes steps S101 to S105:

[0043] Step S101: extract features from image samples to obtain image sample features.

[0044] Among them, the image samples can be understood in a broad sense, which can be various types of images, including real images, such as surveillance images, photos, etc., and can also include non-real images, for example, cartoon images, etc.

[0045] Exemplarily, a specific algorithm and / or artificial intelligence model may be used to extract features from image samples.

[0046] Step S102: Fusing the image sample features with the label text features obtained based on a preset label vocabulary to obtain fused sample features.

[0047] The preset tag word library may be a pre-set word database storing various tags. For example, the tag word library may be dynamically changing, and the tag word library may be updated and expanded according to actual needs.

[0048] Exemplarily, the extraction of label text features may be performed synchronously or asynchronously with the extraction of image sample features.

[0049] In some possible embodiments, feature extraction may be performed on the tag word library in advance to obtain tag text features, and the tag text features may be cached. When needed, the cached tag text features may be directly obtained.

[0050] Step S103: input the fused sample features into a preset basic classification model; the basic classification model is used to generate classification probability information of each label in the label vocabulary of the image sample and image description information of the image sample according to the fused sample features and output them.

[0051] The preset basic classification model may be a pre-set classification model to be trained. For example, an existing classification model may be directly used as the basic classification model.

[0052] Among them, the image description information can be information that describes the image sample based on one or more or all tags in the tag vocabulary. For example, it can be information that describes the content of the image sample based on tags such as "portrait", "mood", and "weather".

[0053] Exemplarily, after the fused sample features are input into a preset basic classification model, the basic classification model can perform a multi-label classification task to obtain classification probability information of each label in the label vocabulary to which the image sample belongs, and generate a description text for the image sample as image description information.

[0054] Step S104: Determine a first multi-label classification loss according to the classification probability information, and train a basic classification model based on the first multi-label classification loss; and, determine a second multi-label classification loss according to the image description information; and assist in training the basic classification model based on the second multi-label classification loss.

[0055] In some possible embodiments, for the classification probability information output by the basic classification model, the classification effect of the model can be evaluated by determining the accuracy of the classification probability information, so that the model can be trained and optimized based on the evaluation. For example, the evaluation can be achieved by calculating the loss. Exemplarily, to avoid the one-sidedness of the evaluation, the evaluation can be performed from two aspects: the classification probability information and the image description information.

[0056] For example, the first multi-label classification loss can be determined according to the classification probability information, and the second multi-label classification loss can be determined according to the image description information. The first multi-label classification loss can be used to perform the main training work on the basic classification model, and the second multi-label classification loss can be used to assist the training of the basic classification model.

[0057] Exemplarily, the basic classification model can parse the direct or indirect connections or relationships between different labels from the image description information and / or the second multi-label classification loss, thereby improving the accuracy of multi-label classification.

[0058] Step S105: Based on the training of the basic classification model, an image multi-label classification model is obtained; the image multi-label classification model is used to perform multi-label classification on the image to be classified.

[0059] Exemplarily, there is no limit on the number of training times for the basic classification model, and it can be adjusted according to actual conditions.

[0060] In this embodiment, by fusing the image sample features with the label text features corresponding to the label vocabulary, the fused sample features are obtained and input into the basic classification model to obtain the corresponding classification probability information and image description information. The basic classification model is trained by the first multi-label classification loss determined based on the classification probability information, and the training of the basic classification model is assisted by the second multi-label classification loss determined based on the image description information. This realizes the "main-auxiliary" joint training of the basic classification model, which enables the basic classification model to learn the relationship between different labels, thereby improving the accuracy of multi-label classification.

[0061] In one embodiment, if Figure 2 As shown, the multi-label classification model training method in the above embodiment may further include steps S201 to S202:

[0062] Step S201: Determine long-tail label information.

[0063] Among them, the long-tail class label can be a rare label. The long-tail class label information can be related information obtained based on the long-tail class label, for example, it can be obtained by extracting features of the long-tail class label. Exemplarily, the long-tail class label information can be determined according to the frequency of the label appearing in the image sample. For example, in 10 consecutive image samples, for labels that appear less than 2 times, they can be classified as long-tail class labels.

[0064] The long-tail phenomenon is a concept in statistics and is prevalent in many practical applications, especially in the context of large-scale data sets. This phenomenon leads to an imbalance in data distribution, which poses a challenge to the training of machine learning models, because the model may tend to predict categories with high frequency and ignore those with low frequency. Similarly, in the training process of a multi-label classification model, long-tail class labels are easily ignored by the model, thereby affecting the model training effect and the accuracy of multi-label classification. In this regard, illustratively, the long-tail class label information can be incorporated into the second multi-label classification loss to avoid the model's neglect of the long-tail class labels, which can be achieved specifically through the following steps.

[0065] Step S202: inputting the long-tail class tag information into the basic classification model; the basic classification model is also used to generate and output semantic description information for the long-tail class tags according to the long-tail class tag information.

[0066] Exemplarily, the long-tail class label information can be input into the basic classification model to obtain semantic description information of the long-tail class label. The input here can be performed synchronously or asynchronously with the aforementioned input of the fused sample features into the basic classification model.

[0067] For example, the long-tail class label information can be packaged or fused with the aforementioned fused sample features and then output together as a basic classification model.

[0068] In the aforementioned embodiment, determining the second multi-label classification loss according to the image description information may include step S203: determining the second multi-label classification loss according to the image description information and the semantic description information.

[0069] Exemplarily, after obtaining the semantic description information for the long-tail class label, the second multi-label classification loss can be determined based on the image description information and the semantic description information, thereby serving the subsequent step of training the basic classification model.

[0070] In this embodiment, by determining the long-tail class label information and inputting it into the basic classification model to obtain semantic description information of the long-tail class label, the semantic description information is used as one of the bases for determining the second multi-label classification loss. This enables the subsequent training of the basic classification model based on the second multi-label classification loss, which can increase the model's attention to the long-tail class labels, thereby helping to improve the accuracy of the model's multi-label classification.

[0071] In one of the embodiments, the multi-label classification model training method also includes: expanding the label vocabulary based on the long-tail label information to increase the types and or quantity of long-tail labels in the label vocabulary; the fusing of the image sample features and the label text features obtained based on the preset label vocabulary in the aforementioned embodiment may include: fusing the image sample features and the label text features obtained based on the expanded label vocabulary.

[0072] As mentioned above, the tag word library can be dynamically changed and can be expanded according to actual needs. In this embodiment, considering that long-tail tags are one of the key factors affecting classification accuracy in multi-tag classification, long-tail tags can be specifically expanded in the tag word library.

[0073] For example, a hierarchy of long-tail class labels can be constructed to classify fine-grained labels into coarser-grained categories while retaining the fine-grained labels; multiple long-tail categories can be merged into a superclass to increase the number of samples, and then fine-grained classification can be performed to achieve expansion.

[0074] Exemplarily, the long-tail tags can be expanded with synonyms to increase the variants of each tag; Exemplarily, the long-tail tags can be expanded with related words, and the word vector model is used to find words related to the existing tags and add them to the vocabulary.

[0075] In this embodiment, for long-tail labels, the label vocabulary is expanded accordingly, which improves the richness of long-tail labels in the label vocabulary; the label text features can be obtained based on the expanded label vocabulary, so that the label text features contain more features or information related to the long-tail labels, which helps to avoid the model's neglect of long-tail labels, thereby improving the model training effect and enabling the model to achieve more accurate multi-label classification.

[0076] In one embodiment, a first multi-label classification loss is determined according to classification probability information, and a basic classification model is trained based on the first multi-label classification loss, including: determining the first multi-label classification loss according to the classification probability information and based on an asymmetric loss function, and training the basic classification model based on the first multi-label classification loss to increase the basic classification model's attention to long-tail class labels.

[0077] For example, when determining the first multi-label classification loss, due to the existence of long-tail class labels, the determined first multi-label classification loss may not be accurate, thereby affecting the training of the model. In this regard, an asymmetric function, such as an ASL function (Asymmetric Loss, asymmetric loss, referred to as ASL), can be used to determine the first multi-label classification loss, thereby increasing the basic classification model's attention to long-tail class labels.

[0078] In this embodiment, in the training task of the basic classification model, an asymmetric function is used to determine the first multi-label classification loss, so that the training of the basic classification model according to the first multi-label classification loss is better, which helps to solve the problem of label category imbalance and thus improve the accuracy of the model's multi-label classification.

[0079] In one of the embodiments, the multi-label classification model training method also includes: constructing a text prompt, the text prompt includes an overall information description task and a detailed information description task for the image; the aforementioned embodiment of inputting the fused sample features into a preset basic classification model may include: inputting the fused sample features and the text prompt into the basic classification model to guide the basic classification model to describe the image sample as a whole and in detail, and obtain image description information for the image sample.

[0080] Exemplarily, as described above, after the fusion sample features are input into the basic classification model, the tasks corresponding to the basic classification model may include generating and outputting classification probability information and image description information. For the generation of image description information, in order to improve the accuracy and comprehensiveness of the image description information, the prompt engineering technology may be used to guide the basic classification model.

[0081] Exemplarily, a large language model may be used to construct all or part of the text prompt.

[0082] For example, a text prompt may be pre-built, and the text prompt includes a task of describing the overall information of the image and a task of describing the detailed information. For example, "Please describe the overall layout of this image, all objects in the image, and the detailed features such as color and shape of each object in detail based on the candidate tag vocabulary. The objects include people, animals, and plants. The description should be no less than 300 words."

[0083] Exemplarily, the text prompt can be pre-constructed and saved locally, and can be directly called when needed; of course, it can also be generated on the spot when needed, which is not limited here.

[0084] Exemplarily, the text prompt may be dynamically changed and may be adjusted according to actual needs.

[0085] In this embodiment, the basic classification model is guided by text prompts to describe the overall and detailed information of the image sample, which can obtain more accurate image description information, thereby helping the subsequent steps to determine a more accurate second multi-label classification loss, thereby helping to improve the accuracy of multi-label classification.

[0086] In the aforementioned embodiments, methods related to multi-label classification model training are provided. The following is an explanation of a method for multi-label classification of images using a multi-label classification model. In an exemplary embodiment, a method for multi-label classification of images is provided. The method can be applied to a server, such as a cloud disk server. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The method may include: extracting features of the image to be classified to obtain image features; fusing the image features with the label text features obtained based on a preset label vocabulary to obtain fused features; inputting the fused features into a pre-trained image multi-label classification model to obtain a classification result of the image to be classified relative to multiple labels; the multiple labels are labels in the label vocabulary; the image multi-label classification model is trained based on the multi-label classification model training method in the aforementioned embodiment.

[0087] Exemplarily, label text features can be pre-extracted. For example, the label text features extracted when training a multi-label classification model are backed up and saved offline. When the trained multi-label classification model is used to perform multi-label classification of images, the saved label text features can be directly obtained without the need to extract features from the label vocabulary, thereby improving classification efficiency and saving computing resources.

[0088] Exemplarily, the training of the image multi-label classification model can be continuous. For example, while using the image multi-label classification model to perform multi-label image classification, the actual classification effect of the model can be evaluated, and the model can be trained and optimized again based on this.

[0089] In this embodiment, image features are obtained by extracting features of the image to be classified; the image features and label text features are fused to obtain fused features; the fused features are analyzed using the image multi-label classification model obtained by the multi-label classification model training method provided in the aforementioned embodiment, which can obtain more accurate classification results.

[0090] In an exemplary embodiment, the traditional CV (computer vision) multi-label classification model is limited in the number of multi-label vocabulary types due to the recognition ability of the model itself, and has insufficient recognition ability in long-tail labels (such as Scottish Fold cat, Tianshan Snow Lotus, glutinous rice cake) and abstract labels (buildings, scenery, stacked objects). In addition, the training method of the existing model often cannot learn the implicit relationship between multiple labels, such as label A depends on labels B and C. This implicit relationship usually requires adding logical rules or setting specific presets in the data post-processing link. Over-design and the introduction of additional modules also limit the application of this type of model and make it impossible to achieve end-to-end effective learning. Based on the above shortcomings, this embodiment proposes an image multi-label classification method based on a multimodal model and a corresponding model training method to solve the shortcomings of traditional CV models in complex multi-class, abstract, and long-tail multi-label tasks.

[0091] like Figure 3 As shown, the method may correspond to a specific system architecture, that is, it mainly includes:

[0092] 1. Image encoder: Use models such as VIT or Swin-transformer to extract image features, and the extracted image features are f(image), that is, image sample features.

[0093] 2. Text encoder: The preset tag vocabulary [tag1, tag2, .. tagn] is subjected to text feature extraction by a text encoder (such as BERT) to form a text feature extraction matrix matrix (V), i.e., the tag text feature.

[0094] 3. Fusion module: The text feature extraction matrix matrix(V) is used as the query, combined with the image feature image sample feature f(image), and the image and label features are fused through the cross-attention module. The output is F (fusion), that is, the fused sample feature.

[0095] 4. Large Language Model (LLM): Input F (fusion) to the embedding layer of the large language model. The large language model can be selected from the Llama3 or Qwen2 series models. The large language model is the preset basic classification model.

[0096] 5. Multi-label output: The final feature output layer of the large language model passes through the fully connected layer (fc) to output the probability value corresponding to each label, that is, to generate the classification probability information of the image sample belonging to each label in the label vocabulary.

[0097] 6. Training auxiliary output: The large language model directly outputs the text description of the image, that is, the image description information of the image sample, as a multi-label auxiliary training task.

[0098] like Figure 4 As shown, the implementation process of the image multi-label classification method in this embodiment is given, which specifically includes steps 1 to 8:

[0099] Step 1: Prepare the image multi-label vocabulary V(tag1, tag2, ..tagn), that is, the tag vocabulary.

[0100] Step 2: Prepare training data: Prepare a picture and text dataset, and perform annotation extraction based on the word segmentation tool and the label vocabulary V.

[0101] Step 3: Based on the multi-label vocabulary V, the label features are pre-extracted through the text encoder to form a label feature matrix matrix(V), that is, the label text features.

[0102] Step 4: The training image, i.e., the image sample, is subjected to feature extraction through the image encoder. The matrix (V) is used as the query and fused with the image features through the cross attention module to output the fused sample feature F (fusion), i.e., the fused sample feature.

[0103] For example, only one image is used for each training. Based on the cross-attention module, the entire label library is matched for feature fusion. The features in the label library with higher semantic similarity to the image will be activated more effectively, which is convenient for the final multi-label training.

[0104] Step 5: Connect the fused sample features to the embedding layer of a large language model (such as Llama3 / Qwen2) and input a text prompt, such as "Please describe this image based on the candidate tag vocabulary". For example, you can use the existing large model to generate a suitable prompt (prompt word, text prompt). For example, if you are an expert in generating image description prompts, please recommend generating 5 sets of prompts for image description, including the following requirements: no hallucinations in the image; the content of the image description is rich enough, not less than 10 Chinese characters; the generated image description should include the overall description of the image, as well as detailed information such as the attributes of the objects that appear and the relationship between the objects.

[0105] Step 6: The final feature output layer of the large language model passes through the fully connected layer, outputs the probability of each label, and trains with ASL loss as the main training task. ASL is an asymmetric loss function, and ASL loss is the loss determined based on the asymmetric loss function.

[0106] Step 7: Use the cross entropy loss function to assist in training the image text description output by the large language model. For example, the image description task here actually contains the intrinsic relationship between the labels of the objects appearing in the image. It can be considered that the generated multi-label information can be parsed by generating image description content. Compared with the previous method of only generating isolated labels for images, it is more conducive to multi-label task learning. Among them, the total loss of model training can be equal to the sum of ASL loss and cross entropy loss.

[0107] Step 8: The trained model performs model inference, passes through the fully connected layer, and outputs the image multi-label inference result. That is, the trained model is used to perform image multi-label classification.

[0108] For example, for long-tail class labels, the semantic description of scarce labels can be supplemented in a targeted manner in the image description auxiliary task, and ASL loss can be used in the training task to alleviate the problem of label category imbalance. At the same time, based on the patent solution, the vocabulary can be expanded as needed to improve the richness of long-tail labels. For abstract class labels, traditional visual algorithms find it difficult to capture clear features. In the technical solution of the present application, the embedding of multimodal text feature vectors is added, and the text description can efficiently solve the feature representation of abstract class labels and the alignment problem with visual features.

[0109] The above-mentioned image multi-label classification method, based on the multi-modal model architecture, such as text and image, can customize the construction of a multi-label vocabulary as a query, and introduce a large language model to effectively learn the downstream tasks of multi-label images and texts. Combined with prompt engineering technology, it solves the problem of multi-label on-demand recognition and on-demand expansion on demand, adds an auxiliary branch for image text generation training, effectively learns the implicit relationship between multiple labels, and improves the recognition accuracy of long-tail and abstract class labels. Specifically, since the long-tail class is for some rare label samples, the semantic description of scarce labels can be supplemented in the image description auxiliary task, and the ASL loss is used in the training task to alleviate the problem of label category imbalance. At the same time, based on this patent solution, the vocabulary can be expanded on demand to improve the richness of long-tail labels; for abstract class labels, traditional visual algorithms are difficult to capture clear features, and in this patent solution, the embedding of multi-modal text feature vectors is added, and the text description can efficiently solve the feature representation of abstract class labels and the alignment problem with visual features.

[0110] This method first uses the custom multi-label vocabulary as a query after feature pre-processing by the text encoder, and then integrates it with the image-text multimodal fusion module and image sample features before inputting it into the large language model + text prompt for training. In addition, an auxiliary branch for image-text generation is introduced in the training phase to achieve implicit learning of image multi-labels in text description tasks. On the one hand, this method can effectively improve the recognition accuracy of long-tail labels and abstract labels, and on the other hand, it can expand the label vocabulary on demand, and combine the text prompts of the large language model to perform on-demand customization and incremental training of multiple labels, with good scalability. What is expanded is the "label vocabulary" in the figure. If you want to do incremental training of additional labels based on this architecture, you only need to add new labels to the label vocabulary for fine-tuning.

[0111] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0112] Based on the same inventive concept, the embodiment of the present application also provides a multi-label classification model training device for implementing the multi-label classification model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multi-label classification model training device embodiments provided below can refer to the limitations of the multi-label classification model training method above, and will not be repeated here.

[0113] In an exemplary embodiment, Figure 5 As shown, a multi-label classification model training device 500 is provided, comprising:

[0114] The feature module 501 is used to extract features from the image sample to obtain image sample features; fuse the image sample features with the label text features obtained based on a preset label vocabulary to obtain fused sample features;

[0115] An input module 502 is used to input the fused sample features into a preset basic classification model; the basic classification model is used to generate classification probability information of each label in the label vocabulary of the image sample and image description information of the image sample according to the fused sample features and output them;

[0116] The training module 503 is used to determine the first multi-label classification loss according to the classification probability information, and train the basic classification model based on the first multi-label classification loss; and determine the second multi-label classification loss according to the image description information; based on the second multi-label classification loss, assist in the training of the basic classification model; based on the training of the basic classification model, obtain the image multi-label classification model; the image multi-label classification model is used to perform multi-label classification on the image to be classified.

[0117] In one embodiment, the input module 502 is also used to determine the long-tail class label information; the long-tail class label information is input into the basic classification model; the basic classification model is also used to generate and output semantic description information for the long-tail class label based on the long-tail class label information; the training module 503 is also used to determine the second multi-label classification loss based on the image description information, including: determining the second multi-label classification loss based on the image description information and the semantic description information.

[0118] In one of the embodiments, the feature module 501 is also used to expand the tag vocabulary based on the long-tail tag information to increase the types and / or quantity of long-tail tags in the tag vocabulary; and fuse the image sample features with the tag text features obtained based on a preset tag vocabulary, including: fusing the image sample features with the tag text features obtained based on the expanded tag vocabulary.

[0119] In one embodiment, the training module 503 is used to determine the first multi-label classification loss according to the classification probability information, and train the basic classification model based on the first multi-label classification loss, including: determining the first multi-label classification loss according to the classification probability information and based on the asymmetric loss function, and training the basic classification model based on the first multi-label classification loss to increase the basic classification model's attention to long-tail class labels.

[0120] In one embodiment, the feature module 501 is also used to construct a text prompt, which includes an overall information description task and a detailed information description task for the image; the input module 502 is also used to input the fused sample features into a preset basic classification model, including: inputting the fused sample features and the text prompt into the basic classification model to guide the basic classification model to describe the image sample as a whole and in detail, and obtain image description information for the image sample.

[0121] Each module in the above-mentioned multi-label classification model training device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0122] In an exemplary embodiment, a multi-label image classification device is provided, comprising:

[0123] The feature fusion module is used to extract features of the image to be classified to obtain image features; the image features are fused with the label text features obtained based on a preset label vocabulary to obtain fused features;

[0124] A classification module is used to input the fusion features into a pre-trained image multi-label classification model to obtain the classification results of the image to be classified relative to multiple labels; the multiple labels are labels in the label vocabulary; the image multi-label classification model is trained based on the multi-label classification model training method in the first aspect mentioned above.

[0125] Each module in the above-mentioned image multi-label classification device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0126] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required for executing a multi-label classification model training method or an image multi-label classification method, for example, image samples or images to be classified. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-label classification model training method or an image multi-label classification method is implemented.

[0127] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0130] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0131] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-label classification model training method, characterized in that: The method comprises: Extract features from image samples to obtain image sample features; Fusing the image sample features with label text features obtained based on a preset label vocabulary to obtain fused sample features; Inputting the fused sample features into a preset basic classification model; the basic classification model is used to generate classification probability information of the image sample belonging to each label in the label vocabulary according to the fused sample features, as well as image description information of the image sample and output it; Determine a first multi-label classification loss according to the classification probability information, and train the basic classification model based on the first multi-label classification loss; And, determining a second multi-label classification loss according to the image description information; and assisting in training the basic classification model based on the second multi-label classification loss; Based on the training of the basic classification model, an image multi-label classification model is obtained; the image multi-label classification model is used to perform multi-label classification on the image to be classified.

2. The method according to claim 1, characterized in that The method further comprises: Determine long-tail label information; Inputting the long-tail class label information into the basic classification model; the basic classification model is also used to generate and output semantic description information for the long-tail class label according to the long-tail class label information; The determining, according to the image description information, a second multi-label classification loss includes: A second multi-label classification loss is determined according to the image description information and the semantic description information.

3. The method according to claim 2, characterized in that The method further comprises: According to the long-tail tag information, the tag vocabulary is expanded to increase the types and / or quantity of long-tail tags in the tag vocabulary; The fusing of the image sample features with the label text features obtained based on a preset label vocabulary includes: The image sample features and the label text features obtained based on the expanded label vocabulary are fused.

4. The method according to claim 2, characterized in that: The step of determining a first multi-label classification loss according to the classification probability information, and training the basic classification model based on the first multi-label classification loss includes: According to the classification probability information and based on an asymmetric loss function, the first multi-label classification loss is determined, and based on the first multi-label classification loss, the basic classification model is trained to increase the attention of the basic classification model to the long-tail class label.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Constructing a text prompt, wherein the text prompt includes a task of describing overall information and a task of describing detailed information of the image; The step of inputting the fused sample features into a preset basic classification model comprises: The fused sample features and the text prompt are input into the basic classification model to guide the basic classification model to describe the overall and detailed aspects of the image sample, thereby obtaining image description information of the image sample.

6. A method for multi-label image classification, characterized in that: The method comprises: Perform feature extraction on the image to be classified to obtain image features; Fusing the image features with label text features obtained based on a preset label vocabulary to obtain fused features; The fusion features are input into a pre-trained image multi-label classification model to obtain a classification result of the image to be classified relative to multiple labels; the multiple labels are labels in the label vocabulary; the image multi-label classification model is trained based on the method described in any one of claims 1 to 5.

7. A multi-label classification model training device, characterized in that: The device comprises: The feature module is used to extract features from image samples to obtain image sample features; fuse the image sample features with label text features obtained based on a preset label vocabulary to obtain fused sample features; An input module, used to input the fused sample features into a preset basic classification model; the basic classification model is used to generate classification probability information of the image sample belonging to each label in the label vocabulary according to the fused sample features, as well as image description information of the image sample and output it; A training module is used to determine a first multi-label classification loss according to the classification probability information, and train the basic classification model based on the first multi-label classification loss; and determine a second multi-label classification loss according to the image description information; based on the second multi-label classification loss, assist in the training of the basic classification model; based on the training of the basic classification model, an image multi-label classification model is obtained; the image multi-label classification model is used to perform multi-label classification on the image to be classified.

8. An image multi-label classification device, characterized in that: The device comprises: The feature fusion module is used to extract features of the image to be classified to obtain image features; the image features are fused with label text features obtained based on a preset label vocabulary to obtain fused features; A classification module is used to input the fusion features into a pre-trained image multi-label classification model to obtain the classification results of the image to be classified relative to multiple labels; the multiple labels are labels in the label vocabulary; the image multi-label classification model is trained based on the method described in any one of claims 1 to 5.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images