Description text generation-based missing multi-label image identification method and device

Abstract

The invention relates to a missing multi-label image recognition method and device based on description text generation, and the method comprises the steps: inputting a missing multi-label image into a text generation branch of an initial prediction model, and outputting a natural language description text; inputting the natural language description text into a label prediction branch of the initial prediction model, and outputting a prediction pseudo label; the prediction pseudo labels are used for supplementing labels of the missing multi-label image to obtain pseudo real labels, the pseudo real labels are used as training samples, parameters of the initial prediction model are optimized, and a target prediction model is obtained; and inputting the to-be-recognized image into the target prediction model for label prediction, and outputting the prediction label set of the to-be-recognized image, thereby solving the problem of poor image recognition performance due to lack of multiple labels, obtaining more potential semantic information by generating the natural language description for the image and analyzing the natural language description, and improving the recognition efficiency. Therefore, insufficiency of label information is made up, and stability and identification precision of the model are improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method and device for missing multi-label image recognition based on description text generation. Background Art

[0002] Image recognition technology has been widely used in many fields. Among them, multi-label image recognition, as one of the core tasks of computer vision, aims to simultaneously identify multiple objects in an image. Traditional multi-label image recognition methods mostly rely on complete labeling of image labels and supervised learning to train models. During the training process, these models usually extract image features and perform image classification tasks in combination with the annotated labels. However, this traditional label-based method has some key problems. In particular, when labels are missing or incomplete, the recognition performance will be significantly reduced. For multi-label image recognition tasks, the lack of labels means that it is impossible to provide sufficient supervisory signals for certain objects or features in the image, which seriously affects the recognition accuracy and generalization ability of the system.

[0003] Currently, some studies have attempted to address the issue of missing training set labels through strategies such as semi-supervised learning and transfer learning, or to improve model robustness by employing data augmentation techniques. However, when training sets are incompletely labeled, these methods still struggle to fully utilize the potential information contained in the training set images, resulting in insufficient stability and recognition accuracy of the trained models, and poor multi-label prediction performance.

[0004] There is currently no effective solution to the problem of poor performance in missing multi-label image recognition in related technologies. Summary of the Invention

[0005] This embodiment provides a missing multi-label image recognition method and apparatus based on description text generation to solve the problem of poor missing multi-label image recognition performance in related technologies.

[0006] In a first aspect, this embodiment provides a missing multi-label image recognition method based on description text generation, the method comprising:

[0007] Acquire an initial prediction model including a text generation branch and a label prediction branch; input the missing multi-label image into the text generation branch to output a natural language description text; input the natural language description text into the label prediction branch to output a predicted pseudo-label; the predicted pseudo-label is used to complete the label of the missing multi-label image to obtain a pseudo-true label; use the pseudo-true label as a training sample to optimize the parameters of the initial prediction model to obtain a target prediction model;

[0008] The image to be identified is input into the target prediction model for label prediction, and a predicted label set of the image to be identified is output.

[0009] In some of the embodiments, the text generation branch includes: a visual encoder and a multimodal generation model;

[0010] The visual encoder is used to extract global features and local region features of the missing multi-label image to obtain a multi-level visual representation;

[0011] The multimodal generation model is used to generate natural language description text including object, attribute and scene information based on the multi-level visual representation.

[0012] In some embodiments, the method further comprises:

[0013] The visual encoder and the multimodal generative model are jointly optimized through contrastive learning to align the semantic content of the natural language description text with the visual content of the multi-level visual representation.

[0014] In some of the embodiments, the tag prediction branch includes: a semantic parser and a multimodal matching model;

[0015] The semantic parser is used to parse the natural language description text to obtain a candidate tag set;

[0016] The multimodal matching model is used to verify and screen labels in the candidate label set based on local region features of the missing multi-label image to obtain predicted pseudo labels.

[0017] In some embodiments, parsing the natural language description text to obtain a candidate tag set includes:

[0018] Identify various entities in the natural language description text to obtain a set of text entities;

[0019] The set of text entities is matched with a label dictionary to obtain a candidate label set; wherein the label dictionary is obtained based on the current task type.

[0020] In some embodiments, matching the set of text entities with a tag dictionary to obtain a candidate tag set includes:

[0021] Calculating the semantic similarity between each tag item in the tag dictionary and each text entity; determining an initial tag set from the tag dictionary based on the semantic similarity;

[0022] The initial label set is input into a graph convolutional network model for correlation judgment, and a candidate label set is output; wherein the adjacency matrix of the graph convolutional network model is obtained based on binary label co-occurrence information.

[0023] In some embodiments, verifying and screening labels in the candidate label set based on local region features of the missing multi-label image to obtain predicted pseudo labels includes:

[0024] Mapping the local region features of the missing multi-label image and the labels in the candidate label set into a unified semantic space to obtain image region representation and candidate label representation;

[0025] Calculating a similarity between the image region representation and the candidate label representation to generate a confidence score;

[0026] A filtering threshold is automatically adjusted based on the distribution of the confidence scores, and the predicted pseudo-label set is screened out based on the adjusted filtering threshold.

[0027] In some embodiments, the initial prediction model is an end-to-end model, which calculates a text generation loss function based on the natural language description text, a label prediction loss function based on the candidate label set, and a cross-modal alignment loss function based on the predicted pseudo-labels; in the process of optimizing the parameters of the initial prediction model using the pseudo-true labels as training samples, the text generation loss function, the label prediction loss function, and the cross-modal alignment loss function are jointly optimized.

[0028] In some embodiments, during the training of the initial prediction model, the weight coefficients of each loss function are adjusted based on the performance of the validation set.

[0029] In some embodiments, the text generation branch adopts an alternating training strategy; the alternating training strategy includes:

[0030] Fixing the parameters of the visual encoder and preliminarily optimizing the parameters of the multimodal generative model;

[0031] Jointly fine-tune the parameters of the visual encoder and the preliminarily optimized multimodal generative model.

[0032] In a second aspect, this embodiment provides a missing multi-label image recognition device based on description text generation, the device comprising:

[0033] A model training module is configured to obtain an initial prediction model including a text generation branch and a label prediction branch; input a missing multi-label image into the text generation branch to output a natural language description text; input the natural language description text into the label prediction branch to output a predicted pseudo-label; based on the predicted pseudo-label, the label of the missing multi-label image is used to obtain a pseudo-true label; and based on the pseudo-true label, the parameters of the initial prediction model are optimized to obtain a target prediction model;

[0034] The model application module is used to input the image to be identified into the target prediction model for label prediction and output the predicted label set of the image to be identified.

[0035] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the missing multi-label image recognition method based on description text generation described in the first aspect is implemented.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the missing multi-label image recognition method based on description text generation as described in the first aspect.

[0037] Compared with the related art, the missing multi-label image recognition method and device based on description text generation provided in this embodiment obtains an initial prediction model including a text generation branch and a label prediction branch; inputs the missing multi-label image into the text generation branch, and outputs a natural language description text; inputs the natural language description text into the label prediction branch, and outputs a predicted pseudo-label; the predicted pseudo-label is used to complete the label of the missing multi-label image to obtain a pseudo-real label, and the pseudo-real label is used as a training sample to optimize the parameters of the initial prediction model to obtain a target prediction model; the image to be identified is input into the target prediction model for label prediction, and the predicted label set of the image to be identified is output, which solves the problem of poor performance in missing multi-label image recognition. By generating a natural language description for the image and analyzing the natural language description, more potential semantic information is obtained, thereby making up for the lack of label information and improving the stability and recognition accuracy of the model.

[0038] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 This is a hardware structure block diagram of a terminal for a method for identifying missing multi-label images based on description text generation in an embodiment of the present application;

[0041] Figure 2 Schematic diagram of the process of the missing multi-label image recognition method based on description text generation in an embodiment of the present application;

[0042] Figure 3 This is a schematic diagram of the training process of the initial prediction model in the preferred embodiment of the present application;

[0043] Figure 4 This is a schematic diagram of the application process of the target prediction model in the preferred embodiment of the present application;

[0044] Figure 5 Schematic diagram of the forward propagation process of the initial prediction model in this preferred embodiment;

[0045] Figure 6 This is a structural block diagram of a missing multi-label image recognition device based on description text generation in an embodiment of the present application.

[0046] Figure numerals: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 61, model training module; 62, model application module. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 : is a hardware structure block diagram of a terminal of the missing multi-label image recognition method based on description text generation in this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 The processor 102 (only one is shown) and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input / output device 108. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0050] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the missing multi-label image recognition method based on descriptive text generation in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] Transmission device 106 is used to receive or transmit data via a network. This network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0052] In this embodiment, a missing multi-label image recognition method based on description text generation is provided. Figure 2 is a flowchart of the missing multi-label image recognition method based on description text generation in this embodiment. Figure 2 As shown, the process includes the following steps:

[0053] Step S201, obtain an initial prediction model including a text generation branch and a label prediction branch; input the missing multi-label image into the text generation branch, and output the natural language description text; input the natural language description text into the label prediction branch, and output the predicted pseudo-label; the predicted pseudo-label is used to fill the label of the missing multi-label image to obtain a pseudo-true label; use the pseudo-true label as a training sample, optimize the parameters of the initial prediction model, and obtain the target prediction model.

[0054] Specifically, the text generation branch is used to generate natural language description text for image data. This can be implemented based on an encoder-decoder architecture model, where the encoder uses a residual network (ResNet) or a visual transformer (ViT), and the decoder uses a multimodal model (BLIP) or a recurrent neural network. The label prediction branch is used to extract label information from the natural language description text. This branch can use a pretrained language model, such as the Bidirectional Encoder Representations from Transformers (BERT). An image-text matching mechanism can be introduced after the pretrained language model to achieve self-calibration of the prediction results, improving system robustness.

[0055] Step S202: input the image to be identified into the target prediction model for label prediction, and output a predicted label set of the image to be identified.

[0056] Specifically, the target prediction model processes the image to be identified by: generating predicted text for the image to be identified; extracting label information from the predicted text to obtain a candidate label set; and performing calibration and denoising on the candidate label set to obtain a predicted label set.

[0057] Taking the wildlife monitoring scenario as an example, the implementation process of step S202 is as follows: (1) Description generation stage: Input a field image containing multiple animals, and the trained text generation branch outputs the predicted text: "There are three zebras drinking water on the left side of the image, a cheetah hiding in the grass in the middle, and three antelopes running in the distance, with the background being the African grassland at dusk." (2) Label prediction stage: The trained label prediction branch identifies objects such as "zebra", "cheetah", and "antelope" from the predicted text, behavioral attributes such as "drinking water", "hiding", and "running", and other entity information, thereby obtaining a candidate label set related to these entity information {zebra, cheetah, antelope, ..., grassland, dusk}. (3) Cross-modal verification stage: Calculate the similarity between the text embedding of each label and the visual features of the central area of ​​the image. For example, if it is confirmed that there are feline features in the area, the label "cheetah" is retained. Filter out mislabeled labels that lack visual evidence, such as "trees". (4) Final output: Output a high-confidence multi-label set, that is, the predicted label set {zebra (0.98), cheetah (0.95), antelope (0.92), drinking water (0.87), grassland (0.93), dusk (0.89)}.

[0058] In this embodiment, the problem of poor recognition performance of missing multi-label images is solved by inputting the missing multi-label image into the text generation branch of the initial prediction model to output a natural language description text; inputting the natural language description text into the label prediction branch of the initial prediction model to output a predicted pseudo-label; predicting the pseudo-label to complete the label of the missing multi-label image to obtain a pseudo-true label, using the pseudo-true label as a training sample to optimize the parameters of the initial prediction model to obtain a target prediction model; inputting the image to be identified into the target prediction model for label prediction, and outputting a predicted label set for the image to be identified. This embodiment generates a natural language description for the image and analyzes the natural language description to obtain more potential semantic information, thereby compensating for the lack of label information and improving the stability and recognition accuracy of the model.

[0059] In some of these embodiments, see Figure 3 , text generation branch, including: visual encoder and multimodal generation model.

[0060] The visual encoder is used to extract global and local features from images lacking multi-labels, resulting in a multi-level visual representation. The multimodal generative model is used to generate natural language descriptions of objects, attributes, and scenes based on the multi-level visual representation.

[0061] Specifically, the visual encoder can use ViT or ResNet. The multimodal generative model can be based on the pre-trained model of unified visual language understanding and generation (Bootstrapping Language-Image Pre-training, BLIP). BLIP will be combined with the attention mechanism to generate natural language description text that covers the input image objects, attributes, and scene information.

[0062] In this embodiment, by generating fine-grained description text, rich semantic clues are provided for label prediction.

[0063] In some of these embodiments, see Figure 3 , the method also includes: jointly optimizing the visual encoder and the multimodal generation model through contrastive learning to align the semantic content of the natural language description text and the visual content of the multi-level visual representation.

[0064] Specifically, contrastive learning is a model training strategy that constructs positive pairs (images and their matching descriptions) and negative pairs (images and mismatched descriptions) to enable the model to learn to distinguish semantically aligned image-text pairs. After training, contrastive learning enables the model to directly generate descriptions that are highly aligned with the image content, such as accurately capturing details like "zebra drinking water on the left" and "grassland background at dusk."

[0065] In this embodiment, the semantic coverage of the generated text is adjusted through contrastive learning to ensure that the description content is highly aligned with the visual content of the image.

[0066] In some of these embodiments, see Figure 3 , label prediction branch, including: semantic parser and multimodal matching model.

[0067] The semantic parser parses natural language descriptions to generate candidate tag sets. Specifically, the semantic parser uses the pre-trained BERT language model to perform named entity recognition and relation extraction on the description text. It identifies at least one of the following entity information items: object entities (e.g., zebra, cheetah, antelope, grassland), location entities (e.g., left, in the grass, far away, background), action entities (e.g., drinking water, hiding, running), scene entities (e.g., dusk, African grassland), and attribute entities (e.g., three (quantity), stripes (appearance)). The candidate tag set is constructed based on this entity information.

[0068] The multimodal matching model is used to verify and filter labels in the candidate label set based on local region features of the missing multi-label image, and obtain predicted pseudo-labels. Specifically, the multimodal matching model can obtain local region features output by the visual encoder, or obtain and segment the missing multi-label image, and further obtain local region features based on the segmented slices.

[0069] In this embodiment, the synergistic effect of multimodal information is taken into consideration, and the confidence of the prediction results is further calibrated with the help of the image-text matching mechanism, thereby effectively improving the robustness and accuracy of the image recognition system.

[0070] In some embodiments, parsing the natural language description text to obtain a candidate tag set includes:

[0071] Step S310: Identify various entities in the natural language description text to obtain a set of text entities.

[0072] Specifically, pre-trained language models (such as BERT) are used to perform named entity recognition and relation extraction on description texts, identifying entity information such as object entities, location entities, action entities, scene entities, and attribute entities in natural language description texts. For example, if the natural language description text is "Three zebras are drinking water on the left side of the image, a cheetah is hiding in the grass in the middle, and three antelopes can be seen running in the distance, with the background being the African grasslands at dusk," the semantic parser will identify object entities such as "zebra," "cheetah," and "antelope," as well as action entities such as "drinking water," "hiding," and "running."

[0073] Step S320 , matching the set of text entities with a tag dictionary to obtain a candidate tag set; wherein the tag dictionary is obtained based on the current task type.

[0074] Specifically, for wildlife monitoring scenarios, the label dictionary can be a combination of one or more of the following: a wildlife dictionary, an animal species dictionary, a natural environment dictionary, and a population interaction dictionary. By matching the text entity set with the animal species dictionary and supplementing the "drinking water" behavior label missed by traditional methods, an initial candidate label set {zebra, cheetah, antelope, grassland, dusk, ..., drinking water} is generated.

[0075] In this embodiment, the label dictionary plays the role of a task-adaptive semantic filter, ensuring that the initial candidate label set has both standardized and low-noise effects, laying a high-quality foundation for subsequent probabilistic graph model optimization, and is a key link in improving model robustness.

[0076] In some embodiments, a set of text entities is matched with a tag dictionary to obtain a candidate tag set, including:

[0077] Step S321 , calculating the semantic similarity between each tag item in the tag dictionary and each text entity; and determining an initial tag set from the tag dictionary based on the semantic similarity.

[0078] Specifically, semantic similarity is calculated by the encoder network, such as cosine similarity, to represent semantic similarity. In one embodiment, a tag item whose cosine similarity with the text entity is greater than a threshold is extracted from the tag dictionary and added to the initial tag set as one of the tags.

[0079] Step S322: Input the initial tag set into the graph convolutional network model for correlation judgment, and output the candidate tag set; wherein the adjacency matrix of the graph convolutional network model is obtained based on the binary tag co-occurrence information.

[0080] Specifically, an adjacency matrix is ​​calculated based on binary tag co-occurrence information, and this matrix is ​​used to construct a graph convolutional network model. This binary tag co-occurrence information is calculated as the number of times different tags appear together in the same instance in the training set. The graph convolutional network model then scores the relevance of candidate tags in the initial tag set and outputs a probabilistic prediction result. This probabilistic prediction result is used to determine whether the tags in the initial tag set meet the required relevance to the text entities identified in the natural language description, thereby enabling further denoising.

[0081] In some embodiments, verifying and screening labels in a candidate label set based on local region features of a missing multi-label image to obtain a predicted pseudo label includes:

[0082] Step S410 : Mapping the local region features of the missing multi-label image and the labels in the candidate label set into a unified semantic space to obtain image region representation and candidate label representation.

[0083] Specifically, we first construct a joint embedding space. We can use the visual-text dual-tower encoding model to map the local region features output by the visual encoder and the label text in the candidate label set to a unified semantic space. Here, contrast loss is used to optimize feature alignment.

[0084] Step S420 : Calculate the similarity between the image region representation and the candidate tag representation to generate a confidence score.

[0085] Specifically, for each tag in the candidate tag set, a category prompt is constructed through prompt word engineering (such as "This is a photo of [Class]", where [Class] is the label text). The multimodal matching model (Contrastive Language-Image Pre-Training, CLIP) is used to calculate the local similarity between the image region representation of the category prompt and the candidate tag representation to generate a confidence score.

[0086] Step S430 : Automatically adjust the filtering threshold based on the distribution of the confidence scores, and filter out the predicted pseudo label set based on the adjusted filtering threshold.

[0087] For details, see Figure 3 During the training phase, the predicted pseudo-labels are merged with the original labels of the missing multi-label image samples (i.e., the missing true labels) to construct pseudo-true labels for fitting. The pseudo-true labels are used as training samples and then fitted with the semantic parser's prediction results to optimize the semantic parser parameters and improve the accuracy of the semantic parser in generating candidate label sets. The semantic parser's prediction results can be label prediction values ​​derived from the candidate label set or label prediction values ​​derived from the initial label set.

[0088] In the forecasting phase, see Figure 4 , a fully trained prediction model is used to extract the global features and local area features of the image to be identified, and obtain a multi-level visual representation. Based on the multi-level visual representation, a predicted text including object, attribute and scene information is generated; entity text is extracted from the predicted text, and the entity text is matched with the label dictionary of the current classification task to obtain a candidate label set; based on the local area features of the image to be identified, the labels in the candidate label set are verified and filtered to obtain a pseudo label, which will be directly used as the predicted label of the image to be identified.

[0089] In this embodiment, cross-modal information between images and labels is effectively fused to improve the accuracy of prediction.

[0090] In some embodiments, the initial prediction model is an end-to-end model, which calculates the text generation loss function based on the natural language description text, calculates the label prediction loss function based on the candidate label set, and calculates the cross-modal alignment loss function based on the predicted pseudo-labels; in the process of optimizing the parameters of the initial prediction model using pseudo-true labels as training samples, the text generation loss function, the label prediction loss function and the cross-modal alignment loss function are jointly optimized.

[0091] Specifically, the text generation loss function adopts the cross entropy loss function, the label prediction loss function adopts the binary cross entropy loss function, and the cross-modal alignment loss function adopts the contrast loss function.

[0092] In this embodiment, multiple loss functions jointly guide parameter updates during backpropagation, forming a unified learning process for the entire model from input to output. The model simultaneously focuses on multiple objectives (such as text generation, label prediction, and cross-modal alignment), thus avoiding bias or overfitting that may result from optimizing a single task. This approach helps the model better understand the multi-level semantic information of the input data.

[0093] In some embodiments, during the training of the initial prediction model, the weight coefficients of each loss function are adjusted based on the performance of the validation set.

[0094] In this embodiment, the validation set enables the model to better adapt to unseen data rather than just optimizing for the training data, which helps reduce the risk of overfitting.

[0095] In some embodiments, the text generation branch adopts an alternating training strategy; the alternating training strategy includes: fixing the parameters of the visual encoder, preliminarily optimizing the parameters of the multimodal generation model; and jointly fine-tuning the parameters of the visual encoder and the preliminarily optimized multimodal generation model.

[0096] In this embodiment, the alternating training strategy can improve the stability and efficiency of the training process and promote the quality of parameter optimization.

[0097] The present embodiment is described and illustrated below through a preferred embodiment. The missing multi-label image recognition method based on description text generation includes:

[0098] Step 1: Build an initial prediction model consisting of a text generation branch and a label prediction branch. The text generation branch includes a visual encoder and a multimodal generation model; the label prediction branch includes a semantic parser and a multimodal matching model.

[0099] Step 2: Input the missing multi-label image into the text generation branch and output the natural language description text; input the natural language description text into the label prediction branch and output the predicted pseudo-label; the predicted pseudo-label is used to fill the label of the missing multi-label image and obtain the pseudo-true label; use the pseudo-true label as the training sample to optimize the parameters of the initial prediction model and obtain the target prediction model.

[0100] Figure 5 This is a schematic diagram of the forward propagation process of the initial prediction model in this preferred embodiment, see Figure 3 and Figure 5 , the training process of the initial prediction model in the above step 2 includes:

[0101] S1. Image-based natural language description text generation: The text generation branch generates natural language description text for the input missing multi-label image, which serves as the text semantic basis for subsequent label prediction. The specific implementation includes:

[0102] (1.1) Visual feature extraction: A pre-trained visual encoder (such as ViT or ResNet) is used to extract global features and local region features of missing multi-label images to form a multi-level visual representation.

[0103] (1.2) Text generation and decoding: Utilize the multimodal generative model (BLIP) combined with the attention mechanism to generate natural language description text covering the object, attribute, and scene information of the input image for the input visual representation.

[0104] S2. Text-based label prediction: The semantic parser of the label prediction branch parses potential labels from the generated natural language description text to solve the problem of insufficient supervision signals caused by the lack of manual annotation. The specific implementation includes:

[0105] (2.1) Semantic parsing: Use a pre-trained language model (such as BERT) to perform named entity recognition and relation extraction on natural language description text to identify entities and their attributes in the text.

[0106] (2.2) Label candidate generation: Generate an initial label set by calculating the semantic similarity (such as cosine similarity) between the entity and its attributes and each label item in the pre-stored label dictionary.

[0107] (2.3) Probabilistic prediction: Based on the graph convolutional network model, the initial candidate labels in the initial label set are scored for relevance to obtain the candidate label set; the adjacency matrix of the graph convolutional network model is obtained based on the co-occurrence information of binary labels.

[0108] S3. Cross-modal verification: The image-text matching mechanism of the multimodal matching model in the label prediction branch is used to achieve self-calibration of the prediction results and improve the robustness of the system. Specific implementations include:

[0109] (3.1) Joint embedding space construction: The visual-textual dual-tower encoding model of the multimodal matching model is used to map the image region features obtained in step (1.1) and the candidate labels obtained in step (2.3) into a unified semantic space, and contrastive loss is used to optimize feature alignment.

[0110] (3.2) Similarity calculation: For each candidate label, a category prompt is constructed through prompt word engineering (e.g., “This is a photo of [Class]”, where [Class] is the label text). The multimodal matching model is used to calculate the local similarity between the text representation of the category prompt and the image region features to generate a confidence score.

[0111] (3.3) Dynamic Threshold Filtering: The multimodal matching model automatically adjusts the filtering threshold based on the confidence distribution, retaining high-confidence predictions to remove noisy labels and generate the final pseudo-labels. During the model training phase, the pseudo-labels are merged with the original missing labels of the samples to construct the pseudo-true labels for fitting.

[0112] Among them, a dual-branch collaborative training strategy is adopted for the initial prediction model, and the dual-branch collaborative training strategy specifically includes:

[0113] (1) Loss function design: Jointly optimize the text generation loss (based on the cross entropy calculated in step (1.2)), the label prediction loss (based on the binary cross entropy calculated in step (2.3)), and the cross-modal alignment loss (based on the contrastive loss calculated in step (3.1)).

[0114] (2) End-to-end training: An alternating training strategy is adopted. The visual encoder is first fixed to train the text multimodal generation model, and then all parameters of the visual encoder and the multimodal generation model are jointly fine-tuned.

[0115] (3) Dynamic weight adjustment: Automatically adjust the weight coefficients of different loss items in (4.1) based on the performance of each module in the validation set.

[0116] This preferred embodiment addresses the recognition accuracy issues inherent in traditional methods due to missing labels. By using descriptive text generated by a large multimodal model, the lack of label information is mitigated, improving image recognition accuracy. Furthermore, the introduction of a cross-modal similarity verification mechanism effectively enhances the robustness of the model and ensures the credibility of the prediction results. The end-to-end training approach balances efficiency and lightweightness, making this method suitable for large-scale datasets, with strong application value and good scalability.

[0117] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0118] In this embodiment, a missing multi-label image recognition device based on description text generation is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the details that have been explained will not be repeated here. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0119] Figure 6 This is a structural block diagram of the missing multi-label image recognition device based on description text generation in this embodiment. Figure 6 As shown, the device includes: a model training module 61 and a model application module 62.

[0120] The model training module 61 is used to obtain an initial prediction model including a text generation branch and a label prediction branch; input the missing multi-label image into the text generation branch to output a natural language description text; input the natural language description text into the label prediction branch to output a predicted pseudo-label; based on the predicted pseudo-label, the label of the missing multi-label image is used to obtain a pseudo-true label, and the parameters of the initial prediction model are optimized based on the pseudo-true label to obtain a target prediction model.

[0121] The model application module 62 is used to input the image to be identified into the target prediction model for label prediction, and output the predicted label set of the image to be identified.

[0122] In some embodiments, the text generation branch includes: a visual encoder and a multimodal generation model; the visual encoder is used to extract global features and local region features of missing multi-label images to obtain a multi-level visual representation; the multimodal generation model is used to generate natural language description text including object, attribute and scene information based on the multi-level visual representation.

[0123] In some embodiments, the method further comprises: jointly optimizing the visual encoder and the multimodal generative model through contrastive learning to align the semantic content of the natural language description text with the visual content of the multi-level visual representation.

[0124] In some embodiments, the label prediction branch includes: a semantic parser and a multimodal matching model; the semantic parser is used to parse the natural language description text to obtain a candidate label set; the multimodal matching model is used to verify and filter the labels in the candidate label set based on the local area features of the missing multi-label image to obtain predicted pseudo labels.

[0125] In some embodiments, natural language description text is parsed to obtain a candidate tag set, including: identifying various entities in the natural language description text to obtain a set of text entities; matching the set of text entities with a tag dictionary to obtain a candidate tag set; wherein the tag dictionary is obtained based on the current task type.

[0126] In some embodiments, a set of text entities is matched with a tag dictionary to obtain a candidate tag set, including: calculating the semantic similarity between each tag item in the tag dictionary and each text entity; determining an initial tag set from the tag dictionary based on the semantic similarity; inputting the initial tag set into a graph convolutional network model for relevance judgment, and outputting a candidate tag set; wherein the adjacency matrix of the graph convolutional network model is obtained based on binary tag co-occurrence information.

[0127] In some embodiments, the labels in the candidate label set are verified and filtered based on the local region features of the missing multi-label image to obtain predicted pseudo labels, including: mapping the local region features of the missing multi-label image and the labels in the candidate label set to a unified semantic space to obtain image region representation and candidate label representation; calculating the similarity between the image region representation and the candidate label representation to generate a confidence score; automatically adjusting the filtering threshold based on the distribution of the confidence scores, and filtering out the predicted pseudo label set based on the adjusted filtering threshold.

[0128] In some embodiments, the initial prediction model is an end-to-end model, which calculates the text generation loss function based on the natural language description text, calculates the label prediction loss function based on the candidate label set, and calculates the cross-modal alignment loss function based on the predicted pseudo-labels; in the process of optimizing the parameters of the initial prediction model using pseudo-true labels as training samples, the text generation loss function, the label prediction loss function and the cross-modal alignment loss function are jointly optimized.

[0129] In some embodiments, during the training of the initial prediction model, the weight coefficients of each loss function are adjusted based on the performance of the validation set.

[0130] In some embodiments, the text generation branch adopts an alternating training strategy; the alternating training strategy includes: fixing the parameters of the visual encoder, preliminarily optimizing the parameters of the multimodal generation model; and jointly fine-tuning the parameters of the visual encoder and the preliminarily optimized multimodal generation model.

[0131] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0132] This embodiment further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0133] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0134] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0135] In addition, in conjunction with the missing multi-label image recognition method based on description text generation provided in the above embodiments, this embodiment may also provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the missing multi-label image recognition methods based on description text generation in the above embodiments.

[0136] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0137] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0138] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0139] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A missing multi-label image recognition method based on description text generation, characterized in that: The method comprises: Acquire an initial prediction model including a text generation branch and a label prediction branch; input the missing multi-label image into the text generation branch to output a natural language description text; input the natural language description text into the label prediction branch to output a predicted pseudo-label; the predicted pseudo-label is used to complete the label of the missing multi-label image to obtain a pseudo-true label; use the pseudo-true label as a training sample to optimize the parameters of the initial prediction model to obtain a target prediction model; The image to be identified is input into the target prediction model for label prediction, and a predicted label set of the image to be identified is output.

2. The missing multi-label image recognition method based on description text generation according to claim 1 is characterized in that The text generation branch includes: a visual encoder and a multimodal generation model; The visual encoder is used to extract global features and local region features of the missing multi-label image to obtain a multi-level visual representation; The multimodal generation model is used to generate natural language description text including object, attribute and scene information based on the multi-level visual representation.

3. The missing multi-label image recognition method based on description text generation according to claim 2 is characterized in that: The method further comprises: The visual encoder and the multimodal generative model are jointly optimized through contrastive learning to align the semantic content of the natural language description text with the visual content of the multi-level visual representation.

4. The missing multi-label image recognition method based on description text generation according to claim 1, characterized in that: The label prediction branch includes: a semantic parser and a multimodal matching model; The semantic parser is used to parse the natural language description text to obtain a candidate tag set; The multimodal matching model is used to verify and screen labels in the candidate label set based on local region features of the missing multi-label image to obtain predicted pseudo labels.

5. The missing multi-label image recognition method based on description text generation according to claim 4 is characterized in that: The natural language description text is parsed to obtain a candidate tag set, including: Identify various entities in the natural language description text to obtain a set of text entities; The set of text entities is matched with a label dictionary to obtain a candidate label set; wherein the label dictionary is obtained based on the current task type.

6. The missing multi-label image recognition method based on description text generation according to claim 5, characterized in that: The set of text entities is matched with the label dictionary to obtain a candidate label set, including: Calculating the semantic similarity between each tag item in the tag dictionary and each text entity; determining an initial tag set from the tag dictionary based on the semantic similarity; The initial label set is input into a graph convolutional network model for correlation judgment, and a candidate label set is output; wherein the adjacency matrix of the graph convolutional network model is obtained based on binary label co-occurrence information.

7. The missing multi-label image recognition method based on description text generation according to claim 4 is characterized in that: Verifying and screening labels in the candidate label set based on local region features of the missing multi-label image to obtain predicted pseudo labels includes: Mapping the local region features of the missing multi-label image and the labels in the candidate label set into a unified semantic space to obtain image region representation and candidate label representation; Calculating a similarity between the image region representation and the candidate label representation to generate a confidence score; A filtering threshold is automatically adjusted based on the distribution of the confidence scores, and the predicted pseudo-label set is screened out based on the adjusted filtering threshold.

8. The missing multi-label image recognition method based on description text generation according to claim 4 is characterized in that: The initial prediction model is an end-to-end model, which calculates a text generation loss function based on the natural language description text, calculates a label prediction loss function based on a candidate label set, and calculates a cross-modal alignment loss function based on the predicted pseudo-labels; in the process of optimizing the parameters of the initial prediction model using the pseudo-true labels as training samples, the text generation loss function, the label prediction loss function, and the cross-modal alignment loss function are jointly optimized.

9. The missing multi-label image recognition method based on description text generation according to claim 8, characterized in that: During the training of the initial prediction model, the weight coefficients of each loss function are adjusted based on the performance of the validation set.

10. The missing multi-label image recognition method based on description text generation according to claim 2, characterized in that: The text generation branch adopts an alternating training strategy; the alternating training strategy includes: Fixing the parameters of the visual encoder and preliminarily optimizing the parameters of the multimodal generative model; Jointly fine-tune the parameters of the visual encoder and the preliminarily optimized multimodal generative model.

11. A missing multi-label image recognition device based on description text generation, characterized in that: The device comprises: A model training module is configured to obtain an initial prediction model including a text generation branch and a label prediction branch; input a missing multi-label image into the text generation branch to output a natural language description text; input the natural language description text into the label prediction branch to output a predicted pseudo-label; based on the predicted pseudo-label, the label of the missing multi-label image is used to obtain a pseudo-true label; and based on the pseudo-true label, the parameters of the initial prediction model are optimized to obtain a target prediction model; The model application module is used to input the image to be identified into the target prediction model for label prediction and output the predicted label set of the image to be identified.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

Images