Method and apparatus for generating missing multi-label image recognition based on description text

Abstract

The application relates to a missing multi-label image recognition method and device based on description text generation. The method comprises the following steps: inputting a missing multi-label image into a text generation branch of an initial prediction model to output a natural language description text; inputting the natural language description text into a label prediction branch of the initial prediction model to output a predicted pseudo label; the predicted pseudo label is used to fill in the labels of the missing multi-label image to obtain a pseudo real label; the pseudo real label is used as a training sample to optimize the parameters of the initial prediction model to obtain a target prediction model; and an image to be recognized is input into the target prediction model for label prediction to output a predicted label set of the image to be recognized, so that the problem of poor recognition performance of the missing multi-label image is solved. More potential semantic information is obtained by generating a natural language description for the image and analyzing the natural language description, so that the shortage of label information is made up, and the stability and recognition precision of the model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a missing multi-label image recognition method and device based on description text generation. BACKGROUND

[0002] Image recognition technology has been widely applied in many fields. Multi-label image recognition, as one of the core tasks of computer vision, aims to recognize multiple objects in an image simultaneously. In traditional multi-label image recognition methods, models are usually trained by relying on complete labeling and supervised learning of image labels. During the training process, these models usually extract image features and perform image classification tasks in combination with labeled labels. However, there are some key problems in this traditional label-based method, especially in the case of label missing or incompleteness, the recognition performance will decrease significantly. For the multi-label image recognition task, the missing of labels means that there is no enough supervision signal for some objects or features in the image, which seriously affects the recognition accuracy and generalization ability of the system.

[0003] Currently, some researches have tried to cope with the problem of label missing in the training set by semi-supervised learning, transfer learning and other strategies, or use data enhancement technology to improve the robustness of the model. However, in the case of incomplete labels in the training set, these methods still cannot fully utilize the potential information contained in the training set images, resulting in insufficient stability and recognition accuracy of the trained model, and poor multi-label prediction effect.

[0004] At present, there is no effective solution to the problem of poor performance of missing multi-label image recognition in related technologies. SUMMARY

[0005] A missing multi-label image recognition method and device based on description text generation are provided in the present embodiment to solve the problem of poor performance of missing multi-label image recognition in related technologies.

[0006] In a first aspect, a missing multi-label image recognition method based on description text generation is provided in the present embodiment, which comprises:

[0007] An initial prediction model including a text generation branch and a label prediction branch is obtained; a missing multi-label image is input into the text generation branch to output a natural language description text; the natural language description text is input into the label prediction branch to output a predicted pseudo label; the predicted pseudo label is used to fill in the labels of the missing multi-label image to obtain pseudo real labels; the pseudo real labels are used as training samples to optimize the parameters of the initial prediction model to obtain a target prediction model;

[0008] inputting the to-be-identified image into the target prediction model to perform label prediction, and outputting a predicted label set of the to-be-identified image.

[0009] In some embodiments, the text generation branch includes a visual encoder and a multi-modal generation model.

[0010] The visual encoder is configured to extract global features and local region features of the missing multi-label image to obtain a multi-level visual representation.

[0011] The multi-modal generation model is configured to generate natural language description text including object, attribute and scene information based on the multi-level visual representation.

[0012] In some embodiments, the method further includes:

[0013] The visual encoder and the multi-modal generation model are jointly optimized through contrast learning to align semantic content of the natural language description text and visual content of the multi-level visual representation.

[0014] In some embodiments, the label prediction branch includes a semantic parser and a multi-modal matching model.

[0015] The semantic parser is configured to parse the natural language description text to obtain a candidate label set.

[0016] The multi-modal matching model is configured to verify and filter labels in the candidate label set based on the local region features of the missing multi-label image to obtain predicted pseudo labels.

[0017] In some embodiments, parsing the natural language description text to obtain a candidate label set includes:

[0018] Identifying various entities in the natural language description text to obtain a set of text entities.

[0019] Matching the set of text entities with a label dictionary to obtain a candidate label set, wherein the label dictionary is obtained based on a current task type.

[0020] In some embodiments, matching the set of text entities with a label dictionary to obtain a candidate label set includes:

[0021] Calculating semantic similarity between each label item in the label dictionary and each text entity, and determining an initial label set from the label dictionary based on the semantic similarity.

[0022] inputting the initial label set into a graph convolution network model for correlation judgment, and outputting a candidate label set; wherein an adjacency matrix of the graph convolution network model is obtained based on binary label co-occurrence information.

[0023] In some embodiments, the local region features of the missing multi-label image are used to verify and filter the labels in the candidate label set, to obtain predicted pseudo labels, including:

[0024] The local region features of the missing multi-label image and the labels in the candidate label set are mapped to a unified semantic space, to obtain image region representation and candidate label representation.

[0025] The similarity between the image region representation and the candidate label representation is calculated to generate a confidence score.

[0026] The filter threshold is automatically adjusted based on the distribution of the confidence score, and the predicted pseudo label set is filtered based on the adjusted filter threshold.

[0027] In some embodiments, the initial prediction model is an end-to-end model, a text generation loss function is calculated based on the natural language description text, a label prediction loss function is calculated based on the candidate label set, and a cross-modal alignment loss function is calculated based on the predicted pseudo label; during the process of optimizing the parameters of the initial prediction model using the pseudo real label as a training sample, the text generation loss function, the label prediction loss function, and the cross-modal alignment loss function are jointly optimized.

[0028] In some embodiments, during the training of the initial prediction model, the weight coefficients of the loss functions are adjusted based on the performance of the verification set.

[0029] In some embodiments, the text generation branch adopts an alternating training strategy; the alternating training strategy includes:

[0030] The parameters of the visual encoder are fixed, and the parameters of the multi-modal generation model are preliminarily optimized.

[0031] The parameters of the visual encoder and the preliminarily optimized multi-modal generation model are jointly fine-tuned.

[0032] In a second aspect, a missing multi-label image recognition device based on description text generation is provided in the embodiments, and the device includes:

[0033] The model training module is configured to obtain an initial prediction model including a text generation branch and a label prediction branch; input a missing multi-label image into the text generation branch to output a natural language description text; input the natural language description text into the label prediction branch to output a predicted pseudo label; based on the predicted pseudo label, fill in the labels of the missing multi-label image to obtain a pseudo real label; based on the pseudo real label, optimize parameters of the initial prediction model to obtain a target prediction model.

[0034] The model application module is configured to input a to-be-identified image into the target prediction model for label prediction to output a predicted label set of the to-be-identified image.

[0035] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method for identifying a missing multi-label image based on a description text generation when executing the computer program.

[0036] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method for identifying a missing multi-label image based on a description text generation.

[0037] Compared with the related art, the method and device for identifying a missing multi-label image based on a description text generation provided in the embodiment solve the problem of poor performance of missing multi-label image identification by obtaining an initial prediction model including a text generation branch and a label prediction branch; inputting a missing multi-label image into the text generation branch to output a natural language description text; inputting the natural language description text into the label prediction branch to output a predicted pseudo label; using the predicted pseudo label to fill in the labels of the missing multi-label image to obtain a pseudo real label; using the pseudo real label as a training sample to optimize parameters of the initial prediction model to obtain a target prediction model; inputting a to-be-identified image into the target prediction model for label prediction to output a predicted label set of the to-be-identified image. By generating a natural language description for an image and analyzing the natural language description, more potential semantic information is obtained to compensate for the lack of label information, thereby improving the stability and recognition accuracy of the model.

[0038] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0040] Figure 1 A hardware structure block diagram of a terminal based on the missing multi-label image recognition method generated based on a description text in the embodiment of the present application;

[0041] Figure 2 A flowchart of the missing multi-label image recognition method generated based on a description text in the embodiment of the present application;

[0042] Figure 3 A schematic diagram of the training process of the initial prediction model in the preferred embodiment of the present application;

[0043] Figure 4 A schematic diagram of the application process of the target prediction model in the preferred embodiment of the present application;

[0044] Figure 5 A schematic diagram of the forward propagation process of the initial prediction model in the preferred embodiment of the present application;

[0045] Figure 6 A structure block diagram of the missing multi-label image recognition device based on the description text generated in the embodiment of the present application.

[0046] Reference signs: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 61, model training module; 62, model application module. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical scheme and advantages of the present application, the present application is described and explained in combination with the drawings and embodiments.

[0048] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as those commonly understood by a person of ordinary skill in the art to which the present application belongs. The terms "one", "a", "an", "the", "these", and similar terms in the present application do not mean "only one" or "exactly one", but can mean "one or more" or "at least one". The terms "include", "contain", "have", and any variant thereof in the present application are intended to cover the non-exclusive inclusion; for example, a process, method, and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. The terms "connect", "connect", "couple" and the like in the present application are not limited to physical or mechanical connection, but can include electrical connection, whether direct or indirect. The term "multiple" in the present application means two or more. The term "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. Generally, the character " / " represents the relationship between the objects before and after it as "or". The terms "first", "second", "third" and the like in the present application are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0049] The method embodiments provided in the present embodiment can be executed in a terminal, a computer or a similar computing device. For example, the method embodiments are executed on a terminal, Figure 1 is a hardware structure diagram of a terminal based on the description text generation of the missing multi-label image recognition method of the present embodiment. As shown in Figure 1 , the terminal can include one or more (only one in Figure 1 ) processor 102 and memory 104 for storing data, wherein the processor 102 can include but not limited to processing device such as microprocessor MCU or programmable logic device FPGA. The above terminal can also include transmission device 106 for communication function and input / output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above terminal. For example, the terminal can include more or less components than those shown in Figure 1 , or have a different configuration from that shown in Figure 1 .

[0050] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the method for generating a missing multi-label image based on a description text in the embodiment. The processor 102 can execute various functional applications and data processing, i.e., implement the method described above, by running the computer program stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0051] The transmission device 106 is configured to receive or send data via a network. The network includes a wireless network provided by a communication provider of the terminal. In an example, the transmission device 106 includes a network interface controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In an example, the transmission device 106 can be a radio frequency (RF) module configured to communicate with the Internet in a wireless manner.

[0052] In the embodiment, a method for generating a missing multi-label image based on a description text is provided, Figure 2 is a flowchart of the method for generating a missing multi-label image based on a description text in the embodiment, as shown in Figure 2 The flowchart includes the following steps:

[0053] In step S201, an initial prediction model including a text generation branch and a label prediction branch is obtained. A missing multi-label image is input into the text generation branch to output a natural language description text. The natural language description text is input into the label prediction branch to output a predicted pseudo label. The predicted pseudo label is used to complete the labels of the missing multi-label image to obtain a pseudo real label. The pseudo real label is used as a training sample to optimize the parameters of the initial prediction model to obtain a target prediction model.

[0054] Specifically, the text generation branch is used to generate natural language description text for image data. The text generation branch can be implemented based on an encoder-decoder architecture model, in which the encoder adopts a residual network (ResNet) or a visual Transformer (ViT); and the decoder adopts a multi-modal model (BLIP) or a recurrent neural network. The label prediction branch is used to extract label information from the natural language description text. The label prediction branch can adopt a pre-trained language model, such as a Bidirectional Encoder Representations from Transformers (BERT), and an image-text matching mechanism can be introduced after the pre-trained language model to realize self-calibration of the prediction result and improve system robustness.

[0055] In step S202, the image to be recognized is input into the target prediction model for label prediction, and a predicted label set of the image to be recognized is output.

[0056] Specifically, the processing process of the target prediction model for the image to be recognized includes: generating a predicted text for the image to be recognized; extracting label information from the predicted text to obtain a candidate label set; and obtaining a predicted label set after calibration and denoising of the candidate label set.

[0057] Taking a wild animal monitoring scene as an example, the implementation process of step S202 is as follows: (1) description generation stage: input an image containing multiple animals in the wild, and the trained text generation branch outputs a predicted text: “There are three zebras drinking water on the left side of the image, a cheetah hidden in the middle grass, three antelopes running in the distance, and the background is an African savannah at dusk.” (2) label prediction stage: the trained label prediction branch identifies “zebras”, “cheetahs”, “antelopes” and other objects, “drinking water”, “hiding”, “running” and other behavior attributes, and other entity information from the predicted text, thereby obtaining a candidate label set {zebras, cheetahs, antelopes, …, savannah, dusk} related to these entity information. (3) Cross-modal verification stage: calculate the similarity between the text embedding of each label and the visual features of the middle region of the image, for example, confirm that there are feline features in the region, and retain the label “cheetah”. Filter mislabeled labels that lack visual evidence, such as “trees”. (4) Final output: output a multi-label set with high confidence, i.e., the predicted label set {zebras (0.98), cheetahs (0.95), antelopes (0.92), drinking water (0.87), savannah (0.93), dusk (0.89)}.

[0058] In the embodiment, the natural language description text is output by inputting the missing multi-label image into a text generation branch of an initial prediction model; the predicted pseudo label is output by inputting the natural language description text into a label prediction branch of the initial prediction model; the pseudo real label is obtained by filling in the label of the missing multi-label image with the predicted pseudo label, and the pseudo real label is used as a training sample to optimize the parameters of the initial prediction model to obtain a target prediction model; and the predicted label set of the to-be-identified image is output by inputting the to-be-identified image into the target prediction model, thereby solving the problem of poor recognition performance of the missing multi-label image. In the embodiment, more potential semantic information is obtained by generating a natural language description for the image and analyzing the natural language description, so as to make up for the deficiency of the label information and improve the stability and recognition accuracy of the model.

[0059] In some embodiments, referring to Figure 3 , the text generation branch comprises a visual encoder and a multi-modal generation model.

[0060] The visual encoder is configured to extract global features and local region features of the missing multi-label image to obtain multi-level visual representations. The multi-modal generation model is configured to generate a natural language description text comprising object, attribute and scene information based on the multi-level visual representations.

[0061] Specifically, the visual encoder can adopt ViT or ResNet. The multi-modal generation model can be based on a Bootstrapping Language-Image Pre-training (BLIP) pre-training model for unified visual language understanding and generation, and the BLIP is combined with an attention mechanism to generate a natural language description text covering object, attribute and scene information of the input picture.

[0062] In the embodiment, fine-grained description text is generated to provide rich semantic clues for label prediction.

[0063] In some embodiments, referring to Figure 3 , the method further comprises: jointly optimizing the visual encoder and the multi-modal generation model by contrast learning to align the semantic content of the natural language description text and the visual content of the multi-level visual representations.

[0064] Specifically, contrast learning is a strategy for model training, which learns to distinguish image-text pairs with semantic alignment by constructing positive sample pairs (image and its matching description text) and negative sample pairs (image and non-matching text). After training, the model can directly generate a description highly aligned with the image content, such as accurately covering details such as "left zebra drinking water" and "sunset grassland background".

[0065] In the embodiment, the semantic coverage of the generated text is adjusted through contrastive learning to ensure that the description content is highly aligned with the visual content of the image.

[0066] In some embodiments, the label prediction branch includes a semantic parser and a multi-modal matching model. Figure 3

[0067] The semantic parser is configured to parse the natural language description text to obtain a candidate label set. Specifically, the semantic parser uses a pre-trained language model (e.g., BERT) to perform named entity recognition and relation extraction on the description text to identify at least one of object entities (e.g., zebra, cheetah, antelope, grassland), location entities (e.g., left side, in the grass, far away, background), action entities (e.g., drinking water, hiding, running), scene entities (e.g., dusk, African savannah), and attribute entities (e.g., three (quantity), stripes (appearance)) in the natural language description text, and constructs a candidate label set based on the obtained entity information.

[0068] The multi-modal matching model is configured to verify and filter labels in the candidate label set based on the local region features of the missing multi-label image to obtain predicted pseudo labels. Specifically, the multi-modal matching model can obtain the local region features output by the visual encoder, or obtain the missing multi-label image and perform segmentation on it to further obtain the local region features in the segmented slices.

[0069] In the embodiment, the synergistic effect of multi-modal information is considered, and the confidence of the prediction result is further calibrated by means of an image-text matching mechanism, thereby effectively improving the robustness and accuracy of the image recognition system.

[0070] In some embodiments, parsing the natural language description text to obtain a candidate label set includes:

[0071] In step S310, various entities in the natural language description text are identified to obtain a set of text entities.

[0072] Specifically, a pre-trained language model (e.g., BERT) is used to perform named entity recognition and relation extraction on the description text to identify object entities, location entities, action entities, scene entities, and attribute entities in the natural language description text. For example, the natural language description text is "There are three zebras drinking water on the left side of the image, a cheetah hiding in the middle of the grass, three antelopes running in the distance, and a background of an African savannah at dusk", and the semantic parser identifies "zebra", "cheetah", "antelope" as object entities, and "drink water", "hide", "run" as action entities.

[0073] ​Step S320, match the set of text entities with a label dictionary to obtain a candidate label set; wherein the label dictionary is obtained based on the current task type.

[0074] Specifically, for the wild animal monitoring scene, the label dictionary can adopt one and a combination of multiple ones of a wild animal dictionary, an animal species dictionary, a natural environment dictionary, and a population interaction dictionary. The set of text entities is matched with the animal species dictionary, and a "drinking water" behavior label missed by a traditional method is supplemented to generate an initial candidate label set {zebra, cheetah, antelope, grassland, dusk,..., drinking water}.

[0075] In this embodiment, the label dictionary functions as a task-adaptive semantic filter, ensures that the initial candidate label set has the effects of standardization and low noise, lays a high-quality foundation for subsequent probability graph model optimization, and is a key link to improve the robustness of the model.

[0076] In some embodiments, matching the set of text entities with the label dictionary to obtain the candidate label set comprises:

[0077] Step S321, calculate semantic similarity between each label item in the label dictionary and each text entity; and determine an initial label set from the label dictionary based on the semantic similarity.

[0078] Specifically, the semantic similarity is calculated by an encoder network, for example, a cosine similarity is calculated to represent the semantic similarity. In one embodiment, a label item with a cosine similarity greater than a threshold value to the text entity is extracted from the label dictionary as one of the labels and added to the initial label set.

[0079] Step S322, input the initial label set into a graph convolution network model for relevance judgment to output the candidate label set; wherein an adjacency matrix of the graph convolution network model is obtained based on binary label co-occurrence information.

[0080] Specifically, the adjacency matrix is calculated based on the binary label co-occurrence information, and the graph convolution network model is constructed based on the adjacency matrix. The binary label co-occurrence information calculation is the number of times different labels appear together in the same instance on the training set. The graph convolution network model scores the relevance of the candidate labels in the initial label set and outputs a probabilistic prediction result. The probabilistic prediction result is used to judge whether the relevance of the labels in the initial label set to the text entities recognized in the natural language description meets the requirements, and further denoising is achieved.

[0081] In some embodiments, verifying and screening the labels in the candidate label set based on the local region features of the missing multi-label image to obtain a predicted pseudo label comprises:

[0082] Step S410, map the local region features of the missing multi-label image and the labels in the candidate label set to a unified semantic space to obtain image region representation and candidate label representation.

[0083] Specifically, first, the joint embedding space is constructed. The visual-textual dual tower encoding model can be used to map the local region features output by the visual encoder and the label texts in the candidate label set to a unified semantic space. Here, the contrast loss is used to optimize the feature alignment.

[0084] Step S420, calculate the similarity of the image region representation and the candidate label representation to generate a confidence score.

[0085] Specifically, for each label in the candidate label set, a class Prompt (such as "This is a photo of [Class]", [Class] is the label text) is constructed through the prompt word engineering. The multi-modal matching model (Contrastive Language-Image Pre-Training, CLIP) is used to calculate the local similarity of the image region representation and the candidate label representation of the class Prompt to generate a confidence score.

[0086] Step S430, automatically adjust the filtering threshold based on the distribution of the confidence score, and filter out the predicted pseudo label set based on the adjusted filtering threshold.

[0087] Specifically, referring to Figure 3 In the training phase, the predicted pseudo label is merged with the original label of the missing multi-label image sample (i.e., the missing true label) to construct a pseudo true label for fitting. The pseudo true label is used as a training sample to fit the prediction result of the semantic parser, optimize the semantic parser parameters, and improve the accuracy of the semantic parser in generating the candidate label set. The prediction result of the semantic parser can be the label prediction value derived from the candidate label set, or the label prediction value derived from the initial label set.

[0088] In the prediction phase, referring to Figure 4 The trained prediction model is used to extract the global features and local region features of the image to be recognized to obtain multi-level visual representations. Based on the multi-level visual representations, a prediction text including object, attribute and scene information is generated. The entity text is extracted from the prediction text and matched with the label dictionary of the current classification task to obtain a candidate label set. The labels in the candidate label set are verified and filtered based on the local region features of the image to be recognized to obtain a pseudo label, which will be directly used as the prediction label of the image to be recognized.

[0089] In this embodiment, the cross-modal information between the image and the label is effectively fused to improve the prediction accuracy.

[0090] In some embodiments, the initial prediction model is an end-to-end model, the text generation loss function is calculated based on the natural language description text, the label prediction loss function is calculated based on the candidate label set, and the cross-modal alignment loss function is calculated based on the predicted pseudo label; in the process of training the initial prediction model with pseudo real labels as training samples and optimizing the parameters of the initial prediction model, the text generation loss function, the label prediction loss function, and the cross-modal alignment loss function are jointly optimized.

[0091] Specifically, the text generation loss function adopts a cross-entropy loss function, the label prediction loss function adopts a binary cross-entropy loss function, and the cross-modal alignment loss function adopts a contrastive loss function.

[0092] In this embodiment, the multiple loss functions jointly guide the parameter update in the backpropagation process, so that the entire model forms a unified learning process from input to output, and the model focuses on multiple targets (such as text generation, label prediction, cross-modal alignment, etc.) at the same time, thereby avoiding the bias or overfitting that may be caused by single task optimization. This way helps the model better understand the multi-level semantic information of the input data.

[0093] In some embodiments, during the training of the initial prediction model, the weight coefficients of the loss functions are adjusted based on the performance of the validation set.

[0094] In this embodiment, the validation set makes the model better adapt to unseen data, rather than just optimizing for training data, which helps reduce the risk of overfitting.

[0095] In some embodiments, the text generation branch adopts an alternating training strategy; the alternating training strategy includes: fixing the parameters of the visual encoder, and preliminarily optimizing the parameters of the multi-modal generation model; jointly fine-tuning the parameters of the visual encoder and the preliminarily optimized multi-modal generation model.

[0096] In this embodiment, the alternating training strategy can improve the stability and efficiency of the training process and promote the quality of parameter optimization.

[0097] The following describes and illustrates the embodiments by preferred embodiments. The method for generating a missing multi-label image based on a description text includes:

[0098] Step one, constructing an initial prediction model including a text generation branch and a label prediction branch. The text generation branch includes a visual encoder and a multi-modal generation model; the label prediction branch includes a semantic parser and a multi-modal matching model.

[0099] Step two, input the missing multi-label image into the text generation branch, output the natural language description text; input the natural language description text into the label prediction branch, output the predicted pseudo label; the predicted pseudo label is used to fill in the missing multi-label image label to obtain the pseudo real label; the pseudo real label is used as a training sample to optimize the parameters of the initial prediction model to obtain the target prediction model.

[0100] Figure 5 is a schematic diagram of the forward propagation process of the initial prediction model in the preferred embodiment, see Figure 3 and Figure 5 The training process of the initial prediction model in step two includes:

[0101] S1, natural language description text generation based on image: the text generation branch generates natural language description text for the input missing multi-label image as the text semantic basis for subsequent label prediction. The specific implementation includes:

[0102] (1.1) Visual feature extraction: a pre-trained visual encoder (such as ViT or ResNet) is used to extract global features and local region features of the missing multi-label image, forming multi-level visual representations.

[0103] (1.2) Text generation decoding: a multi-modal generation model (BLIP) is used to generate natural language description text covering the object, attribute and scene information of the input picture by combining attention mechanism for the input visual representation.

[0104] S2, label prediction based on text: the semantic parser of the label prediction branch parses potential labels from the generated natural language description text, solving the problem of insufficient supervision signal caused by missing artificial annotation. The specific implementation includes:

[0105] (2.1) Semantic parsing: a pre-trained language model (such as BERT) is used to perform named entity recognition and relation extraction on the natural language description text to identify entities and their attributes in the text.

[0106] (2.2) Label candidate generation: an initial label set is generated by calculating the semantic similarity (such as cosine similarity) between entities and their attributes and each label item in the pre-stored label dictionary.

[0107] (2.3) Probabilistic prediction: based on a graph convolution network model, the initial candidate labels in the initial label set are scored for relevance to obtain a candidate label set; the adjacency matrix of the graph convolution network model is obtained based on the binary label co-occurrence information.

[0108] S3, cross-modal verification: the image-text matching mechanism of the multi-modal matching model of the label prediction branch is used to realize self-calibration of the prediction results, improving the robustness of the system. The specific implementation includes:

[0109] (3.1) Joint embedding space construction: The visual-textual dual tower encoding model of the multi-modal matching model maps the image region features obtained in step (1.1) and the candidate labels obtained in step (2.3) to a unified semantic space, and adopts a contrastive loss to optimize feature alignment.

[0110] (3.2) Similarity calculation: For each candidate label, a class Prompt is constructed through prompt word engineering (such as "This is a photo of [Class]", [Class] being the label text), and the multi-modal matching model is used to calculate the local similarity between the text representation of the class Prompt and the image region features, generating a confidence score.

[0111] (3.3) Dynamic threshold filtering: The multi-modal matching model automatically adjusts the filtering threshold according to the confidence distribution, retaining high-confidence predictions to eliminate noisy labels, and generating the final pseudo-labels. In the model training phase, the pseudo-labels will be combined with the original missing labels of the samples, thereby constructing pseudo-real labels for fitting.

[0112] wherein a double-branch collaborative training strategy is adopted for the initial prediction model, and the double-branch collaborative training strategy specifically includes:

[0113] (1) Loss function design: jointly optimize the text generation loss (calculate cross-entropy based on step (1.2)), label prediction loss (calculate binary cross-entropy based on step (2.3)), and cross-modal alignment loss (calculate contrastive loss based on step (3.1)).

[0114] (2) End-to-end training: adopt an alternating training strategy, first fix the visual encoder to train the text multi-modal generation model, and then jointly fine-tune the visual encoder and all parameters of the multi-modal generation model.

[0115] (3) Dynamic weight adjustment: according to the performance of each module on the validation set, automatically adjust the weight coefficients of different loss terms in (4.1).

[0116] In the preferred embodiment, the problem of recognition accuracy caused by label missing in traditional methods is solved. By using the description text generated by the multi-modal large model, the lack of label information is made up, and the accuracy of image recognition is improved. At the same time, the verification mechanism of cross-modal similarity calculation is introduced, which effectively improves the robustness of the model and ensures the credibility of the prediction results. The end-to-end training method takes into account the efficiency and lightness, making the method suitable for large-scale data sets, with strong application value and good expansibility.

[0117] It is noted that the steps shown in the above flow or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0118] In the present embodiment, a missing multi-label image recognition apparatus based on description text generation is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0119] Figure 6 is a structural block diagram of the missing multi-label image recognition apparatus based on description text generation of the present embodiment, as shown in Figure 6 The apparatus includes a model training module 61 and a model application module 62.

[0120] The model training module 61 is configured to obtain an initial prediction model including a text generation branch and a label prediction branch; input the missing multi-label image into the text generation branch to output a natural language description text; input the natural language description text into the label prediction branch to output a predicted pseudo label; based on the predicted pseudo label, fill in the labels of the missing multi-label image to obtain a pseudo real label, and based on the pseudo real label, optimize the parameters of the initial prediction model to obtain a target prediction model.

[0121] The model application module 62 is configured to input an image to be recognized into the target prediction model for label prediction, and output a predicted label set of the image to be recognized.

[0122] In some embodiments, the text generation branch includes a visual encoder and a multi-modal generation model; the visual encoder is configured to extract global features and local region features of the missing multi-label image to obtain a multi-level visual representation; and the multi-modal generation model is configured to generate a natural language description text including object, attribute and scene information based on the multi-level visual representation.

[0123] In some embodiments, the method further includes: jointly optimizing the visual encoder and the multi-modal generation model through contrastive learning to align the semantic content of the natural language description text and the visual content of the multi-level visual representation.

[0124] In some embodiments, the label prediction branch includes: a semantic parser and a multi-modal matching model; the semantic parser is configured to parse the natural language description text to obtain a candidate label set; and the multi-modal matching model is configured to verify and filter labels in the candidate label set based on the local region features of the missing multi-label image to obtain the predicted pseudo label.

[0125] In some embodiments, parsing the natural language description text to obtain the candidate label set includes: identifying various entities in the natural language description text to obtain a set of text entities; and matching the set of text entities with a label dictionary to obtain the candidate label set; wherein the label dictionary is obtained based on a current task type.

[0126] In some embodiments, matching the set of text entities with the label dictionary to obtain the candidate label set includes: calculating semantic similarity between each label item in the label dictionary and each text entity; determining an initial label set from the label dictionary based on the semantic similarity; and inputting the initial label set into a graph convolution network model for relevance judgment to output the candidate label set; wherein an adjacency matrix of the graph convolution network model is obtained based on binary label co-occurrence information.

[0127] In some embodiments, verifying and filtering the labels in the candidate label set based on the local region features of the missing multi-label image to obtain the predicted pseudo label includes: mapping the local region features of the missing multi-label image and the labels in the candidate label set to a unified semantic space to obtain image region representations and candidate label representations; calculating similarity between the image region representations and the candidate label representations to generate a confidence score; automatically adjusting a filtering threshold based on a distribution of the confidence score; and filtering out a predicted pseudo label set based on the adjusted filtering threshold.

[0128] In some embodiments, the initial prediction model is an end-to-end model, a text generation loss function is calculated based on the natural language description text, a label prediction loss function is calculated based on the candidate label set, and a cross-modal alignment loss function is calculated based on the predicted pseudo label; and in the process of optimizing the parameters of the initial prediction model using the pseudo real label as a training sample, the text generation loss function, the label prediction loss function, and the cross-modal alignment loss function are jointly optimized.

[0129] In some embodiments, in the process of training the initial prediction model, the weight coefficients of the loss functions are adjusted based on the performance of the validation set.

[0130] In some embodiments, the text generation branch adopts an alternating training strategy; the alternating training strategy includes: fixing the parameters of the visual encoder, and preliminarily optimizing the parameters of the multi-modal generation model; and jointly fine-tuning the parameters of the visual encoder and the preliminarily optimized multi-modal generation model.

[0131] It should be noted that each of the above modules can be a functional module or a program module, which can be implemented by software or hardware. For the module implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combination.

[0132] In the embodiment, a computer device is also provided, including a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the above method embodiments.

[0133] Optionally, the computer device can further include a transmission device and an input and output device, wherein the transmission device is connected with the processor, and the input and output device is connected with the processor.

[0134] It should be noted that the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, which will not be described herein again.

[0135] In addition, in combination with the description text-based missing multi-label image recognition method provided in the above embodiments, a storage medium can also be provided to implement the method in the embodiment. The storage medium stores a computer program; when the computer program is executed by a processor, any of the above description text-based missing multi-label image recognition methods can be implemented.

[0136] It should be understood that the specific embodiments described herein are only used to explain this application, but not to limit it. According to the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0137] Obviously, the drawings are only some examples or embodiments of the present application, and those skilled in the art can also apply the present application to other similar situations without creative labor. In addition, it can be understood that although the work done in the development process can be complex and long, some design, manufacture or production changes made by those skilled in the art according to the technical content disclosed in the present application are only routine technical means and should not be regarded as insufficient disclosure of the present application.

[0138] The word "implementation" in this application refers to the specific features, structures, or characteristics described in connection with an implementation can be included in at least one implementation of the present application. The phrase appears in various places throughout the specification is not necessarily meant to refer to the same implementation, nor is it meant to imply that the features, structures, or characteristics so described can not be implemented in other implementations. It will be apparent to those having ordinary skill in the art that the implementations described herein can be combined with other implementations without losing the intended effect.

[0139] The above-described implementations only express several implementation manners of the present application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent protection scope. It should be noted that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for missing multi-label image recognition based on description text generation, characterized in that, The method comprises: acquiring an initial prediction model comprising a text generation branch and a label prediction branch; inputting a missing multi-label image into the text generation branch to output natural language description text; inputting the natural language description text into the label prediction branch to output predicted pseudo labels; the predicted pseudo labels are used to complete the labels of the missing multi-label image to obtain pseudo real labels; using the pseudo real labels as training samples to optimize the parameters of the initial prediction model to obtain a target prediction model; inputting an image to be recognized into the target prediction model for label prediction to output a predicted label set of the image to be recognized; wherein the text generation branch comprises a visual encoder and a multi-modal generation model; the visual encoder is configured to extract global features and local region features of a missing multi-label image to obtain multi-level visual representations; the multi-modal generation model is configured to generate natural language description text comprising object, attribute and scene information based on the multi-level visual representations. 2.The method of claim 1, wherein, The method further comprises: jointly optimizing the visual encoder and the multi-modal generation model through contrastive learning to align the semantic content of the natural language description text and the visual content of the multi-level visual representations. 3.The method of claim 1, wherein, The label prediction branch comprises a semantic parser and a multi-modal matching model; the semantic parser is configured to parse the natural language description text to obtain a candidate label set; the multi-modal matching model is configured to verify and filter labels in the candidate label set based on the local region features of the missing multi-label image to obtain predicted pseudo labels.

4. The method of claim 3, wherein the generating the missing multi-label image based on the description text comprises: Parsing the natural language description text to obtain a candidate label set comprises: identifying various entities in the natural language description text to obtain a set of text entities; matching the set of text entities with a label dictionary to obtain a candidate label set; wherein the label dictionary is obtained based on the current task type.

5. The method of claim 4, wherein the generating the missing multi-label image based on the description text comprises: Matching the set of text entities with a label dictionary to obtain a candidate label set comprises: calculating the semantic similarity between each label item in the label dictionary and each text entity; determining an initial label set from the label dictionary based on the semantic similarity; inputting the initial label set into a graph convolution network model for relevance judgment to output a candidate label set; wherein the adjacency matrix of the graph convolution network model is obtained based on binary label co-occurrence information. 6.The method of claim 3, wherein, Verifying and filtering labels in the candidate label set based on the local region features of the missing multi-label image to obtain predicted pseudo labels comprises: mapping the local region features of the missing multi-label image and the labels in the candidate label set to a unified semantic space to obtain image region representations and candidate label representations; calculating the similarity between the image region representations and the candidate label representations to generate a confidence score; automatically adjusting a filtering threshold based on the distribution of the confidence score, and filtering out the predicted pseudo label set based on the adjusted filtering threshold.

7. The method of claim 3, wherein the generating the missing multi-label image based on the description text comprises: The initial prediction model is an end-to-end model, a text generation loss function is calculated based on the natural language description text, a label prediction loss function is calculated based on a candidate label set, and a cross-modal alignment loss function is calculated based on the predicted pseudo label; in the process of optimizing the parameters of the initial prediction model with the pseudo real label as a training sample, the text generation loss function, the label prediction loss function and the cross-modal alignment loss function are jointly optimized.

8. The method of claim 7, wherein the method is based on description text generation. In the process of training the initial prediction model, based on the performance of the validation set, the weight coefficients of each loss function are adjusted. 9.The method of claim 1, wherein, The text generation branch adopts an alternating training strategy; the alternating training strategy comprises: The parameters of the visual encoder are fixed, and the parameters of the multi-modal generation model are preliminarily optimized; The parameters of the visual encoder and the preliminarily optimized multi-modal generation model are jointly fine-tuned.

10. A device for recognizing a missing multi-label image based on a description text generation, characterized by, The device comprises: A model training module is configured to obtain an initial prediction model comprising a text generation branch and a label prediction branch; input a missing multi-label image into the text generation branch to output a natural language description text; input the natural language description text into the label prediction branch to output a predicted pseudo label; based on the predicted pseudo label, fill in the labels of the missing multi-label image to obtain a pseudo real label, optimize the parameters of the initial prediction model based on the pseudo real label to obtain a target prediction model; wherein the text generation branch comprises a visual encoder and a multi-modal generation model; the visual encoder is configured to extract global features and local region features of a missing multi-label image to obtain multi-level visual representations; the multi-modal generation model is configured to generate a natural language description text comprising object, attribute and scene information based on the multi-level visual representations; A model application module is configured to input an image to be recognized into the target prediction model for label prediction, and output a predicted label set of the image to be recognized. 11.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-10 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 9.

Images