Method, device and medium for improving the generalization ability of cross-modal image retrieval model

Abstract

The present invention discloses a method, device, and medium for improving the generalization capability of a cross-modal image retrieval model. The method comprises: obtaining an image dataset, annotating the image data, and obtaining a large-scale image-text dataset with a single descriptive style; analyzing the large-scale image-text dataset with a single descriptive style to extract sentence templates with a single style; generating a set of sentence templates with diverse styles based on the single sentence templates; combining the set of sentence templates with diverse styles and using a template-based diversity enhancement strategy, annotating the image data again to obtain a large-scale image-text dataset with diverse descriptive styles; constructing and initializing a multimodal backbone network; and training the multimodal backbone network using a noise-aware masking strategy based on the large-scale image-text dataset. The present invention improves the generalization performance of a cross-modal image retrieval model and can be widely applied in the fields of image processing and recognition technology.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing and cross-modal retrieval, and in particular to a method, device and medium for improving the generalization capability of a cross-modal image retrieval model. Background Art

[0002] With the development of society, multimodal applications are becoming increasingly widespread in various fields, such as product search, image annotation, and person re-identification based on text descriptions. These multimodal technologies are developing rapidly, providing a tremendous impetus for social development. However, despite these remarkable achievements, these applications still face the challenge of data distribution differences in practice, resulting in a need for improvement in their generalization capabilities. While collecting data from workplace scenarios and then fine-tuning application models is an ideal solution, this requires additional costs. Therefore, training a model that can be directly applied to various target domains is of great significance.

[0003] To improve the generalization capabilities of models, many methods are currently available, such as data augmentation, improved model structure, and training strategies. Among them, training models using large-scale datasets shows great potential. Large-scale datasets can provide more diverse and comprehensive training samples, enabling models to better understand the data distribution and characteristics of different domains. The challenge is to collect high-quality large-scale datasets at low cost. The latest solution relies on multimodal large-scale model technology. Multimodal large-scale language models have demonstrated excellent capabilities in image annotation. These models combine the capabilities of natural language processing and computer vision to generate accurate and descriptive text annotations based on image content. How to use multimodal large-scale language model technology to enhance model generalization performance is a research area worthy of further exploration. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, device and medium for improving the generalization capability of a cross-modal image retrieval model.

[0005] The first technical solution adopted by the present invention is:

[0006] A method for improving the generalization ability of a cross-modal image retrieval model comprises the following steps:

[0007] Obtain an image dataset, annotate the image data, and obtain a large-scale image-text dataset with a single description style;

[0008] Analyze a large-scale image-text dataset with a single description style and extract sentence templates with a single style;

[0009] Generate a set of sentence templates with diverse styles based on a sentence template with a single style;

[0010] Combining a collection of sentence templates with diverse styles, and using a template-based diversity enhancement strategy, we annotate the image data again to obtain a large-scale image-text dataset with diverse description styles.

[0011] Build a multimodal Backbone network and initialize it;

[0012] A multimodal backbone network is trained using a noise-aware masking strategy based on a large-scale image-text dataset.

[0013] Furthermore, the step of acquiring an image dataset and annotating the image data to obtain a large-scale image-text dataset with a single description style includes:

[0014] Based on the image annotation technology of multimodal large language model, static instructions are used to drive the multimodal large language model to generate text descriptions for images in the image dataset, and a large-scale image-text dataset with a single description style is obtained, in which the text is named static text (Static Text; T s ).

[0015] Furthermore, a large multimodal language model specifically refers to a large artificial intelligence neural network based mainly on the Transformer structure that can process and understand various types of data inputs (such as text, images, sounds, etc.); image annotation technology specifically refers to the technology of generating corresponding text descriptions based on the image content, which requires the use of "instructions" to drive the work of the large multimodal language model. The instructions here are named "static instructions".

[0016] Furthermore, the image data is annotated to obtain a large-scale image-text dataset with a single description style, including:

[0017] Communicate with the large language model ChatGPT to obtain a series of candidate instructions; test the experimental results of the candidate instructions, analyze the instruction defects based on the results, and finally modify the static instructions;

[0018] Use the above static instructions with a multimodal large language model to annotate the image dataset.

[0019] Furthermore, the analysis of a large-scale image-text dataset with a single description style to extract sentence templates with a single style includes:

[0020] Analyze large-scale image-text datasets with a single descriptive style, and extract sentence templates with a single style based on the language analysis capabilities of a large language model. The large language model specifically refers to a large artificial intelligence model based on the Transformer structure, which is used to understand and generate natural language.

[0021] Furthermore, the analysis of a large-scale image-text dataset with a single description style to extract sentence templates with a single style includes:

[0022] The text description generated by static instructions is fed into ChatGPT, and the sentence structure of the text is obtained through multiple rounds of dialogue.

[0023] Furthermore, the generation of a set of sentence templates with diverse styles based on a sentence template with a single style includes:

[0024] Design appropriate instructions to drive the preset large language model to innovate multiple new sentence templates based on existing single-style sentence templates through imitation and reasonable divergence;

[0025] Summarize new sentence templates and obtain a collection of sentence templates with diverse styles.

[0026] Furthermore, based on the generative capabilities of the large language model, a set of sentence templates with diverse styles is generated based on a single-style sentence template:

[0027] Require ChatGPT to imitate static sentence templates to generate new sentence templates;

[0028] After manual screening, a collection of sentence templates with diverse styles was obtained.

[0029] Furthermore, the image data is annotated again using a template-based diversity enhancement strategy in combination with a set of sentence templates with diverse styles, thereby obtaining a large-scale image-text dataset with diverse description styles, including:

[0030] Obtain static instructions from image annotation technology, randomly select a template from a collection of sentence templates with diverse styles, and embed it into a preset position in the static instruction to obtain dynamic instructions;

[0031] For all samples in the image dataset, dynamic instructions are used to drive the multimodal large model to regenerate text descriptions and obtain dynamic text;

[0032] By combining dynamic text with static text, we obtain a large-scale image-text dataset with diverse description styles.

[0033] Furthermore, we improved the image annotation technology of the multimodal large language model, combined it with a collection of sentence templates with diverse styles, and used a template-based diversity enhancement strategy to annotate the image data again, ultimately obtaining a large-scale image-text dataset with diverse description styles:

[0034] Modify the static instructions, and embed any template from a set of sentence templates with diverse styles into the static instructions to obtain dynamic instructions;

[0035] Using dynamic instructions and combining a large multimodal language model, the same batch of images are annotated again, ultimately obtaining a large-scale image-text dataset with diverse description styles.

[0036] Furthermore, the multimodal backbone network is a deep neural network based on Transformer, and the construction method of the deep neural network is:

[0037] The classic CLIP-ViT-B / 16 network is used, which consists of an image encoder consisting of a 12-layer Visual Transformer and a text encoder consisting of a 12-layer Transformer.

[0038] Furthermore, before training the network, the following steps are also included to enhance the dataset:

[0039] For a training image, performing enhancement processing and normalization processing on it; wherein the enhancement processing includes scaling processing, flipping processing or cropping processing;

[0040] The description text T for this training image full , describe the text T full The word segmentation is a word sequence of length N, and the word sequence is enclosed by [SOS] and [EOS] to indicate the start and end of the sequence; get the description text T full The noise level stored in the noise dictionary in the previous cycle is used to describe the text T full Perform noise-aware masking to obtain the text T nam .

[0041] The training image is input to the image encoder and the description text T full and text T nam Input into the text encoder and output feature F v 、F t 、v cls , t ′ eos Among them, F v is the output of the image block at the lth layer of the image encoder, F t Is the description text Tfull At the lth layer output of the text encoder, v cls is the global feature of the image output at the last layer of the image encoder, t ′ eos It is text T nam Global features output at the last layer of the text encoder.

[0042] Furthermore, the multimodal Backbone network is trained using the noise-aware masking strategy, including:

[0043] Build a dictionary of text word noise levels;

[0044] The training image passes through the image encoder, and the image block (patch) features are output at the lth layer of the image encoder Output image semantic features in the last layer

[0045] The original text retrieves the noise level of the text word in the previous training cycle (epoch) recorded in the text word noise level dictionary, and implements a noise-aware masking strategy on the original text based on the noise level;

[0046] The original text and the masked text are input into the text encoder together, and the original text token features are output at the lth layer of the text encoder. Output the semantic features of the masked text in the last layer

[0047] According to the image block feature F v and the original text word feature F t Calculate the similarity to obtain the new noise level of the original text word, change the corresponding element value in the text word noise level dictionary to the new value and save it for use in the next training cycle;

[0048] According to the image semantic features v cls and the semantic features of the masked text t ′ eos Calculate the similarity distribution matching (SDM) loss; calculate the backpropagation gradient based on the loss function, and update the network model parameters based on the gradient and learning rate.

[0049] Furthermore, the image block feature F v and the original text word feature F t Calculate the similarity to obtain the new noise level of the original text word, change the corresponding element value in the text word noise level dictionary to the new value and save it for use in the next training cycle, including:

[0050] According to the image block feature F v and the original text word feature F t Calculate the cosine similarity S:

[0051]

[0052] in, is the cosine similarity matrix; S ij Represents the similarity between the i-th word and the j-th image block;

[0053] The noise level of each word is obtained and saved according to the cosine similarity value between each word and all image blocks. The noise level r of each word i is i It can be calculated like this:

[0054]

[0055] The set list of noise levels for each word in the entire text is r = [r1,…,r N The noise level represents the probability that the word is a noise word. The higher the noise level, the higher the probability that the word should be masked. In order to stabilize the training process, the word noise level of the entire text is normalized as follows:

[0056]

[0057]

[0058] After normalization, the average masking probability of the entire text is stabilized at the hyperparameter ρ. ′ Store the text word noise level dictionary and wait for the next training cycle to call it.

[0059] Furthermore, the image semantic feature v cls and the semantic features of the masked text t ′ eos Calculate the similarity distribution matching (SDM) loss; calculate the backpropagation gradient based on the loss function, and update the network model parameters based on the gradient and learning rate, including:

[0060] Given a mini-batch of B matching image-text pairs First, determine the matching relationship between each image and text. where y i,j =1 and y i,j = 0 for positive and negative matching image-text pairs, respectively;

[0061] Calculate the true matching distribution q of the i-th image i , whose jth element is

[0062] Predicted probability distribution p i With q i Align as follows:

[0063]

[0064] where ε is a small value to avoid numerical problems, and

[0065]

[0066] Here Calculate the cosine similarity between u and v, where τ is a temperature coefficient.

[0067] By using the equation L i2t and equation p i,j Exchange v cls and t ′ eos At the location of , we can calculate the SDM loss L from text to image t2i Finally, the complete SDM loss is calculated as follows:

[0068]

[0069] Furthermore, in the test, the trained Backbone network is used to test on any downstream task data.

[0070] Another technical solution adopted in the present invention is:

[0071] A device for improving the generalization capability of a cross-modal image retrieval model, comprising:

[0072] at least one processor;

[0073] at least one memory for storing at least one program;

[0074] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0075] Another technical solution adopted in the present invention is:

[0076] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.

[0077] The beneficial effects of the present invention are as follows: the present invention proposes a novel template-based diversity-enhanced image annotation technology, which can be transferred to any multimodal large language model and improve the diversity of the generated text. At the same time, in order to deal with the erroneous descriptions that may occur during generation, the present invention proposes a novel noise-aware masking method. This method can automatically mask words with high noise levels in the generated text. The combination of the two methods forms a complete framework from data to model, and a multimodal network with excellent transfer capabilities can be obtained with a small amount of manual work, without the need to use additional data for fine-tuning in downstream tasks, effectively saving manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0079] Figure 1 is a flowchart of the steps of a method for improving the generalization ability of a cross-modal image retrieval model in an embodiment of the present invention;

[0080] Figure 2 is a flowchart of the steps of the template-based diversity enhancement method in an embodiment of the present invention;

[0081] Figure 3 4 is a flowchart of the steps of the noise perception masking method in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0083] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0084] In the description of the present invention, the meaning of "several" is one or more, the meaning of "many" is two or more, and "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of the first and the second, it is only for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, "and / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0085] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0086] Explanation of terms:

[0087] Transformer: Transformer is a deep learning model structure used to process sequential data, such as text and language. It captures contextual information in the sequence through self-attention and positional encoding, and has good modeling capabilities and parallel computing performance.

[0088] Multi-modal Large Language Model: A multi-modal large language model is a language model capable of processing data from multiple modalities (such as text, images, and audio). It combines information from multiple input modalities and is trained using large-scale data to provide more accurate and comprehensive understanding and generation of multimodal data. In this article, it specifically refers to the text-image modality.

[0089] Large Language Model: A large language model is a deep learning model capable of processing natural language. It learns the statistical laws and semantic representations of language by pre-training on a large corpus. Large language models typically have high language understanding and generation capabilities and can be used for a variety of natural language processing tasks.

[0090] Image Captioning: Image captioning is the task of converting images into text descriptions. Image captioning models can automatically generate sentences describing the image content, thereby achieving semantic understanding and description of the image. Such models are typically trained using deep learning techniques and large-scale image and text datasets.

[0091] like Figure 1As shown, this embodiment provides a method for improving the generalization ability of a text-to-image person re-identification model, which includes a template-based diversity enhancement method to obtain a large number of image-text datasets with diverse styles, and a noise-aware masking training strategy that can resist text noise. The template-based diversity enhancement method can improve the shortcomings of the existing multimodal large model in generating single text diversity, enrich the text distribution in the training set, and thus improve the generalization ability of the model; at the same time, the noise-aware masking training strategy that resists text noise can autonomously locate the descriptive words that may be wrong in the generated text, and targeted masking of the wrong words can prevent the model from learning incorrect semantics, improve the discriminability of the features, and thus enhance the generalization ability of the model. The method specifically includes the following steps:

[0092] S1. Obtain a computer vision task image dataset, annotate the image data based on the image annotation technology of the multimodal large language model, and initially obtain a large-scale image-text dataset with a single description style.

[0093] In this example, the computer vision task image dataset selected is LUPerson, the multimodal large language model selected is Qwen and Shikra, and the large language model selected is ChatGPT. The image data is annotated, specifically: a dialogue is conducted with ChatGPT to obtain a series of candidate instructions; the experimental results of the candidate instructions are tested, and the instruction defects are analyzed based on the results. Finally, a static instruction is modified: "Write a description about the overall appearance of the person in the image, including the attributes: clothing, shoes, hairstyle, gender and belongings. If any attribute is not visible, you can ignore it. Do not imagine any contents that are not in the image." The image of LUPerson and the static instruction are sent to Qwen and Shikra to generate two sets of static descriptions respectively. These two sets of static descriptions and LUPerson images constitute a large-scale image-text dataset with a single description style.

[0094] S2. Analyze large-scale image-text datasets with a single descriptive style, and extract sentence templates with a single style based on the language analysis capabilities of large language models.

[0095] In some embodiments, step S2 is specifically to analyze Randomly sample several samples from these two groups of descriptions and feed them into ChatGPT to analyze the sentence structure and obtain sentences with a single style.

[0096] ChatGPT was required to imitate static sentence templates to generate several new sentence templates and manually screen out a set of 46 sentence templates with diverse styles.

[0097] S3. Based on the generative capabilities of the large language model, a set of sentence templates with diverse styles is generated based on a single-style sentence template.

[0098] Specifically, appropriate instructions are designed to drive the large language model to innovate multiple new sentence templates that have never been seen before based on existing sentence templates with a single style through imitation and reasonable divergence; the new sentence templates are aggregated to obtain a collection of sentence templates with diverse styles.

[0099] S4. Improve the image annotation technology of the multimodal large language model, combine it with a set of sentence templates with diverse styles, use a template-based diversity enhancement strategy, annotate the image data again, and finally obtain a large-scale image-text dataset with diverse description styles.

[0100] In some embodiments, see Figure 2 , modify the static instructions, and embed any template in the set of sentence templates with diverse styles into the static instructions to obtain the dynamic instructions: "Generate a description about the overall appearance of the person, including clothing, shoes, hairstyle, gender, and belongings, in a style similar to the template: {template}. If some requirements in the template are not visible, you can ignore them. Do not imagine any contents that are not in the image." where {template} is randomly sampled from the set of 46 sentence templates with diverse styles obtained in S2.

[0101] Using dynamic instructions, combined with the multimodal large language models Qwen and Shikra, the same batch of LUPerson pictures are annotated again to obtain Finally, we obtain the large-scale image-text dataset LUPerson-MLLM with diverse description styles.

[0102] S5. Build a multimodal Backbone network and initialize it.

[0103] Step S5 specifically involves constructing and initializing a Transformer-based deep neural network.

[0104] First, perform data augmentation: for a LUPerson-MLLM training image, scale its height and width to 384 and 128 respectively; flip it horizontally with a probability of 50%; randomly crop it with a padding of 10, so that the image is still 384 in height and 128 in width after cropping; finally, normalize the image, and the normalized mean and standard deviation are [0.48145466, 0.4578275, 0.40821073] and [0.26862954, 0.26130258, 0.27577711] respectively. The description text for this training image may be Any one of them, the present invention is uniformly named T full . dull The word segmentation is a word sequence of length N, N is based on T full The length of the word sequence varies. The word sequence is bracketed by [SOS] and [EOS] to indicate the beginning and end of the sequence. At the same time, we check T full The noise level stored in the noise dictionary in the previous cycle (initialized to 0.15 for the first training cycle). The noise level is used to adjust T full Perform a noise-aware masking operation to obtain T nam .

[0105] Then the picture is input to the image encoder, the text T full and T nam are input into the text encoder independently. F v 、F t 、v cls , t ′ eos . F v is the output of the image block at the lth layer of the image encoder, F t It's T full At the lth layer output of the text encoder, v cls is the global feature of the image output at the last layer of the image encoder, t ′ eos It's T nam Global features output at the last layer of the text encoder.

[0106] S6. Use noise-aware masking strategy in training.

[0107] In some embodiments, see Figure 3, step S6 specifically includes steps S61-S63:

[0108] S61, image block feature F v and the original text word feature F t Calculate the cosine similarity S:

[0109]

[0110] Is the cosine similarity matrix. Where S ij Represents the similarity between the i-th word and the j-th image block.

[0111] The noise level of each word is obtained and saved according to the cosine similarity value between each word and all image blocks. The noise level r of each word i is i It can be calculated like this:

[0112]

[0113] The set list of noise levels for each word in the entire text is r = [r1,…,r N The noise level represents the possibility that the word is a noise word. The higher the noise level, the higher the probability that the word should be masked. In order to stabilize the training process, the word noise level of the entire text is normalized as follows:

[0114]

[0115]

[0116] After normalization, the average masking probability of the entire text is stabilized at the hyperparameter ρ, which is set to 0.15 in this example. ′ Store the text word noise level dictionary and wait for the next training cycle to call it.

[0117] S62. Given a mini-batch of N matching image-text pairs First, determine the matching relationship between each image and text. where y i,j =1 and y i,j = 0 for positive and negative matching image-text pairs, respectively;

[0118] Calculate the true matching distribution q of the i-th image i , whose jth element is

[0119] Predicted probability distribution p i With q i Align as follows:

[0120]

[0121] Here, ε is a very small value used to avoid numerical problems. In this example, it is set to 1e-6, and

[0122]

[0123] Here Calculate the cosine similarity between u and v. τ is a temperature coefficient, which is set to 0.02 in this example.

[0124] By using the equation L i2t and equation p i,j Exchange v cls and t ′ eos At the location of , we can calculate the SDM loss L from text to image t2i Finally, the complete SDM loss is calculated as follows:

[0125]

[0126] S63, after the loss function is passed, the calculated gradient is passed back to Backbone, and the network parameters of Backbone are updated using the gradient descent update method; specifically, let the network parameter be θ, and the update formula is:

[0127]

[0128] Where α is the learning rate, which is set to 1e-5 in this example.

[0129] S7. In the test, the trained Backbone network is used to test on any downstream task data.

[0130] To verify the effectiveness of the present invention, as shown in Table 1 below, direct tests were conducted on the CUHK-PEDES, ICFG-PEDES, and RSTPReID datasets, using the mean Average Precision (mAP) and Rank-1 accuracy metrics. The mAP calculation method is as follows: the retrieval accuracy is calculated for each pedestrian image included in the query among all gallery images, and the average prediction accuracy of all query images is mAP; the Rank-1 accuracy calculation method is as follows: the first place in the retrieval ranking list returned by all queries is the average accuracy of the retrieval target.

[0131] Table 1 Performance comparison of the dataset LUPerson-MLLM of the present invention and other datasets in direct testing

[0132]

[0133]

[0134] To further verify the effectiveness of this method, more experiments were conducted in this example: Assuming that downstream datasets can be obtained for fine-tuning, the final performance of the model will be further improved, as shown in Table 2:

[0135] Table 2 Performance comparison data of the method of the present invention and other methods in fine-tuning test

[0136]

[0137] In summary, compared with the prior art, the method of this embodiment has at least the following advantages and beneficial effects:

[0138] (1) The method of this embodiment includes a novel template-based diversity enhancement method, which alleviates the problem of single text style in existing multimodal large language models, can effectively generate rich new style text, and help pedestrian re-identification models better improve their generalization ability.

[0139] (2) The noise-aware masking method can locate and mask incorrect words in the text description, preserve the information of the remaining correct text, effectively utilize the correct semantic information of the generated text, and greatly improve the discriminative power of the features.

[0140] (3) During the test, it is possible to obtain impressive performance without fine-tuning the model using downstream task datasets; if fine-tuning is possible, the performance of the model is significantly enhanced, and the present invention is a very powerful method.

[0141] (4) The template-based diversity enhancement method and noise-aware masking method of this example can be combined with other methods or models and have wide application value.

[0142] This embodiment further provides a device for improving the generalization capability of a cross-modal image retrieval model, including:

[0143] at least one processor;

[0144] at least one memory for storing at least one program;

[0145] When the at least one program is executed by the at least one processor, the at least one processor implements the following Figure 1 The method shown.

[0146] A device for improving the generalization capability of a cross-modal image retrieval model in this embodiment can execute a method for improving the generalization capability of a cross-modal image retrieval model provided by an embodiment of the method of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0147] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0148] This embodiment also provides a storage medium storing instructions or programs that can execute a method for improving the generalization capability of a cross-modal image retrieval model provided by an embodiment of the method of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0149] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0150] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0152] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0153] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0154] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0155] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0156] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0157] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for improving the generalization ability of a cross-modal image retrieval model, characterized in that: The following steps are involved: Obtain an image dataset, annotate the image data, and obtain a large-scale image-text dataset with a single description style; Analyze a large-scale image-text dataset with a single description style and extract sentence templates with a single style; Generate a set of sentence templates with diverse styles based on a sentence template with a single style; Combining a collection of sentence templates with diverse styles, and using a template-based diversity enhancement strategy, we annotate the image data again to obtain a large-scale image-text dataset with diverse description styles. Build a multimodal Backbone network and initialize it; Based on a large-scale image-text dataset, a multimodal backbone network is trained using a noise-aware masking strategy. The method of training a multimodal Backbone network using a noise-aware masking strategy includes: Build a dictionary of text word noise levels; The training image passes through the image encoder, and the image block feature F is output at the lth layer of the image encoder. v , output image semantic features v in the last layer cls ; The original text retrieves the noise level of the text word in the previous training cycle recorded in the text word noise level dictionary, and implements the noise perception masking strategy on the original text based on the noise level; The original text and the masked text are input into the text encoder together, and the original text word feature F is output at the lth layer of the text encoder. t , in the last layer, the semantic features t of the masked text are output ′ eos ; According to the image block feature F v and the original text word feature F t Calculate the similarity and get the new noise level of the original text words, Change the corresponding element value in the text word noise level dictionary to a new value and save it, waiting for use in the next training cycle; according to the image semantic feature v cls and the semantic features of the masked text t ′ eos Calculate the similarity distribution matching loss; calculate the backpropagation gradient based on the loss function, and update the network model parameters based on the gradient and learning rate; According to the image block feature F v and the original text word feature F t Calculate the similarity to obtain the new noise level of the original text word, change the corresponding element value in the text word noise level dictionary to the new value and save it for use in the next training cycle, including: According to the image block feature F v and the original text word feature F t Calculate cosine similarity; The noise level of each word is obtained and saved according to the cosine similarity value between each word and all image blocks; When used in the next cycle, the original masking probability of each word is calculated based on the noise level value of each word; The original masking probabilities of all word units are normalized so that the overall average masking probability is equal to the hyperparameter ρ.

2. The method for improving the generalization ability of a cross-modal image retrieval model according to claim 1, characterized in that: The step of acquiring an image dataset and annotating the image data to obtain a large-scale image-text dataset with a single description style includes: Image annotation technology based on a multimodal large language model uses static instructions to drive the multimodal large language model to generate text descriptions for images in an image dataset, obtaining a large-scale image-text dataset with a single description style, where the text is named static text.

3. The method for improving the generalization ability of a cross-modal image retrieval model according to claim 1, characterized in that: The large-scale image-text dataset with a single description style is analyzed to extract sentence templates with a single style, including: Analyze large-scale image-text datasets with a single descriptive style, and extract sentence templates with a single style based on the language analysis capabilities of a large language model. The large language model specifically refers to a large artificial intelligence model based on the Transformer structure.

4. The method for improving the generalization ability of a cross-modal image retrieval model according to claim 1, characterized in that: The method of generating a set of sentence templates with diverse styles based on a sentence template with a single style includes: Design instructions to drive the preset large language model to innovate multiple new sentence templates based on the existing single-style sentence template through imitation and reasonable divergence; Summarize new sentence templates and obtain a collection of sentence templates with diverse styles.

5. The method for improving the generalization ability of a cross-modal image retrieval model according to claim 2, characterized in that: The above method combines a set of sentence templates with diverse styles and uses a template-based diversity enhancement strategy to annotate the image data again, obtaining a large-scale image-text dataset with diverse description styles, including: Obtain static instructions from image annotation technology, randomly select a template from a collection of sentence templates with diverse styles, and embed it into a preset position in the static instruction to obtain dynamic instructions; For all samples in the image dataset, dynamic instructions are used to drive the multimodal large model to regenerate text descriptions. Get dynamic text; By combining dynamic text with static text, we obtain a large-scale image-text dataset with diverse description styles.

6. The method for improving the generalization ability of a cross-modal image retrieval model according to claim 1, characterized in that: The multimodal Backbone network is a deep neural network based on Transformer, which includes an image encoder composed of multiple layers of Visual Transformer and a text encoder composed of multiple layers of Transformer.

7. A device for improving the generalization capability of a cross-modal image retrieval model, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

Images