Description generation method based on continuous zero sample

Abstract

The invention provides a description generation method based on continuous zero samples. The description generation method comprises the following steps: sequencing plain text training corpora according to the difficulty of continuous learning; setting that each training step t only can access the t-th corpus, and for the t-th learning step, generating a synthetic image for an input text of the t-th corpus by using a diffusion model; extracting image embedding of the synthesized image by using a CLIP image encoder; retrieving a group of text description embedding with similar semantics in a text corpus by using image embedding; image embedding and text description embedding are input into a fusion module network, and the output of a fusion module is defined as a soft prompt of a large language model; for an input text, key entities are extracted from the text through word frequency statistics so as to construct a soft prompt and a hard prompt of the large language model; hard prompts and soft prompts are input into a large language model network, and key parts in input are concerned through an attention mechanism.

Description

TECHNICAL FIELD

[0001] The application relates to an image description method, in particular to a description generation method based on persistent zero samples. BACKGROUND

[0002] The progress of the image description task mainly depends on the supervised learning of large-scale image-text pairs, and although significant progress has been made in improving the alignment of visual and textual information, the construction cost of such a dataset is still a major limiting factor. Pure text training effectively solves this problem, and noise is injected into the latent space of CLIP to realize zero-shot image description; this method no longer requires paired data, but there is a modal gap: the model is based on text embedding during training and image embedding during inference. In addition, the offline nature of pure text training will cause catastrophic forgetting when adapting to new text corpora in sequence; fine-tuning on new tasks often quickly drifts from existing knowledge, weakening the generalization ability across multiple text domains. SUMMARY

[0003] The application aims to provide a description generation method based on persistent zero samples, which includes a training phase and an inference phase, wherein the training phase includes:

[0004] Step S100, define a zero sample image description dataset ZCCL of continuous learning as a pure text training corpus for continuous adaptation, and sort the pure text training corpus according to the difficulty of continuous learning to obtain {D1, D2,..., D T} encode the text in the text corpus using a CLIP text encoder;

[0005] Step S200, set that only D t corpus can be accessed at each training step t, for the tth learning step, generate a synthetic image using a diffusion model for the input text Text of the D t corpus;

[0006] Step S300, use a CLIP image encoder to extract a CLIP image embedding from the synthetic image;

[0007] Step S400, use the image embedding to retrieve a set of semantically similar text description embeddings R from the text corpus;

[0008] Step S500, input and R into a fusion module network, and define the output of the fusion module as a soft prompt for a large language model;

[0009] Step S600, for the input text Text, extract key entities from the text Text by word frequency statistics to construct a soft prompt hard prompt for a large language model;

[0010] Step S700, input the hard prompt and the soft prompt into the large language model network, and focus on the key part in the input through the attention mechanism;

[0011] Step S800, the learning parameter set Ψ t of the fusion module network , and the learning parameter set of the large language model network are stored in the text domain memory;

[0012] The inference stage includes:

[0013] Step S110, encoding the input image using the CLIP image encoder to obtain an image embedding I;

[0014] Step S210, performing image-to-text retrieval on the image embedding I to obtain a similar text embedding Ts;

[0015] Step S210, querying the CLIP image embedding of the synthesized image closest to the image embedding I in the text domain memory ;

[0016] Step S310, obtaining the CLIP image embedding of the synthesized image corresponding to {Ψ t , , } for guiding the fusion module network and the large language model network;

[0017] Step S410, injecting the image embedding I and the text embedding Ts into the FM to obtain a soft prompt;

[0018] Step S510, extracting key entities from the text embedding Ts by word frequency statistics to construct a hard prompt;

[0019] Step S610, inputting the hard prompt and the soft prompt into the large language model network to obtain a final image description.

[0020] Further, in step S200, the process of generating a synthesized image using a diffusion model includes:

[0021] Step S201, using the text encoder of CLIP to convert the input text Text into a text feature vector;

[0022] Step S202, starting from a random noise image, gradually adding noise to the image until the image is completely covered by noise;

[0023] Step S203: Starting from a completely noisy image, noise is gradually removed through a reverse diffusion process until a satisfactory quality is achieved, generating a synthetic image that matches the input text content.

[0024] Further, in step S500, calculation is performed in the CLIP embedding space. With D t For each text description vector in the corpus, the cosine similarity is used to select the top ρ text description embeddings based on the highest similarity.

[0025] Furthermore, in step S500, LoRA-based memory units are introduced into the linear layer and QKV projection layer of the fusion module network. The specific process of step S500 is as follows:

[0026] Step S501, the linear layers f1 and f2 are paired A linear transformation of R yields f1( f1 and f2(R);

[0027] Step S502, let W be the transformation matrix of the linear layer, for f1( f1 and f2(R) are augmented through LoRA memory cells. and ,

[0028] ,

[0029] Among them I e =f1( ), R e =f2(R), where A and B represent the dimension reduction and dimension increase projection layers of LoRA, respectively;

[0030] Step S503: In each M* cross-attention layer, perform low-rank augmentation on the query, key, and value according to the LoRA principle to obtain F. CL

[0031] ,

[0032] ,

[0033] Among them, Q CL K CL V CL This refers to the query, key, and value matrix in the cross-attention mechanism.

[0034] Step S504, F CL With learnable parameter L p Cascade the layers, and in each L* mapping layer, augment the query, key, and value of each self-attention layer according to the LoRA principle to obtain F. ML

[0035] ,

[0036] ,

[0037] wherein Q ML , K ML , V ML are query, key and value matrices in self-attention mechanism;

[0038] Step S505 extracts an embedding of the same length as L ML from F p as a soft prompt for the large language model.

[0039] Further, step S600 specifically comprises:

[0040] Step S601, for the input text Text, obtains a text embedding, removes irrelevant characters in the text, and divides the text into words or phrases;

[0041] Step S602, apply a named entity recognition tool to automatically identify entities in the text;

[0042] Step S603, count the number of occurrences of each entity in the text, and sort the entities according to the number of occurrences from high to low;

[0043] Step S604, set a threshold, and select entities with a number of occurrences exceeding the threshold as key entities;

[0044] Step S605, integrate the selected key entities into a discrete string;

[0045] Step S606, tokenize the discrete string and input it into the CLIP text encoder to obtain a hard prompt F HP .

[0046] Further, in step S700, the large language model adopts a GPT-2 model, and a LoRA module is inserted in each self-attention block and feedforward layer of the GPT-2 model; for a given GPT-2 self-attention layer GPTSelfAttn, calculate

[0047] ,

[0048] ;

[0049] For the GPT-2 feedforward layer GPTFeedForward, calculate

[0050] ,

[0051] ,

[0052] where W FF is the first layer weight matrix of the feedforward network layer.

[0053] Compared with the prior art, the present application has the following advantages: (1) the pre-training is initially completed on the text modal D pre of the image description data set, and the model is continuously adapted on a series of new text domains {D1, D2,..., D T}, and at each time step t, further optimization is performed on the current domain D t , so that the updated model can have effective generalization ability on each domain {D pre , D1, D2,..., D T} that has been seen previously and the newly contacted domain D t ; (2) the "text retrieval based on synthetic images" is introduced into zero-shot image description, and similar sentence retrieval and fusion are performed using synthetic images to alleviate the modal gap between text conditional training and image conditional reasoning; (3) the "text domain memory retrieval" mechanism maintains the dynamic buffer of previous text domain knowledge and performs adaptive retrieval, which effectively suppresses catastrophic forgetting in the continuous zero-shot description process.

[0054] The present application will be further described below in conjunction with the accompanying drawings of the specification. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The present application relates to the structure of a network.

[0056] Figure 2 The present application relates to the structure of a network.

[0057] Figure 3 The present application relates to the structure of a network.

[0058] Figure 4 The present application relates to the structure of a network. DETAILED DESCRIPTION

[0059] In conjunction with Figure 1 , a description generation method based on continuous zero-shot includes a training phase (Training in the figure) and an inference phase (Inference in the figure). The training phase includes the following steps:

[0060] Step S100, define the zero-shot image description data set ZCCL of continuous learning as a pure text training corpus for continuous adaptation, and sort the pure text training corpus according to the difficulty of continuous learning to obtain {D1, D2,..., D T}, the text corpus is encoded by a CLIP text encoder; a model at each training step t can only access D t corpus;

[0061] At step S200, for the t-th learning step, the input text Text of the corpus D t is used to generate a synthetic image by a diffusion model;

[0062] At step S300, the synthetic image CLIP image embedding is extracted using a CLIP image encoder (CLIP Image Encoder in the figure) ;

[0063] At step S400, the image embedding is used to retrieve a set of semantically similar text description embeddings in the text corpus, denoted as R (Synthetic Image-based Text Retrieval in the figure);

[0064] At step S500, the and R are input into a fusion module network (Fusion Module, FM in the figure), denoted as , where is the parameter set of the fusion module, and the output of the fusion module is defined as a soft prompt for a large language model network (LLM in the figure);

[0065] At step S600, for the input text Text, key entities are extracted from the text Text by word frequency statistics to construct a hard prompt for the large language model;

[0066] At step S700, the hard prompt and the soft prompt are input into the large language model network, and the attention mechanism is used to focus on the key parts in the input, denoted as , is the parameter set of the LLM;

[0067] At step S800, the learning parameter set Ψ t of the CLIP image encoder, the learning parameter set of the fusion module network, and the learning parameter set of the large language model network are stored in the text domain memory (Text-domain Memory in the figure).

[0068] In step S100, the ZCCL is derived from existing image-text resources, covering three types of tasks, namely natural image description, video description, and remote sensing image description, as shown in Figure 2 .

[0069] In step S200, given a zero-shot image description (ZC) model f (parameters are θf ), which is initially pre-trained on the text modality D pre (D1=D pre ) and aims to continuously adapt the model on a series of new text domains {D2,...,D T};

[0070] In step S200, the process of generating a synthetic image using the diffusion model includes:

[0071] In step S201, the text encoder of CLIP is used to convert the input text Text into a text feature vector, which serves as conditional information to guide the image generation process.

[0072] In step S202, starting from a random noise image, noise is gradually added to the image until the image is completely covered by noise.

[0073] In step S203, starting from a completely noisy image, noise is gradually removed through the reverse diffusion process until a satisfactory quality is achieved, generating a synthetic image that matches the content of the input text Text. In each step of the reverse diffusion, the text feature vector is used to guide the image generation process, ensuring that the generated image matches the content of the input text Text.

[0074] In step S400, the cosine similarity between and each text description vector in the D t corpus is calculated; the top ρ text description embeddings are selected based on similarity.

[0075] In step S500, the fusion module network structure is as shown in Figure 3 , and LoRA-based memory units are introduced in the linear layer and QKV projection layer of the fusion module network (in the figure ). The specific process is as follows:

[0076] In step S501, linear transformation is performed on and R by linear layers f1 and f2 to obtain f1( ) and f2(R);

[0077] In step S502, let the transformation matrix of the linear layer be W, and augment f1( ) and f2(R) through the LoRA memory unit to obtain and

[0078]

[0079] where I e =f1( ), Re = f2(R), A, B represent the dimension reduction and dimension increase projection layers of LoRA respectively;

[0080] Step S503, in each M*cross attention layer (Mx Cross Attention layers in the figure), the query Q, the key K and the value V are augmented in low rank according to the LoRA principle to obtain F CL

[0081]

[0082] ,

[0083] Wherein, Q CL , K CL , V CL are the query (Query), key (Key) and value (Value) matrices in the cross attention mechanism;

[0084] Step S504, the F CL is concatenated with the learnable parameter L p , in each L*mapping layer (Lx Mapping layers in the figure), the query Q, the key K and the value V of each self-attention are augmented according to the LoRA principle to obtain F ML

[0085]

[0086] ,

[0087] Wherein, Q ML , K ML , V ML are the query (Query), key (Key) and value (Value) matrices in the self-attention mechanism;

[0088] Step S505, the embedding of the same length as L p is extracted from F ML , as a soft prompt of the large language model.

[0089] Step S600 specifically includes:

[0090] Step S601, for the input text Text, obtain the text embedding to remove irrelevant characters in the text, and divide the text into words or word groups; irrelevant characters are punctuation marks, special symbols, etc.

[0091] Step S602, apply a named entity recognition (NER) tool to automatically identify entities (parts of speech) in the text, such as names, places, organizations, etc.

[0092] Step S603, count the number of occurrences of each entity in the text, and sort them from high to low according to the number of occurrences;

[0093] Step S604, set a threshold, and select entities with occurrence numbers exceeding the threshold as key entities;

[0094] Step S605, integrate the selected key entities into a discrete string;

[0095] Step S606, tokenize the discrete string and send it to the CLIP text encoder (CLIP TextEncoder in the figure) to obtain hard prompts F HP .

[0096] In step S700, the large language model adopts the GPT-2 model, and inserts the LoRA module in each self-attention block and the feedforward layer to realize efficient domain adaptation (as shown in Figure 4 By equipping each layer of GPT-2 with a low-rank LoRA matrix, the network can dynamically adjust its attention to match the semantic nuances of different text domains. Specifically, for a given GPT-2 self-attention layer GPTSelfAttn, the calculation

[0097]

[0098] ;

[0099] For the GPT-2 feedforward layer GPTFeedForward, the calculation is

[0100]

[0101]

[0102] where W FF is the first layer weight matrix of the feedforward network layer.

[0103] By modulating each projection, each layer of GPT-2 can generate an output that is consistent in style with the current domain while continuing to learn knowledge from previous domains.

[0104] Describe the input image. The description stage includes the following steps:

[0105] Step S110, encode the input image using the CLIP image encoder to obtain the image embedding I;

[0106] Step S210, perform image-to-text retrieval on the image embedding I to obtain a similar text embedding Ts;

[0107] Step S210, querying the CLIP image embedding of the synthesized image closest to the image embedding I in the text domain memory ;

[0108] Step S310, obtaining the CLIP image embedding of the synthesized image corresponding {Ψ t , , }, used to guide the fusion module network and the large language model network;

[0109] Step S410, injecting the image embedding I and the text embedding Ts into the FM to obtain a soft prompt;

[0110] Step S510, extracting key entities from the text embedding Ts to construct a hard prompt through word frequency statistics;

[0111] Step S610, inputting the hard prompt and the soft prompt into the large language model network to obtain a final image description.

Claims

1. A description generation method based on persistent zero samples, characterized in that, It includes a training phase and an inference phase. The training phase includes: Step S100: Define the zero-shot image description dataset ZCCL for continuous learning as the pure text training corpus for continuous adaptation. Sort the pure text training corpus according to the difficulty of continuous learning to obtain {D1, D2, ..., D...} T The CLIP text encoder is used to encode the text in the text corpus. Step S200, assume that D can only be accessed in each training step t. t The corpus, for the t-th learning step, for D t The input text (Text) from the corpus is used to generate a synthetic image using a diffusion model; Step S300: Use the CLIP image encoder to extract the CLIP image embedding of the synthesized image. ; Step S400, using image embedding Retrieve a set of semantically similar text description embeddings R from a text corpus; Step S500, will The input to the R fusion module network is used to define the output of the fusion module as a soft cue for the large language model. Step S600: For the input text Text, extract key entities from the text Text through word frequency statistics to construct soft and hard prompts for the large language model; Step S700: Input hard cues and soft cues into the large language model network, and use an attention mechanism to focus on key parts of the input; Step S800: The learning parameter set Ψ of the CLIP image encoder is... t The set of learning parameters for fusion module networks Large Language Model Network Learning Parameter Set Stored in the text field memory; The reasoning stage includes: Step S110: Encode the input image using the CLIP image encoder to obtain image embedding I; Step S210: Perform image-to-text retrieval on image embedding I to obtain similar text embeddings Ts; Step S210: Query the CLIP image embedding of the synthetic image that is closest to image embedding I in the text domain memory. ; Step S310: Obtain the CLIP image embedding of the synthesized image. The corresponding {Ψ t , , }, used to guide the fusion module network and the large language model network; Step S410: Inject the image embedding I and the text embedding Ts into FM to obtain a soft cue; Step S510: Extract key entities from text embedding Ts through word frequency statistics to construct hard cues; Step S610: Input the hard cues and soft cues into the large language model network to obtain the final image description.

2. The method according to claim 1, characterized in that, In step S200, the process of generating a synthetic image using a diffusion model includes: Step S201: Using CLIP's text encoder, the input text Text is converted into a text feature vector; Step S202: Starting with a random noisy image, gradually add noise to the image until the image is completely covered by noise; Step S203: Starting from a completely noisy image, noise is gradually removed through a reverse diffusion process until a satisfactory quality is achieved, generating a synthetic image that matches the input text content.

3. The method according to claim 1, characterized in that, In step S400, calculation is performed in the CLIP embedding space. With D t For each text description vector in the corpus, the cosine similarity is used to select the top ρ text description embeddings based on the highest similarity.

4. The method according to claim 1, characterized in that, In step S500, LoRA-based memory units are introduced into the linear layer and QKV projection layer of the fusion module network. The specific process of step S500 is as follows: Step S501, the linear layers f1 and f2 are paired A linear transformation of R yields f1( f1 and f2(R); Step S502, let W be the transformation matrix of the linear layer, for f1( f1 and f2(R) are augmented through LoRA memory cells. and , ， Among them I e =f1( ), R e =f2(R), where A and B represent the dimension reduction and dimension increase projection layers of LoRA, respectively; Step S503: In each M* cross-attention layer, perform low-rank augmentation on the query, key, and value according to the LoRA principle to obtain F. CL ， ， Among them, Q CL K CL V CL This refers to the query, key, and value matrix in the cross-attention mechanism. Step S504, F CL With learnable parameter L p Cascade the layers, and in each L* mapping layer, augment the query, key, and value of each self-attention layer according to the LoRA principle to obtain F. ML ， ， Among them, Q ML K ML V ML This represents the query, key, and value matrix in a self-attention mechanism. Step S505, from F ML Extracting L p Equal-length embeddings serve as soft prompts for large language models.

5. The method according to claim 4, characterized in that, Step S600 specifically includes: Step S601: For the input text Text, obtain the text embedding, remove irrelevant characters from the text, and segment the text into words or phrases; Step S602: Apply a named entity recognition tool to automatically identify entities in the text; Step S603: Count the number of times each entity appears in the text and sort them from highest to lowest according to the number of occurrences; Step S604: Set a threshold and select entities that appear more than the threshold as key entities; Step S605: Integrate the selected key entities into a discrete string; Step S606: Segment the discrete string and feed it into the CLIP text encoder to obtain the hard clue F. HP .

6. The method according to claim 5, characterized in that, In step S700, the large language model adopts the GPT-2 model, and LoRA modules are inserted into each self-attention block and feedforward layer of GPT-2; for a given GPT-2 self-attention layer GPTSelfAttn, the calculation is performed. ， ; For the GPT-2 feedforward layer GPTFTeedForward, the calculation is as follows: ， ， Among them, W FF It is the weight matrix of the first layer of the feedforward network.

Images