Zero sample image description generation method based on CSC and prompt learning

Abstract

The invention discloses a zero sample image description generation method based on CSC and prompt learning, and the method comprises the steps: constructing a two-layer image sparse feature extraction model based on a convolution sparse coding thought, reducing interference features through extracting sparse features, and then importing an image encoder to further extract features. The obtained features of the image pass through a text decoder to obtain category hard prompts of the image, and finally the category hard prompts of the image and the image features are input into a large language model for image description generation. According to the method, key information in an original image can be fully utilized when the image description is generated by the large language model through image sparse feature extraction and class hard prompt addition, interference of background features in the image is reduced, the illusion problem of the large model is relieved, and a better image description generation effect can be obtained in an open world environment.

Description

Technical Field

[0001] The invention belongs to the field of image description generation in artificial intelligence, and in particular relates to a zero-sample image description (Image Captioning) generation method based on CSC (Convolutional Sparse Coding) and prompt learning. Background Art

[0002] Image captioning is the task of automatically generating natural language descriptions of images, which lies at the intersection of computer vision and natural language processing. Its goal is to enable machines to understand the visual content of an image and translate this understanding into coherent descriptive text. This capability involves not only recognizing objects and scenes in an image, but also understanding the relationships between these elements and expressing them in a meaningful way. Image captioning usually relies on deep learning techniques, using Convolutional Neural Networks (CNNs) for visual feature extraction and Recurrent Neural Networks (RNNs) or Transformers for text generation. However, these image captioning models rely heavily on large annotated datasets where each image is paired with a corresponding description. These datasets, such as MS COCO and Flickr30k, have driven significant progress, but their content coverage is limited. Adding detailed descriptions to images requires a lot of human resources and costs, which makes it impractical to cover the huge diversity of visual content in the real world.

[0003] With the development of large pre-trained visual-language models (VLMs) like CLIP and ALIGN, and large language models (LLMs) like GPT, zero-shot learning has become a hot topic. This has spawned many zero-shot image-to-text generation models. Zero-shot image description addresses the limitations of datasets by leveraging models that can generalize to unseen data, thereby expanding the applicability and scalability of image description systems. However, current zero-shot image description models perform poorly when dealing with images with complex backgrounds. The complexity of the background may interfere with the model's ability to correctly identify the subject of the image, causing the generated description to deviate from the actual content of the image. Summary of the invention

[0004] In order to solve the shortcomings of the prior art, the present invention aims to provide a zero-shot image description (Image Captioning) generation method based on CSC (convolutional sparse coding) and prompt learning, which uses the unique advantages of CSC to show excellent sparse feature extraction effects, and extracts key information from the original image to the maximum extent by adding category hard prompts, greatly improving the accuracy of zero-shot image description generation. It is beneficial to improve the effects of all fields that require "picture description" in practical applications, such as smart medical care, human-computer interaction, image content management and retrieval, and auxiliary judgment.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes a zero-shot image description generation method based on CSC and prompt learning, comprising the following steps:

[0007] 1) Add noise to the images in the DIV2k dataset and perform data augmentation operations to use them as training sets;

[0008] 2) Based on convolutional sparse coding, a two-layer image sparse feature extraction model is constructed;

[0009] 3) Using the image sparse feature extraction model established in step 2) to train the images of the training set, calculating the loss function value of the network model for each iteration until the loss function value converges to a minimum, and saving the training model;

[0010] 4) Load the trained image sparse feature extraction model, perform sparse feature extraction on the test image and import it into the image encoder to further extract features;

[0011] 5) The features of the obtained image are passed through a text decoder to obtain a hard hint of the image category;

[0012] 6) Input the image category hard hints and image features into the large language model to generate image description.

[0013] Further, step 1) specifically includes:

[0014] 1-1) Import the DIV2k dataset;

[0015] 1-2) Add different degrees of Gaussian noise to the data set and save it;

[0016] 1-3) Read the image after adding noise, deblur the image using the Lucy-Richardson algorithm, and adjust the image size.

[0017] Furthermore, the specific calculation method of the Lucy-Richardson algorithm for image deblurring in step (1-3) is:

[0018]

[0019] Where f(x, y) represents the original image, n is the number of iterations, g(x, y) represents the degraded image, h(x, y) is the spatial description of the degradation function, and the symbol “*” represents spatial convolution; f n+1 (x,y) means f(x,y) is iterated n+1 times, f n (x,y) means f(x,y) is iterated n times, and T is the matrix transpose symbol.

[0020] Further, step 2) specifically includes:

[0021] First, a convolutional sparse coding module is constructed; the input signal is represented as follows through convolutional sparse coding:

[0022]

[0023] Where x∈R L×W×C is the input signal, L×W represents the spatial dimension of the input signal, R is a real number set, and C is the number of channels of the input signal; d a represents the ath atom in the dictionary, which corresponds to a convolution kernel, s a is the corresponding sparse coding coefficient;

[0024] The optimization goal at this time is to minimize the following loss function:

[0025]

[0026] Where λ is the sparsity parameter;

[0027] Let D = (d1, d2, ..., d C ), Γ=(s1,s2,…,s C ) T ; d1,d2,…,d C They represent atoms in the dictionary, and D represents a vector composed of atoms; s1, s2, …, s C Respectively represent sparse coding coefficients, Γ represents the vector composed of them;

[0028] The above formula can be simplified to:

[0029]

[0030] Among them, the first term represents the reconstruction error, and the second term represents the sparsity constraint;

[0031] The optimization target is solved using the fast soft threshold iterative algorithm FISTA. The FISTA algorithm consists of two steps: the first step is to calculate a new point along the calculation direction of the first two steps, and the second step is to perform an approximate point gradient iteration at the new point, that is,

[0032]

[0033] Where k represents the kth iteration of the FISTA algorithm, represents the soft threshold operator of h, x k is the result of the algorithm’s kth iteration, x k-1 is the result of the algorithm's k-1th iteration, x k-2 is the result of the algorithm's k-2th iteration; y k It represents the weighted sum of the algorithm's k-1th iteration result and the k-2th iteration result, t k represents the step size of algorithm iteration, represents the derivative operation;

[0034] Use the two-step FISTA algorithm to solve the optimization objective and take the initial Γ value Γ 0 = 0 to initialize the Γ value Γ of the first iteration 1 :

[0035]

[0036] Where X is the input image, D T X is equivalent to performing a convolution operation. For D T DX, DX is equivalent to a deconvolution operation, D T is the transpose of D; then multiply it by D T This is equivalent to performing another convolution operation, S β / L Equivalent to the activation function ReLU;

[0037] Thus, the basic convolutional sparse coding module is obtained. Then, two convolutional sparse coding modules are superimposed and a BN layer is added in the middle to obtain the image sparse feature extraction model. The input image is X and the output sparse feature is Y. The process can be expressed by the following formula:

[0038] Y=CSC(BN(CSC(X))).

[0039] Further, step 3) specifically includes:

[0040] 3-1) Read the training set in step 1) and adjust the image size;

[0041] 3-2) Set the number of iterations required for the training set to p, and the number of images read in at a time to q, where p≥1, q≥1;

[0042] 3-3) Import the noisy image into the image sparse feature extraction model to obtain sparse features and calculate the pixel difference between it and the original image, and use MSE as the loss function to calculate the loss value in each round;

[0043] 3-4) Repeat step 3-3) until the loss function value is minimized and save the trained model.

[0044] Furthermore, the category hard prompt generated in step 5) is “This is a image of (category)”.

[0045] The present invention has the following beneficial effects:

[0046] 1. The zero-shot image description generation method based on convolutional sparse coding and prompt learning proposed in the present invention applies convolutional sparse coding to image description generation for the first time, which can improve the accuracy of image description generation and enhance the model's recognition of key image information.

[0047] 2. The present invention can effectively extract key features in the image and thus reduce the interference of background features in the image on downstream image tasks, thereby enhancing the model's ability to capture image details and key information.

[0048] 3. The present invention adds category hard prompts to enable the large language model to make more full use of image information and alleviate the large model hallucination problem when generating image descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a zero-shot image description generation method based on convolutional sparse coding and prompt learning according to the present invention;

[0050] Figure 2 A comparison chart of the effects of generating the image description of the present invention. DETAILED DESCRIPTION

[0051] like Figure 1-2 As shown, the present invention proposes a zero-shot image description generation method based on convolutional sparse coding and prompt learning, which uses the unique advantages of CSC to show excellent sparse feature extraction effect to construct an image sparse feature model, and extracts key information in the original image to the maximum extent by adding category hard prompts, greatly improving the accuracy of zero-shot image description generation and alleviating the large model hallucination problem. It is beneficial to improve the effects of all fields that require "picture description" in practical applications, such as smart medical care, human-computer interaction, image content management and retrieval, and auxiliary judgment. The method specifically includes the following steps:

[0052] 1) Add noise to the images in the DIV2k dataset and perform data augmentation operations to use them as training sets. Specifically:

[0053] 1-1) Import the DIV2k dataset;

[0054] 1-2) Add different degrees of Gaussian noise to the data set and save it;

[0055] 1-3) Read the image after adding noise, deblur the image using the Lucy-Richardson algorithm, and adjust the image size.

[0056] Among them, the specific calculation method of Lucy-Richardson algorithm for image deblurring is:

[0057]

[0058] Where f(x,y) represents the original image, n is the number of iterations, g(x,y) represents the degraded image, h(x,y) is the spatial description of the degradation function, and the symbol “*” represents spatial convolution. n+1 (x,y) means f(x,y) is iterated n+1 times, f n (x,y) means f(x,y) is iterated n times, and T is the matrix transpose symbol.

[0059] 2) Based on convolutional sparse coding, a two-layer image sparse feature extraction model is constructed. Specifically, it includes:

[0060] First, a convolutional sparse coding module is constructed; the input signal is represented as follows through convolutional sparse coding:

[0061]

[0062] Where x∈R L×W×C is the input signal, L×W represents the spatial dimension of the input signal, R is a real number set, and C is the number of channels of the input signal; d a represents the ath atom in the dictionary, which corresponds to a convolution kernel, s a is the corresponding sparse coding coefficient;

[0063] The optimization goal at this time is to minimize the following loss function:

[0064]

[0065] Where λ is the sparsity parameter;

[0066] Let D = (d1, d2, ..., d C ), Γ=(s1,s2,…,s C ) T ; d1,d2,…,d C They represent atoms in the dictionary, and D represents a vector composed of atoms; s1, s2, …, sC Respectively represent sparse coding coefficients, Γ represents the vector composed of them;

[0067] The above formula can be simplified to:

[0068]

[0069] Among them, the first term represents the reconstruction error, and the second term represents the sparsity constraint;

[0070] The optimization target is solved using the fast soft threshold iterative algorithm FISTA. The FISTA algorithm consists of two steps: the first step is to calculate a new point along the calculation direction of the first two steps, and the second step is to perform an approximate point gradient iteration at the new point, that is,

[0071]

[0072] Where k represents the kth iteration of the FISTA algorithm, represents the soft threshold operator of h, x k is the result of the algorithm’s kth iteration, x k-1 is the result of the algorithm's k-1th iteration, x k-2 is the result of the algorithm's k-2th iteration; y k It represents the weighted sum of the algorithm's k-1th iteration result and the k-2th iteration result, t k represents the step size of algorithm iteration, represents the derivative operation;

[0073] Use the two-step FISTA algorithm to solve the optimization objective and take the initial Γ value Γ 0 = 0 to initialize the Γ value Γ of the first iteration 1 :

[0074]

[0075] Where X is the input image, D T X is equivalent to performing a convolution operation. For D T DX, DX is equivalent to a deconvolution operation, D T is the transpose of D; then multiply it by D T This is equivalent to performing another convolution operation, S β / L Equivalent to the activation function ReLU;

[0076] Thus, the basic convolutional sparse coding module is obtained. Then, two convolutional sparse coding modules are superimposed and a BN layer is added in the middle to obtain the image sparse feature extraction model. The input image is X and the output sparse feature is Y. The process can be expressed by the following formula:

[0077] Y = CSC(BN(CSC(X))).

[0078] 3) Use the image sparse feature extraction model to train the images in the training set, calculate the loss function value of the network model for each iteration, until the loss function value converges to the minimum, and save the training model. Specifically include:

[0079] 3-1) Read the training set in step 1) and adjust the image size;

[0080] 3-2) Set the number of iterations required for the training set to p, and the number of images read in at a time to q, where p≥1, q≥1;

[0081] 3-3) Import the noisy image into the image sparse feature extraction model to obtain sparse features and calculate the pixel difference between it and the original image, and use MSE as the loss function to calculate the loss value in each round;

[0082] 3-4) Repeat step 3-3) until the loss function value is minimized and save the trained model.

[0083] 4) Load the trained image sparse feature extraction model, perform sparse feature extraction on the test image and import it into the image encoder to further extract features;

[0084] 5) The features of the obtained image are passed through a text decoder to obtain a hard prompt of the category of the image, wherein the generated hard prompt of the category is “This is a image of (category)”.

[0085] 6) Input the image category hard hints and image features into the large language model to generate image description.

[0086] The present invention constructs an image sparse feature extraction module based on the idea of ​​convolutional sparse coding, and by adding category hard prompts, the large language model can maximize the use of key information in the original image for zero-sample image description generation. The present invention uses the unique advantages of CSC to demonstrate excellent sparse feature extraction effects, and by adding category hard prompts to maximize the extraction of key information in the original image, greatly improving the accuracy of zero-sample image description generation. Using this set of zero-sample image description methods, a special method is used to extract the main features of the image, thereby reducing the interference of background features in the image, and better image description generation effects can be obtained in an open world environment, which is conducive to improving all practical applications that require "picture description" in all fields such as smart medical care, human-computer interaction, image content management and retrieval, and auxiliary judgment.

Claims

1. A zero-shot image description generation method based on CSC and hint learning, characterized by: The following steps are involved: 1) Add noise to the images in the DIV2k dataset and perform data augmentation operations to use them as training sets; 2) Based on convolutional sparse coding, a two-layer image sparse feature extraction model is constructed; 3) Using the image sparse feature extraction model established in step 2) to train the images of the training set, calculating the loss function value of the network model for each iteration until the loss function value converges to a minimum, and saving the training model; 4) Load the trained image sparse feature extraction model, perform sparse feature extraction on the test image and import it into the image encoder to further extract features; 5) The features of the obtained image are passed through a text decoder to obtain a hard hint of the image category; 6) Input the image category hard hints and image features into the large language model to generate image description.

2. The zero-shot image description generation method based on CSC and hint learning according to claim 1, characterized in that: Step 1) specifically includes: 1-1) Import the DIV2k dataset; 1-2) Add different degrees of Gaussian noise to the data set and save it; 1-3) Read the image after adding noise, deblur the image using the Lucy-Richardson algorithm, and adjust the image size.

3. The zero-shot image description generation method based on CSC and hint learning according to claim 2, characterized in that: The specific calculation method of the Lucy-Richardson algorithm for image deblurring in step (1-3) is: Where f(x,y) represents the original image, n is the number of iterations, g(x,y) represents the degraded image, h(x,y) is the spatial description of the degradation function, and the symbol "*" represents spatial convolution; f n+1 (x,y) means f(x,y) is iterated n+1 times, f n (x,y) means f(x,y) is iterated n times, and T is the matrix transpose symbol.

4. The method for generating zero-shot image description based on CSC and hint learning according to claim 1, characterized in that: Step 2) specifically includes: First, a convolutional sparse coding module is constructed; the input signal is represented as follows through convolutional sparse coding: Where x∈R L×W×C is the input signal, L×W represents the spatial dimension of the input signal, R is a real number set, and C is the number of channels of the input signal; d a represents the ath atom in the dictionary, which corresponds to a convolution kernel, s a is the corresponding sparse coding coefficient; The optimization goal at this time is to minimize the following loss function: Where λ is the sparsity parameter; Let D = (d1, d2, ..., d C ), Γ=(s1,s2,…,s C ) T ; d1,d2,…,d C They represent atoms in the dictionary, and D represents a vector composed of atoms; s1, s2, …, s C Respectively represent sparse coding coefficients, Γ represents the vector composed of them; The above formula can be simplified to: Among them, the first term represents the reconstruction error, and the second term represents the sparsity constraint; The optimization target is solved using the fast soft threshold iterative algorithm FISTA. The FISTA algorithm consists of two steps: the first step is to calculate a new point along the calculation direction of the first two steps, and the second step is to perform an approximate point gradient iteration at the new point, that is, Where k represents the kth iteration of the FISTA algorithm, represents the soft threshold operator of h, x k is the result of the algorithm’s kth iteration, x k-1 is the result of the algorithm's k-1th iteration, x k-2 is the result of the algorithm's k-2th iteration; y k represents the weighted sum of the algorithm's k-1th iteration result and k-2th iteration result, t k represents the step size of algorithm iteration, represents the derivative operation; Use the two-step FISTA algorithm to solve the optimization objective and take the initial Γ value Γ 0 = 0 to initialize the Γ value Γ of the first iteration 1 : Where X is the input image, D T X is equivalent to performing a convolution operation. For D T DX, DX is equivalent to a deconvolution operation, D T is the transpose of D; then multiply it by D T This is equivalent to performing another convolution operation, S β / L Equivalent to the activation function ReLU; Thus, the basic convolutional sparse coding module is obtained. Then, two convolutional sparse coding modules are superimposed and a BN layer is added in the middle to obtain the image sparse feature extraction model. The input image is X and the output sparse feature is Y. The process can be expressed by the following formula: Y=CSC(BN(CSC(X))).

5. The method for generating zero-shot image description based on CSC and prompt learning according to claim 1, characterized in that: Step 3) specifically includes: 3-1) Read the training set in step 1) and adjust the image size; 3-2) Set the number of iterations required for the training set to p, and the number of images read in at a time to q, where p≥1, q≥1; 3-3) Import the noisy image into the image sparse feature extraction model to obtain sparse features and calculate the pixel difference between it and the original image, and use MSE as the loss function to calculate the loss value in each round; 3-4) Repeat step 3-3) until the loss function value is minimized and save the trained model.

6. The method for generating zero-shot image description based on CSC and hint learning according to claim 1, characterized in that: The category hard prompt generated in step 5) is "This is aimage of (category).".

Images