A zero-shot image description generation method based on CSC and hint learning

Abstract

This invention discloses a zero-shot image description generation method based on convolutional sparse coding (CSC) and hint learning. This method, based on the concept of convolutional sparse coding, constructs a two-layer image sparse feature extraction model. Sparse features are extracted to reduce interfering features, which are then fed into an image encoder for further feature extraction. The resulting image features are then passed through a text decoder to obtain hard category hints for the image. Finally, the image category hints and image features are fed into a large language model for image description generation. By extracting sparse features and incorporating hard category hints, this method enables the large language model to fully utilize key information in the original image when generating image descriptions, reducing interference from background features in the image and alleviating the hallucination problem of the large model. This method can achieve better image description generation in an open-world environment.

Description

Technical Field

[0001] The present invention belongs to the field of image description generation in artificial intelligence, and in particular relates to a zero-shot image description (Image Captioning) generation method based on CSC (Convolutional Sparse Coding) and prompt learning. Background Art

[0002] Image captioning, the task of automatically generating natural language descriptions of images, lies at the intersection of computer vision and natural language processing. Its goal is to enable machines to understand the visual content of an image and translate this understanding into coherent descriptive text. This capability involves not only identifying objects and scenes in an image, but also understanding the relationships between these elements and expressing them in a meaningful way. Image captioning typically relies on deep learning techniques, utilizing convolutional neural networks (CNNs) for visual feature extraction and recurrent neural networks (RNNs) or transformers for text generation. However, these image captioning models rely heavily on large, annotated datasets in which each image is accompanied by a corresponding description. These datasets, such as MS COCO and Flickr30k, have driven significant progress, but their content coverage is limited. Adding detailed descriptions to images requires significant human resources and costs, making it impractical to capture the vast diversity of real-world visual content.

[0003] With the development of large pre-trained visual language models (VLMs) such as CLIP and ALIGN, as well as large language models (LLMs) such as GPT, zero-shot learning has become a hot topic. This has spawned a number of zero-shot image-to-text generation models. Zero-shot image captioning addresses dataset limitations by leveraging models that can generalize to unseen data, thereby expanding the applicability and scalability of image captioning systems. However, current zero-shot image captioning models perform poorly when handling images with complex backgrounds. The complexity of the background may interfere with the model's ability to correctly identify the image subject, causing the generated description to deviate from the actual content of the image. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this paper aims to provide a zero-shot image captioning method based on CSC (Convolutional Sparse Coding) and hint learning. This method leverages the unique advantages of CSC to achieve excellent sparse feature extraction. By incorporating category hints, it maximizes the extraction of key information from the original image, significantly improving the accuracy of zero-shot image captioning. This approach is beneficial for improving the effectiveness of practical applications in all areas requiring "image description," such as smart healthcare, human-computer interaction, image content management and retrieval, and aided decision-making.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes a zero-shot image description generation method based on CSC and hint learning, comprising the following steps:

[0007] 1) Add noise to the images in the DIV2k dataset and perform data augmentation operations to use them as training sets;

[0008] 2) Based on convolutional sparse coding, a two-layer image sparse feature extraction model is constructed;

[0009] 3) Using the image sparse feature extraction model established in step 2) to train the images of the training set, calculating the loss function value of the network model for each iteration until the loss function value converges to a minimum, and saving the training model;

[0010] 4) Load the trained image sparse feature extraction model, perform sparse feature extraction on the test image and import it into the image encoder to further extract features;

[0011] 5) The features of the obtained image are passed through the text decoder to obtain the hard hint of the image category;

[0012] 6) Input the image category hard hints and image features into the large language model to generate image description.

[0013] Furthermore, step 1) specifically includes:

[0014] 1-1) Import the DIV2k dataset;

[0015] 1-2) Add different degrees of Gaussian noise to the data set and save it;

[0016] 1-3) Read the image after adding noise, deblur the image using the Lucy-Richardson algorithm, and adjust the image size.

[0017] Furthermore, the specific calculation method of the Lucy-Richardson algorithm for image deblurring in steps (1-3) is:

[0018]

[0019] Where f(x,y) represents the original image, n is the number of iterations, g(x,y) represents the degraded image, h(x,y) is the spatial description of the degradation function, and the symbol “*” represents spatial convolution; f n+1 (x,y) means f(x,y) iterates n+1 times, f n (x,y) represents f(x,y) iteration n times, and T is the matrix transpose symbol.

[0020] Furthermore, step 2) specifically includes:

[0021] First, a convolutional sparse coding module is constructed; the input signal is represented as follows through convolutional sparse coding:

[0022]

[0023] Where x∈R L×W×C is the input signal, L×W represents the spatial dimension of the input signal, R is a set of real numbers, and C is the number of channels of the input signal; d a represents the ath atom in the dictionary, which corresponds to a convolution kernel, s a is the corresponding sparse coding coefficient;

[0024] The optimization goal at this time is to minimize the following loss function:

[0025]

[0026] Where λ is the sparsity parameter;

[0027] Let D=(d1,d2,…,d C ), Γ=(s1,s2,…,s C ) T ; d1,d2,…,d C They represent atoms in the dictionary, and D represents a vector composed of atoms; s1, s2, ..., s C Represent sparse coding coefficients respectively, and Γ represents the vector composed of them;

[0028] The above formula can be simplified to:

[0029]

[0030] Among them, the first term represents the reconstruction error, and the second term represents the sparsity constraint;

[0031] The optimization objective is solved using the fast soft threshold iterative algorithm FISTA. The FISTA algorithm consists of two steps: the first step is to calculate a new point along the calculation direction of the first two steps, and the second step is to perform an approximate point gradient iteration at the new point, that is,

[0032]

[0033] Where k represents the kth iteration of the FISTA algorithm, represents the soft threshold operator of h, x k is the result of the kth iteration of the algorithm, x k-1 is the result of the k-1th iteration of the algorithm, x k-2 is the result of the k-2th iteration of the algorithm; y k It represents the weighted sum of the k-1th iteration result and the k-2th iteration result of the algorithm, t k represents the step size of the algorithm iteration, represents the derivative operation;

[0034] Use the two-step FISTA algorithm to solve the optimization objective and take the initial Γ value Γ 0 = 0 to initialize the Γ value Γ of the first step of the iteration 1 :

[0035]

[0036] Where X is the input image, D T X is equivalent to performing a convolution operation, for D T DX, DX is equivalent to a deconvolution operation, D T is the transpose of D; then multiply it by D T This is equivalent to performing another convolution operation, S β / L Equivalent to the activation function ReLU;

[0037] Thus, the basic convolutional sparse coding module is obtained. Then, by superimposing two convolutional sparse coding modules and adding a BN layer in the middle, the image sparse feature extraction model is obtained. The input image is X and the output sparse feature is Y. The process can be expressed by the following formula:

[0038] Y=CSC(BN(CSC(X))).

[0039] Furthermore, step 3) specifically includes:

[0040] 3-1) Read the training set from step 1) and adjust the image size;

[0041] 3-2) Set the number of iterations required for the training set to p and the number of images read in at a time to q, where p ≥ 1 and q ≥ 1;

[0042] 3-3) Import the noisy image into the image sparse feature extraction model to obtain sparse features and calculate the pixel difference between the sparse features and the original image. Use MSE as the loss function to calculate the loss value in each round.

[0043] 3-4) Repeat step 3-3) until the loss function value is minimized and save the trained model.

[0044] Furthermore, the category hard prompt generated in step 5) is “This is a image of (category).”

[0045] The beneficial effects of the present invention are:

[0046] 1. The zero-shot image description generation method based on convolutional sparse coding and prompt learning proposed in this paper applies convolutional sparse coding to image description generation for the first time, which can improve the accuracy of image description generation and enhance the model's recognition of key image information.

[0047] 2. The present invention can effectively extract key features in the image and thus reduce the interference of background features in the image on downstream image tasks, thereby enhancing the model's ability to capture image details and key information.

[0048] 3. By adding category hard prompts, the present invention enables the large language model to make more full use of image information and alleviate the large model hallucination problem when generating image descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Flowchart of the zero-shot image description generation method based on convolutional sparse coding and hint learning of the present invention;

[0050] Figure 2 Generate an effect comparison chart for the image description of the present invention. DETAILED DESCRIPTION

[0051] like Figure 1-2 As shown, the present invention proposes a zero-shot image description generation method based on convolutional sparse coding and prompt learning. It utilizes the unique advantages of CSC to demonstrate excellent sparse feature extraction effects to construct an image sparse feature model. By adding category hard prompts, it maximizes the extraction of key information from the original image, greatly improving the accuracy of zero-shot image description generation and alleviating the problem of large model hallucination. This is beneficial for improving the effects of all practical applications that require "image description" in fields such as smart medical care, human-computer interaction, image content management and retrieval, and auxiliary judgment technologies. The method specifically includes the following steps:

[0052] 1) Add noise to the images in the DIV2k dataset and perform data augmentation operations to use as a training set. Specifically:

[0053] 1-1) Import the DIV2k dataset;

[0054] 1-2) Add different degrees of Gaussian noise to the data set and save it;

[0055] 1-3) Read the image after adding noise, deblur the image using the Lucy-Richardson algorithm, and resize the image.

[0056] Among them, the specific calculation method of the Lucy-Richardson algorithm for image deblurring is:

[0057]

[0058] Where f(x,y) represents the original image, n is the number of iterations, g(x,y) represents the degraded image, h(x,y) is the spatial description of the degradation function, and the symbol “*” represents spatial convolution. n+1 (x,y) means f(x,y) iterates n+1 times, f n (x,y) represents f(x,y) iteration n times, and T is the matrix transpose symbol.

[0059] 2) Based on convolutional sparse coding, a two-layer image sparse feature extraction model is constructed. Specifically, it includes:

[0060] First, a convolutional sparse coding module is constructed; the input signal is represented as follows through convolutional sparse coding:

[0061]

[0062] Where x∈R L×W×C is the input signal, L×W represents the spatial dimension of the input signal, R is a set of real numbers, and C is the number of channels of the input signal; d a represents the ath atom in the dictionary, which corresponds to a convolution kernel, s a is the corresponding sparse coding coefficient;

[0063] The optimization goal at this time is to minimize the following loss function:

[0064]

[0065] Where λ is the sparsity parameter;

[0066] Let D=(d1,d2,…,d C ), Γ=(s1,s2,…,s C ) T ; d1, d2, …, d C They represent atoms in the dictionary, and D represents a vector composed of atoms; s1, s2, ..., sC Represent sparse coding coefficients respectively, and Γ represents the vector composed of them;

[0067] The above formula can be simplified to:

[0068]

[0069] Among them, the first term represents the reconstruction error, and the second term represents the sparsity constraint;

[0070] The optimization objective is solved using the fast soft threshold iterative algorithm FISTA. The FISTA algorithm consists of two steps: the first step is to calculate a new point along the calculation direction of the first two steps, and the second step is to perform an approximate point gradient iteration at the new point, that is,

[0071]

[0072] Where k represents the kth iteration of the FISTA algorithm, represents the soft threshold operator of h, x k is the result of the kth iteration of the algorithm, x k-1 is the result of the k-1th iteration of the algorithm, x k-2 is the result of the k-2th iteration of the algorithm; y k It represents the weighted sum of the k-1th iteration result and the k-2th iteration result of the algorithm, t k represents the step size of the algorithm iteration, represents the derivative operation;

[0073] Use the two-step FISTA algorithm to solve the optimization objective and take the initial Γ value Γ 0 = 0 to initialize the Γ value Γ of the first step of the iteration 1 :

[0074]

[0075] Where X is the input image, D T X is equivalent to performing a convolution operation, for D T DX, DX is equivalent to a deconvolution operation, D T is the transpose of D; then multiply it by D T This is equivalent to performing another convolution operation, S β / L Equivalent to the activation function ReLU;

[0076] Thus, the basic convolutional sparse coding module is obtained. Then, by superimposing two convolutional sparse coding modules and adding a BN layer in the middle, the image sparse feature extraction model is obtained. The input image is X and the output sparse feature is Y. The process can be expressed by the following formula:

[0077] Y = CSC(BN(CSC(X))).

[0078] 3) Use the image sparse feature extraction model to train the images in the training set, calculate the loss function value of the network model for each iteration, until the loss function value converges to the minimum, and save the training model. Specifically include:

[0079] 3-1) Read the training set from step 1) and adjust the image size;

[0080] 3-2) Set the number of iterations required for the training set to p and the number of images read in at a time to q, where p ≥ 1 and q ≥ 1;

[0081] 3-3) Import the noisy image into the image sparse feature extraction model to obtain sparse features and calculate the pixel difference between the sparse features and the original image. Use MSE as the loss function to calculate the loss value in each round.

[0082] 3-4) Repeat step 3-3) until the loss function value is minimized and save the trained model.

[0083] 4) Load the trained image sparse feature extraction model, perform sparse feature extraction on the test image and import it into the image encoder to further extract features;

[0084] 5) The features of the obtained image are passed through a text decoder to obtain a hard category prompt of the image, where the generated hard category prompt is "This is an image of (category)."

[0085] 6) Input the image category hard hints and image features into the large language model to generate image description.

[0086] The present invention constructs an image sparse feature extraction module based on the idea of ​​convolutional sparse coding, and by adding category hard prompts, maximizes the use of key information in the original image by the large language model for zero-sample image description generation. The present invention utilizes the unique advantages of CSC to demonstrate excellent sparse feature extraction effects, and by adding category hard prompts to maximize the extraction of key information in the original image, greatly improving the accuracy of zero-sample image description generation. Using this set of zero-sample image description methods, a special method is used to extract the main features of the image, thereby reducing the interference of background features in the image, and better image description generation effects can be obtained in an open world environment, which is beneficial to improving all practical applications that require "speaking by pictures" in fields such as smart medical care, human-computer interaction, image content management and retrieval, and auxiliary judgment technologies.

Claims

1. A zero-shot image description generation method based on CSC and hint learning, characterized by: The following steps are involved: 1) Add noise to the images in the DIV2k dataset and perform data augmentation operations to use them as training sets; 2) Based on convolutional sparse coding, a two-layer image sparse feature extraction model is constructed; 3) Using the image sparse feature extraction model established in step 2) to train the images of the training set, calculating the loss function value of the network model for each iteration until the loss function value converges to a minimum, and saving the training model; 4) Load the trained image sparse feature extraction model, perform sparse feature extraction on the test image and import it into the image encoder to further extract features; 5) The features of the obtained image are passed through the text decoder to obtain the hard hint of the image category; 6) Input the image category hard hint and image features into the large language model to generate image description; Step 2) specifically includes: First, a convolutional sparse coding module is constructed; the input signal is represented as follows through convolutional sparse coding: Where x∈R L×W×C is the input signal, L×W represents the spatial dimension of the input signal, R is a set of real numbers, and C is the number of channels of the input signal; d a represents the ath atom in the dictionary, which corresponds to a convolution kernel, s a is the corresponding sparse coding coefficient; The optimization goal at this time is to minimize the following loss function: Where λ is the sparsity parameter; Let D=(d1,d2,…,d C ), Γ=(s1,s2,…,s C ) T ; d1, d2, …, d C They represent atoms in the dictionary, and D represents a vector composed of atoms; s1, s2, ..., s C Represent sparse coding coefficients respectively, and Γ represents the vector composed of them; The above formula can be simplified to: Among them, the first term represents the reconstruction error, and the second term represents the sparsity constraint; Use the two-step FISTA algorithm to solve the optimization objective and take the initial Γ value Γ 0 = 0 to initialize the Γ value Γ of the first step of the iteration 1 : Where X is the input image, D T X is equivalent to performing a convolution operation, for D T DX, DX is equivalent to a deconvolution operation, D T is the transpose of D; then multiply it by D T This is equivalent to performing another convolution operation, S β / L Equivalent to the activation function ReLU; Thus, the basic convolutional sparse coding module is obtained. Then, by superimposing two convolutional sparse coding modules and adding a BN layer in the middle, the image sparse feature extraction model is obtained. The input image is X and the output sparse feature is Y. The process can be expressed by the following formula: Y=CSC(BN(CSC(X))).

2. The zero-shot image description generation method based on CSC and hint learning according to claim 1, characterized in that: Step 1) specifically includes: 1-1) Import the DIV2k dataset; 1-2) Add different degrees of Gaussian noise to the data set and save it; 1-3) Read the image after adding noise, deblur the image using the Lucy-Richardson algorithm, and adjust the image size.

3. The zero-shot image description generation method based on CSC and hint learning according to claim 2, characterized in that: The specific calculation method of the Lucy-Richardson algorithm for image deblurring in steps (1-3) is: Where f(x,y) represents the original image, n is the number of iterations, g(x,y) represents the degraded image, h(x,y) is the spatial description of the degradation function, and the symbol "*" represents spatial convolution; f n+1 (x,y) means f(x,y) iterates n+1 times, f n (x,y) represents f(x,y) iteration n times, and T is the matrix transpose symbol.

4. The zero-shot image description generation method based on CSC and hint learning according to claim 1, characterized in that: The optimization target in step 2) is solved using the fast soft threshold iterative algorithm FISTA. The FISTA algorithm consists of two steps: the first step is to calculate a new point along the calculation direction of the first two steps, and the second step is to perform an approximate point gradient iteration at the new point, that is, Where k represents the kth iteration of the FISTA algorithm, represents the soft threshold operator of h, x k is the result of the kth iteration of the algorithm, x k-1 is the result of the k-1th iteration of the algorithm, x k-2 is the result of the k-2th iteration of the algorithm; y k It represents the weighted sum of the k-1th iteration result and the k-2th iteration result of the algorithm, t k represents the step size of the algorithm iteration, Represents a derivative operation.

5. The zero-shot image description generation method based on CSC and hint learning according to claim 1, characterized in that: Step 3) specifically includes: 3-1) Read the training set from step 1) and adjust the image size; 3-2) Set the number of iterations required for the training set to p and the number of images read in at a time to q, where p ≥ 1 and q ≥ 1; 3-3) Import the noisy image into the image sparse feature extraction model to obtain sparse features and calculate the pixel difference between the sparse features and the original image. Use MSE as the loss function to calculate the loss value in each round. 3-4) Repeat step 3-3) until the loss function value is minimized and save the trained model.

6. The zero-shot image description generation method based on CSC and hint learning according to claim 1, characterized in that: The category hard prompt generated in step 5) is "This is a image of (category)."

Images