Image subtitle method based on attention and state space model

Abstract

The invention is applied to the technical field of image subtitle generation, and particularly discloses an image subtitle method based on an attention and state space model, which comprises the following steps: S1, constructing an image subtitle dynamic hybrid network model based on an attention and state space; according to the image subtitle method based on the attention and the state space model, an encoder and a decoder serve as a framework, a hybrid encoder is constructed, multi-modal features of an image serve as input of the encoder, an attention mechanism is adopted to capture relevance in the modal features, serialization features are extracted in combination with the state space model, and the image subtitle method based on the attention and the state space model is obtained. Fusion is carried out through a self-adaptive gating mechanism, and rich feature information is provided for the decoder; according to the method, word embedding and multi-modal feature interaction are carried out in a decoder, a dependency relationship between word embedding and multi-modal features is obtained, a dynamic fusion module is designed to realize multi-modal dynamic fusion, multi-modal heterogeneous features of an image can be fully utilized to provide richer feature information, and sentence description better conforming to human cognition is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image caption generation, and specifically to an image caption method based on attention and state space model. Background Technique

[0002] Image caption generation is an interdisciplinary field of computer vision and natural language processing. The role of image caption generation is to convert image information into concise and easy-to-understand text descriptions. Image caption generation technology can be widely applied to scenarios such as intelligent assistance and content retrieval. As an interdisciplinary field of computer vision and natural language processing, it is a key research hotspot for the bridge connecting vision and text, which can help computers better understand the complex real world. And image caption generation technology has been widely applied in fields such as industrial control, sentiment analysis, and assistive technology for visually impaired people.

[0003] The encoder-decoder architecture, as a breakthrough framework in the field of machine translation, has been widely applied to image caption generation tasks. Visual features in images are extracted by a visual encoder, and accurate text descriptions are generated by a language decoder. In order to provide rich feature representations for image caption generation, a method that currently introduces self-attention mechanism to model the relationships within feature modalities and then realizes the interaction between feature modalities through cross-attention, and finally integrates the information of different modality features in a cascaded manner to obtain effective feature information has been introduced. However, the dispersion of attention when self-attention processes the relationships within feature modalities may lead to the inability to effectively focus on the correct targets of images, and simple cascading will introduce noise interference between modalities. Especially when there are distribution differences between different modality features, it will significantly reduce the semantic coherence of the generated descriptions. Summary of the Invention

[0004] The purpose of the present invention is to provide an image caption method based on attention and state space model to solve the problem that the distribution differences between different modality features will reduce the semantic coherence of the generated descriptions proposed in the above background technique.

[0005] To achieve the above purpose, the present invention provides the following technical solution: An image caption method based on attention and state space model, including the following steps:

[0006] S1: Construct an image caption dynamic hybrid network model based on attention and state space;

[0007] The image caption dynamic hybrid network model includes a hybrid feature encoder and a dynamic fusion sentence decoder; the hybrid feature encoder includes a global attention feature extraction module, a state space model feature extraction module, and an adaptive gating fusion module; the dynamic fusion sentence decoder includes a feature interaction module and a multi-modal feature dynamic fusion module;

[0008] S2: Obtain a natural image-text pair dataset from a public dataset;

[0009] Among them, the public dataset MSCOCO 2014 dataset includes 123,287 images, and each image is accompanied by 5 different sentence descriptions;

[0010] S3: Convert all sentences to lowercase, remove punctuation marks and words that appear less than 5 times to obtain a refined vocabulary containing the dataset, which includes sentence start symbols and end symbols;

[0011] S4: Input the image data in the public dataset into a frozen pre-trained model to obtain the global grid features and local region features of the image;

[0012] Use a contrastive language-image pre-trained model to extract the global grid features of the image, and a vision-language pre-trained model to extract the local region features of the image;

[0013] S5: Obtain text description features with high similarity to the image content through the contrastive language-image pre-trained model for the global grid features of the image obtained in S4; Segment the image into a fixed size, obtain the text information features of different regions of the image, and encode them through the contrastive language-image pre-trained model;

[0014] S6: Input the global grid features, local region features and text description features of the image obtained in S4 and S5 into a hybrid feature encoder, and output multi-modal features containing rich information; The hybrid feature encoder is a shared encoder, which encodes the global grid features, local region features and text description features of the image at the same time, and uses a contrastive loss to align the feature spaces of different modal features. The global attention feature extraction module learns the global information of different modal features, and the state space model feature extraction module learns the local information of different modal features, and uses an adaptive gating fusion module to fuse the feature information output by the global attention feature module and the state space model feature extraction module to obtain fusion features of different modalities containing richer information;

[0015] S7: Embed the corpus text in the data as word vectors, calculate the position encoding of the corpus text, and use self-attention to obtain the dependency information of the sequence for the word embedding;

[0016] S8: Input the high-level feature representations of the global grid, local region and text description obtained in S6 into a feature interaction module, interact with the word embedding information output in S7, and output the dependency information features between modalities;

[0017] S9: Input the high-level features of the global grid, local region, and text description obtained in S6 and the inter-modal dependency information features obtained in S8 into the multi-modal feature dynamic fusion module, adaptively adjust and fuse the feature information of different modalities to obtain the fusion features for generating the current word;

[0018] S10: Input the fusion features obtained in S9 into the feed-forward neural network layer, fully connected layer, and normalization layer to obtain a probability distribution vector equivalent to the length of the vocabulary. Find the word with the highest probability in the corresponding vocabulary until the end-of-sentence symbol is generated, and obtain the generated image caption;

[0019] S11: Set the loss function and perform iterative training on the image caption dynamic hybrid network model based on attention and state space;

[0020] S12: Repeat S6 - S11 until the model converges to obtain a trained image caption dynamic hybrid network model based on attention and state space;

[0021] S13: Input the natural image into the trained image caption dynamic hybrid network model based on the attention and state space model to obtain the corresponding image caption.

[0022] Preferably, the output end of the global attention feature extraction module is connected to the adaptive gating fusion module, and the output end of the state space model feature extraction module is connected to the adaptive gating fusion module; the output end of the adaptive gating fusion module is connected to the feature interaction module, and the output ends of the hybrid feature encoder and the feature interaction module are connected to the multi-modal feature dynamic fusion module.

[0023] With the above technical solutions, through the cooperation of multiple modules, from feature extraction, fusion, interaction to final dynamic fusion, a complete information processing flow can be constructed.

[0024] Preferably, in S4, the image data in the public dataset is input into the frozen pre-trained model to obtain the global grid features and local region features of the image. The specific method is as follows: Use the multi-modal pre-trained model CLIP to extract the global grid features of the image, and use the visual encoder CLIP-I of CLIP ViT-B / 32 to extract the global grid features of the image. The image is divided into 9x9 grids; use the vision-language pre-trained model VinVL to extract the local region features of the image, and the maximum number of region features of a single image is limited to 50.

[0025] With the above technical solutions, the capabilities of the pre-trained model can be fully utilized to provide rich and valuable feature information for subsequent image understanding and caption generation.

[0026] Preferably, S5 obtains text description features with high similarity to the image content from the global grid features of the image obtained in S4 through a contrastive language-image pre-trained model. The specific method is as follows: Obtain a sentence corpus from the Visual Genome dataset. First, use CLIP to retrieve the text description with the highest similarity to the image content in the Visual Genome corpus for the entire picture; divide the picture into a 2x2 grid to obtain 4 images of the same size, then take the middle part with the same size as the divided images, and retrieve the text description with the highest similarity to the content of each part; use CLIP-T to encode the retrieved text descriptions to obtain text description features similar to the image content.

[0027] With the above technical solution, text description features with high similarity to the image content can be obtained from the global grid features of the image through a contrastive language-image pre-trained model.

[0028] Preferably, S6 inputs the global grid features, local region features, and text description features obtained in S4 and S5 into a hybrid feature encoder and outputs multi-modal features containing rich information. The specific method is as follows: Use a global attention feature extraction module and a state space model feature extraction module to obtain a primary feature representation. Among them, the expression formula for the extraction operation of the primary features is as follows:

[0029] X = (X G , X R , X T )

[0030] Q = XW q , K = XW k , V = XW v

[0031]

[0032] X SSM = SSM(Conv1d(Linear(X)))

[0033] In the formula, X G represents the global grid features of the image; X R represents the local region features of the image; X T represents the text description features similar to the image content; Q, K, V represent the query, key, and value respectively; W q , W k , W k correspond to the weight matrices of the query, key, and value respectively; Attention represents the global attention module; X Att represents the feature representation extracted by the global attention; SSM represents the state space model; Conv1d represents a one-dimensional convolution operation; Linear represents a linear projection; XSSM It represents the feature representation extracted by the state space model; then the adaptive gating module fuses the feature representations extracted by the two modules and outputs a feedforward neural network. The expression formula of the process is as follows:

[0034] α = sigmoid(Linear(X Att + X SSM ))

[0035] X′ = LN(X Att ) × α + LN(X SSM ) × (1 - α)

[0036] X′ = Add&Norm(FFN(X′))

[0037] X′ = (X′ G , X′ R , X′ T )

[0038] In the formula, sigmoid represents the sigmoid activation function; α represents the obtained weight value; LN represents layer normalization; Add&Norm represents residual connection and normalization; FFN represents the feedforward neural network; X′ G , X′ R , X′ T , respectively represent the high-level feature representations of the global image grid feature, local region feature, and text description feature obtained by the hybrid encoder.

[0039] Adopting the above technical solution, a hybrid feature encoder can output multimodal features containing rich information.

[0040] Preferably, S7 embeds the corpus text in the data as word vectors, calculates the positional encoding of the corpus text, and self-attention obtains the dependency information of the word embedding for the sequence. The specific method is as follows: Using the self-attention mechanism to obtain the dependency relationship of the word embedding sequence, the expression formula of the process is as follows:

[0041] yl : t -1 = Attention(YWq, YWk, YWv)

[0042] In the formula, Y represents the word embedding vector; l represents the sentence length.

[0043] Adopting the above technical solution, using the self-attention mechanism to obtain the dependency relationship of the word embedding sequence can improve the quality of image caption generation.

[0044] Preferably, the S8 inputs the high-level feature representations of the global grid, local regions, and text descriptions obtained by S6 into the feature interaction module, interacts with the word embedding information output by S7, and outputs the inter-modal dependency information features. The specific approach is as follows: Use multi-head cross-attention to achieve the interaction between word embeddings and features of different modalities. The specific expression formula is as follows:

[0045] y l:t-1 =MSA(y l:t-1 W q ,X′W k ,X′W v ,)

[0046] In the formula, MSA represents multi-head cross-attention.

[0047] Adopting the above technical solution, using multi-head cross-attention to achieve the interaction between word embeddings and features of different modalities can improve the quality of the generated image captions.

[0048] Preferably, the S9 inputs the high-level features of the global grid, local regions, and text descriptions obtained by S6 and the inter-modal dependency information features obtained by S8 into the multi-modal feature dynamic fusion module, adaptively adjusts and fuses the feature information of different modalities to obtain the fusion features for generating the current word. The specific approach is as follows: Use normalization and linear projection to alleviate the internal covariate shift between feature dimensions, calculate the affinity scores between features of different modalities and word embeddings, and achieve context-aware weight assignment. The expression formula for this process is:

[0049] X′=Linear(Norm(X′))

[0050] y l:t-1 =Linear(Norm(y l:t-1 ))

[0051] β,δ,τ=softmax(X′×y l:t-1 )

[0052] y l:t =β×X′ G +δ×X′ R +τ×X′ T

[0053] In the formula, Norm represents the normalization layer; softmax represents the softmax function; β, δ, τ represent the weight values of the global grid feature of the image, local region feature, and text description feature, respectively.

[0054] Adopting the above technical solution can make full use of the information of different modalities and generate richer and more comprehensive fusion features.

[0055] Preferably, in S10, the fused features obtained in S9 are input into a feed-forward neural network layer, a fully connected layer, and a normalization layer to obtain a probability distribution vector equivalent to the length of the vocabulary. The word with the highest probability is found corresponding to the vocabulary until the end-of-sentence symbol is generated, and the generated image caption is obtained.

[0056] By adopting the above technical solution, through the processing of the feed-forward neural network layer, the fully connected layer, and the normalization layer, the generated word sequence is more in line with the expression habits of natural language in terms of grammar and semantics, and the generation quality of the caption can be improved.

[0057] Preferably, in S11, a loss function is set to iteratively train the dynamic hybrid network model for image captioning based on attention and state space. The specific method is as follows: It is trained in two stages. The cross-entropy loss function is used for the first stage of training, and then the reinforcement learning strategy is used for the second stage of training. Its expression formula is as follows:

[0058]

[0059] L XE =L cap +λL con

[0060]

[0061] In the formula, L cap represents the cross-entropy loss between the real sentence and the generated caption; θ represents the model parameters; P θ represents the probability distribution of the generated caption; L con represents the contrast loss of different modalities; λ represents the weight of the contrast loss; L XE represents the composite loss in the cross-entropy training stage; γ represents the CIDEr score of the sequence; represents the sample in the sentence generated by the model; i represents the input image; represents the k-th example caption; b represents the average CIDEr measure of all generated captions; L R represents the objective loss based on gradient optimization in the reinforcement learning stage.

[0062] By adopting the above technical solution, the two-stage training of the dynamic hybrid network model for image captioning can improve the model quality.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows: The method for image captioning based on the attention and state space model:

[0064] 1. In the present invention, a hybrid encoder is constructed with an encoder-decoder as the architectural framework. The multi-modal features of an image are used as the input to the encoder. The attention mechanism is adopted to capture the correlation within the modal features, the state space model is combined to extract serialized features, and they are fused through an adaptive gating mechanism to provide rich feature information for the decoder. In the decoder, word embedding interacts with the multi-modal features to obtain the dependency relationship between the word embedding and the multi-modal features. A dynamic fusion module is designed to achieve the dynamic fusion of multi-modalities, which can make full use of the multi-modal heterogeneous features of the image to provide richer feature information and generate sentence descriptions that are more in line with human cognition.

[0065] 2. In the present invention, the image caption dynamic hybrid network model is set for training, and a two-stage training method is adopted. First, the cross-entropy loss function is used for the first-stage training to lay a foundation for the model and initially adjust the model parameters. Then, the reinforcement learning strategy is used for the second-stage training to optimize the objective loss based on the gradient, which can further optimize the model, make the model more in line with the actual needs when generating image captions, and improve the caption generation quality of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the process structure of the present invention;

[0067] Figure 2 It is a schematic diagram of the structure of the image caption dynamic hybrid network model constructed by the present invention;

[0068] Figure 3 It is a schematic diagram of the structure of the hybrid feature encoder of the present invention;

[0069] Figure 4 It is a schematic diagram of the structure of the dynamic fusion sentence decoder of the present invention;

[0070] Figure 5 It is a schematic diagram of the qualitative analysis result of the image caption generation of the present invention and the baseline model. DETAILED DESCRIPTION OF THE INVENTION

[0071] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0072] Please refer to Figures 1-5 , the present invention provides a technical solution: an image caption method based on attention and state space model.

[0073] S1: Construct an image caption dynamic hybrid network model based on attention and state space; the image caption dynamic hybrid network model includes a hybrid feature encoder and a dynamic fusion sentence decoder; the hybrid feature encoder includes a global attention feature extraction module, a state space model feature extraction module, and an adaptive gating fusion module; the dynamic fusion sentence decoder includes a feature interaction module and a multimodal feature dynamic fusion module;

[0074] As Figure 1 shown, Figure 1 The figure shows the flow chart of the image caption method based on attention and state space model, and the architecture diagram of the image caption dynamic hybrid network model is as Figure 2 shown. This image caption dynamic hybrid network model is implemented on an NVIDIA 4090 GPU. Before using the image caption dynamic hybrid network model to generate image captions, the model needs to be constructed. The image caption dynamic hybrid network model includes a hybrid feature encoder and a dynamic fusion sentence decoder. The hybrid feature encoder includes a global self-attention feature extraction module, a state space model feature extraction module, and an adaptive gating fusion module; the output end of the global self-attention feature extraction module is connected to the adaptive gating fusion module, and the output end of the state space model feature extraction module is connected to the adaptive gating fusion module. The dynamic fusion sentence decoder includes multi-head self-attention, multi-head cross-attention, a dynamic fusion module, a feed-forward neural network layer, a fully connected layer, and a normalization layer; the output end of the multi-head self-attention is connected to the multi-head cross-attention, the output end of the adaptive gating fusion module is connected to the multi-head cross-attention, the output end of the multi-head cross-attention is connected to the dynamic fusion module, the output end of the adaptive gating fusion module is connected to the dynamic fusion module, the output end of the dynamic fusion module is connected to the feed-forward neural network layer, the output end of the feed-forward neural network layer is connected to the fully connected layer, and the output end of the fully connected layer is connected to the normalization layer.

[0075] S2: Obtain a natural image-text pair dataset from a public dataset; among them, the public dataset MSCOCO 2014 dataset includes 123,287 images, and each image is accompanied by 5 different sentence descriptions;

[0076] The present invention selects the MSCOCO 2014 caption dataset, which contains three sub-files: train, val, and test. Using the Karpathy split, the data in the train and val files are selected as the dataset used in the present invention. The train file contains 82,783 images, and the val file contains 40,504 images, totaling 123,287 natural images. Each image is accompanied by five different captions.

[0077] S3: Convert all sentences to lowercase, remove punctuation marks and words that appear less than 5 times to obtain a refined vocabulary list containing the sentence start symbol and end symbol in the dataset; S4: Input the image data in the public dataset into the frozen pre-trained model to obtain the global grid features and local region features of the image; Use the contrastive language-image pre-trained model to extract the global grid features of the image, and the vision-language pre-trained model to extract the local region features of the image;

[0078] As Figure 2 shown, use the contrastive language pre-trained model CLIP to obtain the global grid features of the image, use the vision-language pre-trained model VinVL to obtain the local region features of the image, input the obtained global grid features and local region features into the global self-attention feature extraction module and the state space model feature extraction module, use the multi-modal pre-trained model CLIP to extract the global grid features of the image, use the visual encoder CLIP-I of CLIP ViT-B / 32 to extract the global grid features of the image, and divide the image into 9x9 grids; Use the vision-language pre-trained model VinVL to extract the local region features of the image, and the maximum number of regional features of a single image is limited to 50.

[0079] S5: Obtain the text description features with high similarity to the image content from the global grid features of the image obtained in S4 through the contrastive language-image pre-trained model; Divide the image into a fixed size, obtain the text information features of different regions of the image, and encode them through the contrastive language-image pre-trained model;

[0080] As Figure 2 shown, calculate the cosine similarity between the global grid features of the image obtained in S4 and the text corpus in the external corpus for cross-modal retrieval to obtain the text corpus knowledge with high similarity to the image content, including the whole image and the five equal-part images, and input them into the global self-attention feature extraction module and the state space model feature extraction module. Obtain the sentence corpus from the Visual Genome dataset. First, use CLIP to retrieve the text description with the highest similarity to the image content in the Visual Genome corpus for the whole image; Divide the picture into 2x2 grids to get 4 images of the same size, then take the middle part with the same size as the divided image, and retrieve the text description with the highest similarity to the content of each part; Use CLIP-T to encode the retrieved text description to obtain the text description features similar to the image content.

[0081] S6: Input the global grid features, local region features, and text description features of the images obtained in S4 and S5 into a hybrid feature encoder, and output multimodal features containing rich information; the hybrid feature encoder is a shared encoder that encodes the global grid features, local region features, and text description features of the images simultaneously, and uses contrastive loss to align the feature spaces of different modality features. The global attention feature extraction module learns the global information of different modality features, and the state space model feature extraction module learns the local information of different modality features. Then, an adaptive gating fusion module is used to fuse the feature information output by the global attention feature module and the state space model feature extraction module to obtain fusion features of different modalities containing richer information;

[0082] The primary feature representation is obtained by using the global attention feature extraction module and the state space model feature extraction module. The expression formula for the extraction operation of the primary features is as follows:

[0083] X = (X G , X R , X T )

[0084] Q = XW q , K = XW k , V = XW v

[0085]

[0086] X SSM = SSM(Conv1d(Linear(X)))

[0087] In the formula, X G represents the global grid features of the image; X R represents the local region features of the image; X T represents the text description features similar to the image content; Q, K, and V represent the query, key, and value respectively; W q , W k , W k correspond to the weight matrices of the query, key, and value respectively; Attention represents the global attention module; X Att represents the feature representation extracted by the global attention; SSM represents the state space model; Conv1d represents the one-dimensional convolution operation; Linear represents the linear projection; X SSM represents the feature representation extracted by the state space model; Then, the adaptive gating module fuses the feature representations extracted by the two modules and outputs a feed-forward neural network. The expression formula for this process is:

[0088] α = sigmid(Linear(X Att + X SSM ))

[0089] X′ = LN(X Att ) × α + LN(X SSM ) × (1 - α)

[0090] X′ = Add&Norm(FFN(X′))

[0091] X′ = (X′ G , X′ R , X′ T )

[0092] In the formula, sigmoid represents the sigmoid activation function; α represents the obtained weight value; LN represents layer normalization; Add&Norm represents residual connection and normalization; FFN represents the feed-forward neural network; X′ G , X′ R , X′ T , respectively represent the high-level feature representations of the global grid features, local region features, and text description features of the image obtained by the hybrid encoder. The hybrid feature encoder is a shared encoding that concatenates the multi-modal features of the image and inputs them into the global self-attention feature extraction module and the state space model feature extraction module at the same time. The global self-attention feature extraction module uses multi-head self-attention, with a total of 8 heads. The state space model feature extraction module uses Vim (Vision Mamba), the local convolution width is 4, and the number of layers N of the hybrid feature encoder is 3.

[0093] S7: Embed the corpus text in the data as word vectors, calculate the positional encoding of the corpus text, and use self-attention to obtain the dependency information of the word embedding for the sequence; S8: Input the high-level feature representations of the global grid, local region, and text description obtained in S6 into the feature interaction module, interact with the word embedding information output in S7, and output the dependency information features between modalities; S9: Input the high-level features of the global grid, local region, and text description obtained in S6 and the dependency information features between modalities obtained in S8 into the multi-modal feature dynamic fusion module, adaptively adjust and fuse the feature information of different modalities to obtain the fusion features for generating the current word; S10: Input the fusion features obtained in S9 into the feed-forward neural network layer, fully connected layer, and normalization layer to obtain a probability distribution vector equivalent to the length of the vocabulary, find the word with the highest probability in the corresponding vocabulary until the end-of-sentence symbol is generated, and obtain the generated image caption;

[0094] Use the self-attention mechanism to obtain the dependency relationship of the word embedding sequence, and the expression formula of its process is:

[0095] y l:t-1 = Attention(YW q , YW k , YWv )

[0096] In the formula, Y represents the word embedding vector; l represents the sentence length,

[0097] S8 inputs the high-level feature representations of the global grid, local region, and text description obtained in S6 into the feature interaction module, interacts with the word embedding information output in S7, and outputs the cross-modal dependency information features. The specific approach is as follows: Use multi-head cross-attention to achieve the interaction between word embeddings and features of different modalities. The specific expression formula is as follows:

[0098] y l:t-1 = MSA(y l:t-1 W q , X′W k , X′W v ,)

[0099] In the formula, MSA represents multi-head cross-attention;

[0100] Input the high-level feature representations of the global grid, local region, and text description obtained in S6 into the feature interaction module, interact with the word embedding dependency information obtained in S7, output the cross-modal dependency information features, use multi-head cross-attention to achieve the interaction between word embeddings and features of different modalities, with 8 heads for cross-attention. Input the high-level features of the global grid, local region, and text description obtained in S6 and the cross-modal dependency information features obtained in S8 into the multi-modal feature dynamic fusion module. First, use normalization and linear projection to alleviate the internal covariate shift between feature dimensions, calculate the affinity between features of different modalities and cross-modal dependency information features, use the softmax function to calculate the weight values of the global grid features, local region features, and text description features, adaptively adjust and fuse the feature information of different modalities to obtain the fused features for generating the current word. Input the fused features obtained in S9 into the feed-forward neural network layer, fully connected layer, and normalization layer to obtain a probability distribution vector equivalent to the vocabulary length, find the word with the highest probability in the corresponding vocabulary until the sentence end symbol is generated, and obtain the generated image caption. The number of layers N of the sentence decoder is 3;

[0101] Use normalization and linear projection to alleviate the internal covariate shift between feature dimensions, calculate the affinity scores between features of different modalities and word embeddings, and achieve context-aware weight assignment. The expression formula for this process is:

[0102] X′ = Linear(Norm(X′))

[0103] y l:t-1 = Linear(Norm(y l:t-1 ))

[0104] β, δ, τ = softmax(X′ × y l:t-1 )

[0105] y l:t = β × X′ G + δ × X′ R + τ × X′ T

[0106] In the formula, Norm represents the normalization layer; softmax represents the softmax function; β, δ, and τ represent the weight values of the global image grid features, local region features, and text description features respectively; S10 inputs the fused features obtained by S9 into the feed-forward neural network layer, fully connected layer, and normalization layer to obtain a probability distribution vector equivalent to the length of the vocabulary, finds the word with the highest probability in the corresponding vocabulary until the sentence end symbol is generated, and obtains the generated image caption.

[0107] S11: Set the loss function and perform iterative training on the dynamic hybrid network model for image captioning based on attention and state space; S12: Repeat S6 - S11 until the model converges to obtain the trained dynamic hybrid network model for image captioning based on attention and state space; S13: Input the natural image into the trained dynamic hybrid network model for image captioning based on the attention and state space model to obtain the corresponding image caption;

[0108] Set the loss function to perform iterative training on the generated dynamic hybrid network model for image captioning. The training is divided into two stages. The cross-entropy loss function is used for the first stage of training, and then the reinforcement learning strategy is used for the second stage of training. Its expression formula is as follows:

[0109]

[0110] L XE = L cap + λL con

[0111]

[0112] In the formula, L cap represents the cross-entropy loss between the real sentence and the generated caption;

[0113] θ represents the model parameters; P θ represents the probability distribution of the generated caption;

[0114] L con represents the contrastive loss of different modalities;

[0115] λ represents the weight of the contrastive loss; L XE represents the composite loss in the cross-entropy training stage;

[0116] γ represents the CIDEr score of the sequence; represents a sample in the sentence generated by the model;

[0117] I represents the input image;

[0118] represents the k-th example caption;

[0119] b represents the average CIDEr metric of all generated captions; L R represents the objective loss based on gradient optimization in the reinforcement learning stage to improve the accuracy of the model.

[0120] In this invention, on the MSCOCO benchmark, the performance of the dynamic hybrid network of image captioning based on attention and state space model and other methods is quantitatively analyzed and compared in terms of BLUE@N (B@N), METEOR (M), ROUGR-L (R), CIDEr (C) and SPICE (S) respectively.

[0121] See Table 1 below for details:

[0122] Table 1

[0123]

[0124] As shown in Table 1, the higher each index is, the more accurate and semantically fluent the generated caption is. The method proposed in this invention achieves the best results in the MSCOCO 2014 data;

[0125] Such as Figure 5 shown, is the qualitative analysis result of the image caption generation of the method described in this invention and the benchmark model. From the comparison in the figure, it can be seen that the method proposed in this invention can capture local details while obtaining global information and can generate captions closer to the true values.

[0126] Working principle: In this invention, with the encoder-decoder as the architecture, a hybrid encoder is constructed. The multi-modal features of images and texts are input into the hybrid encoder. The attention mechanism is used to capture the correlation within the modal features, and the state space model is combined to extract serialized features, providing rich feature information for the decoder. Through operations such as normalization and linear projection, weights are dynamically allocated according to the context, and the fused features for generating words are obtained by fusing the features. Then, through processing such as the feed-forward neural network layer, a probability distribution vector is obtained, and the word with the highest probability is selected until the sentence end symbol is generated, completing the image caption generation.

[0127] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention.

Claims

1. An image captioning method based on attention and state space model, characterized in that, It includes the following steps: S1: Construct an image caption dynamic hybrid network model based on attention and state space; The image caption dynamic hybrid network model includes a hybrid feature encoder and a dynamic fusion sentence decoder; the hybrid feature encoder includes a global attention feature extraction module, a state space model feature extraction module, and an adaptive gating fusion module; the dynamic fusion sentence decoder includes a feature interaction module and a multimodal feature dynamic fusion module; S2: Obtain a natural image-text pair dataset from a public dataset; Among them, the public dataset MSCOCO 2014 dataset includes 123,287 images, and each image is accompanied by 5 different sentence descriptions; S3: Convert all sentences to lowercase, remove punctuation marks and words that appear less than 5 times to obtain a refined vocabulary list containing the dataset, which includes a sentence start symbol and an end symbol; S4: Input the image data in the public dataset into a frozen pre-trained model to obtain the global grid features and local region features of the image; Use a contrastive language-image pre-trained model to extract the global grid features of the image, and a vision-language pre-trained model to extract the local region features of the image; S5: Pass the image global grid features obtained in S4 through a contrastive language-image pre-trained model to obtain text description features with high similarity to the image content; segment the image into a fixed size, obtain text information features of different regions of the image, and encode them through a contrastive language-image pre-trained model; S6: Input the image global grid features, local region features, and text description features obtained in S4 and S5 into the hybrid feature encoder, and output multimodal features containing rich information; the hybrid feature encoder is a shared encoder that encodes the image global grid features, local region features, and text description features at the same time, and uses a contrastive loss to align the feature spaces of different modal features. The global attention feature extraction module learns the global information of different modal features, the state space model feature extraction module learns the local information of different modal features, and the adaptive gating fusion module fuses the feature information output by the global attention feature module and the state space model feature extraction module to obtain fusion features of different modalities containing richer information; S7: Embed the corpus text in the data as word vectors, calculate the positional encoding of the corpus text, and self-attention obtains the dependency information of the sequence for the word embedding; S8: Input the high-level feature representations of the global grid, local region, and text description obtained in S6 into the feature interaction module, interact with the word embedding information output in S7, and output the dependency information features between modalities; S9: Input the high-level features of the global grid, local region, and text description obtained in S6 and the dependency information features between modalities obtained in S8 into the multimodal feature dynamic fusion module, and adaptively adjust and fuse the feature information of different modalities to obtain the fusion features for generating the current word; S10: Input the fused features obtained in S9 into a feed-forward neural network layer, a fully-connected layer, and a normalization layer to obtain a probability distribution vector equivalent to the length of the vocabulary. Find the word with the highest probability corresponding to the vocabulary until the end-of-sentence symbol is generated, and obtain the generated image caption. S11: Set the loss function and perform iterative training on the image caption dynamic hybrid network model based on attention and state space. S12: Repeat S6 - S11 until the model converges to obtain a trained image caption dynamic hybrid network model based on attention and state space. S13: Input a natural image into the trained image caption dynamic hybrid network model based on the attention and state space model to obtain the corresponding image caption.

2. The image captioning method based on attention and state space model according to claim 1, wherein: The output end of the global attention feature extraction module is connected to the adaptive gating fusion module, and the output end of the state space model feature extraction module is connected to the adaptive gating fusion module; the output end of the adaptive gating fusion module is connected to the feature interaction module, and the output ends of the hybrid feature encoder and the feature interaction module are connected to the multi-modal feature dynamic fusion module.

3. The image captioning method based on attention and state space model according to claim 1, wherein: In S4, the image data in the public dataset is input into the frozen pre-trained model to obtain the global grid features and local region features of the image. The specific method is as follows: The multi-modal pre-trained model CLIP is used to extract the global grid features of the image, and the visual encoder CLIP-I of CLIPViT-B / 32 is used to extract the global grid features of the image. The image is divided into a 9x9 grid; the visual language pre-trained model VinVL is used to extract the local region features of the image, and the maximum number of region features for a single image is limited to 50.

4. The image captioning method based on attention and state space model according to claim 1, characterized in that: In S5, the image global grid features obtained in S4 are used to obtain text description features with high similarity to the image content through a contrastive language-image pre-trained model. The specific method is as follows: A sentence corpus is obtained from the Visual Genome dataset. First, CLIP is used to retrieve the text description with the highest similarity to the image content in the Visual Genome corpus for the entire image; the image is divided into a 2x2 grid to obtain 4 images of the same size, and then the middle part with the same size as the divided image is taken to retrieve the text description with the highest similarity to the content of each part; CLIP-T is used to encode the retrieved text description to obtain text description features similar to the image content.

5. The image captioning method based on attention and state space model according to claim 1, characterized in that: In S6, the image global grid features, local region features, and text description features obtained in S4 and S5 are input into the hybrid feature encoder, and multi-modal features containing rich information are output. The specific method is as follows: The global attention feature extraction module and the state space model feature extraction module are used to obtain the primary feature representation. Among them, the expression formula for the extraction operation of the primary feature is as follows: X = (X G , X R , X T ) Q = XW q , K = XW k , V = XW v X SSM = SSM(Conv1d(Linear(X))) Where X G represents the global grid feature of the image; X R represents the local region feature of the image; X T represents the text description feature similar to the image content; Q, K, and V represent the query, key, and value respectively; W q , W k , W k correspond to the weight matrices of the query, key, and value respectively; Attention represents the global attention module; X Att represents the feature representation extracted by the global attention; SSM represents the state space model; Conv1d represents the one-dimensional convolution operation; Linear represents the linear projection; X SSM represents the feature representation extracted by the state space model; then the adaptive gating module fuses the feature representations extracted by the two modules and outputs a feed-forward neural network, and the expression formula of the process is: α = sigmoid(linear(X Att + X SSM )) X′ = LN(X Att ) × α + LN(X SSM ) × (1 - α) X′ = Add&Norm(FFN(X′)) X′ = (X′ G , X′ R , X′ T ) where, sigmoid represents the sigmoid activation function; α represents the obtained weight value; LN represents layer normalization; Add&Norm represents residual connection and normalization; FFN represents the feed-forward neural network; X′ G , X′ R , X′ T , respectively represent the high-level feature representations of the global image grid features, local region features, and text description features obtained by the hybrid encoder.

6. The image captioning method based on attention and state space model according to claim 1, wherein: In S7, the corpus text in the data is embedded as word vectors, the positional encoding of the corpus text is calculated, and self-attention is used to obtain the dependency information of the word embedding for the sequence. The specific method is as follows: The self-attention mechanism is used to obtain the dependency relationship of the word embedding sequence, and the expression formula for the process is as follows: yl : t -1 = Attention(YWq, YWk, YWv) Where, Y represents the word embedding vector; l represents the sentence length.

7. The image captioning method based on attention and state space model according to claim 1, wherein: The S8 inputs the high-level feature representations of the global grid, local region, and text description obtained by S6 into the feature interaction module, interacts with the word embedding information output by S7, and outputs the inter-modal dependency information features. The specific approach is as follows: Use multi-head cross-attention to achieve the interaction between word embeddings and different modal features. The specific expression formula is as follows: y l:t-1 = MSA(y l:t-1 W q , X′W k , X′W v, ) Where, MSA represents multi-head cross-attention.

8. The image captioning method based on attention and state space model according to claim 1, characterized in that: The S9 inputs the high-level features of the global grid, local region, and text description obtained by S6 and the inter-modal dependency information features obtained by S8 into the multi-modal feature dynamic fusion module, adaptively adjusts and fuses the feature information of different modalities to obtain the fusion features for generating the current word. The specific approach is as follows: Use normalization and linear projection to alleviate the internal covariate shift between feature dimensions, calculate the affinity scores between different modal features and word embeddings, and achieve context-aware weight assignment. The expression formula for this process is: X′ = Linear(Norm(X′)) y l:t-1 = Linear(Norm(y l:t-1 )) β, δ, τ = softmax(X′ × y l:t-1 ) y l:t = β × X' G + δ × X' R + τ × X T Where, Norm represents the normalization layer; softmax represents the softmax function; β, δ, τ respectively represent the weight values of the global grid feature of the image, local region feature, and text description feature.

9. The image captioning method based on attention and state space model according to claim 1, characterized in that: The S10 inputs the fusion features obtained by S9 into the feed-forward neural network layer, fully connected layer, and normalization layer, obtains a probability distribution vector equivalent to the vocabulary length, finds the word with the highest probability in the corresponding vocabulary until the sentence end symbol is generated, and obtains the generated image caption.

10. The image captioning method based on attention and state space model according to claim 1, wherein: The S11 sets the loss function and iteratively trains the image caption dynamic hybrid network model based on attention and state space. The specific approach is as follows: Train in two stages. Use the cross-entropy loss function for the first stage of training, and then use the reinforcement learning strategy for the second stage of training. The expression formula is as follows: L XE = L cap + λL con where L cap represents the cross - entropy loss between the real sentence and the generated caption; θ represents the model parameters; P θ represents the probability distribution of the generated caption; L con represents the contrastive loss for different modalities; λ represents the weight of the contrastive loss; L XE represents the composite loss in the cross - entropy training stage; γ represents the CIDEr score of the sequence; represents the sample in the sentence generated by the model; I represents the input image; represents the k - th sampled caption; b represents the average CIDEr metric of all generated captions; L R represents the objective loss based on gradient optimization in the reinforcement learning stage.

Images