An end-to-end image description generation method based on enhanced attention mechanism

Abstract

The application provides an end-to-end image description generation method based on an enhanced attention mechanism, and belongs to the technical field of artificial intelligence. An image description generation model is generated, which comprises an image feature extraction layer, a multi-granularity feature fusion encoder, a self-adaptive bidirectional graph decoding device, a linear transformation layer and a scoring and sorting layer; the image description generation model is trained using a cross-entropy loss, then a self-criticism training is used to optimize a CIDEr score, and the trained image description generation model is used to describe an image. The evaluation index is superior to the prior art, the image description method has improved image semantic understanding ability, is closer to human description habits, and has good interpretability.

Description

Technical Field

[0001] This invention proposes an end-to-end image description generation method based on an enhanced attention mechanism, belonging to the field of artificial intelligence technology. Background Technology

[0002] Image captioning (IC), a typical task in multimodal learning, integrates computer vision (CV) and natural language processing (NLP) to identify image elements and relationships, generating accurate and fluent natural language descriptions. In many fields, such as multimodal image retrieval and visual assistance, accurate image captioning is crucial for improving efficiency and quality of life. It also plays an important role in advertising, marketing, virtual reality, and other areas, providing users with richer and more intuitive experiences. Therefore, improving the quality and accuracy of image captioning is essential for driving development and innovation across multiple fields.

[0003] With the development of neural networks, researchers in recent years have treated image description as a sequence-to-sequence task, often utilizing the transformer architecture to fuse image features and generate effective descriptions. For example, Ji et al. used transformers to capture global information of images and generate high-quality descriptions. Peter et al. fused local and global information based on traditional transformers to generate image descriptions that are closer to human intuition. Z et al. adopted a two-stream tightly coupled mechanism, artificially setting attention to the context of the description information to simultaneously acquire past and future information and improve the description effect. From the existing image description methods, most methods fail to fully exploit the multi-granular semantic information within the image when extracting image features, using only single-stream feature inputs or incompletely fusing multi-stream inputs. Furthermore, while attention masking mechanisms effectively solve the problem of exposure bias, they lose the model's ability to obtain contextual information, which is inconsistent with human habits when describing images. It is necessary to consider the past and future information of the descriptive statement to generate more accurate natural language descriptions that conform to human expression habits. Therefore, this invention proposes a new image description method, characterized by solving the problem of insufficient fusion of multi-granularity features in images, while simultaneously enabling the model to adaptively focus on past and future information. Summary of the Invention

[0004] Objective: To address the shortcomings of existing technologies, this invention proposes a method for generating descriptions by fusing multi-granularity features and employing bidirectional autoregression. By using a multi-granularity image information fusion method, different granularities of information from the image being described are captured and effectively fused. An adaptive bidirectional mask attention method captures past and future information from the linguistic description. Furthermore, due to the acquisition of complete image features, this invention demonstrates stronger image semantic understanding capabilities. The use of an adaptive bidirectional decoding method more closely aligns with human language descriptive habits, resulting in better interpretability.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0006] An end-to-end image caption generation method based on an enhanced attention mechanism, comprising the following steps:

[0007] Step 1: Construct an end-to-end image description generation model with an enhanced attention mechanism based on the transformer architecture. The image description generation model includes a sequentially connected feature extraction layer, a multi-granularity feature fusion encoder, an adaptive bidirectional decoder, a linear transformation layer, and a scoring and ranking layer. The feature extraction layer includes the VinVL model and the ResNet101 model. The multi-granularity feature fusion encoder includes a sequentially connected multi-granularity feature fusion, a dense global attention mechanism, and a feedforward network layer. The adaptive bidirectional decoder includes a sequentially connected adaptive bidirectional mask attention, a cross attention mechanism, and a feedforward network layer. The granularity feature fusion encoder is connected to the cross attention mechanism, and the adaptive bidirectional mask attention is connected to description statement encoding and position encoding.

[0008] Step 2: Select images from the MSCOCO2014 image dataset, and select two description statements from the description statements that come with the selected images to generate a training set. One of the two description statements is a sequential description, and the other is a reverse description. Segment and filter the image description statements in the MSCOCO2014 image dataset to generate a vocabulary. Assign pre-trained parameters to the VinVL model and ResNet101 model of the image feature extraction layer.

[0009] Step 3: Input the images in the training set into the image feature extraction layer to obtain the unencoded original image features. Input the original image features into the multi-granularity feature fusion encoder and use the multi-granularity feature fusion module to encode the images: fuse the original image features to obtain the image fusion features, use the dense global attention mechanism to add the global features of the image to the fusion features, and then pass through the feedforward network layer to obtain the encoded image features.

[0010] Step 4: Input the two descriptions corresponding to the image in the input image feature extraction layer into the description encoding and position encoding, and input the encoded description information into the adaptive bidirectional decoder. After the adaptive bidirectional mask attention mechanism, the bidirectional description decoding features are obtained. The image features obtained in Step 3 are input into the multi-head cross attention mechanism to interact with the bidirectional description decoding features to obtain the decoding information. The decoding information is input into the feedforward network layer to output the final decoding features.

[0011] Step 5: The final decoded features generated by the adaptive bidirectional decoder are mapped to the vocabulary through a linear transformation layer to obtain the final description. The final description includes two sentences: a sequential description and a reverse description. The loss is calculated by comparing the two sentences of description of the original input image with the description of the image. The gradient is backpropagated to correct the weights of the image description generation model until the set training is completed.

[0012] Step 6: After the image description generation model is trained, it enters the inference stage. At this time, the image description generation model does not change the model weights. The image to be described is passed to the image feature extraction layer of the trained image description generation model to obtain the original image features. Then, it is passed to the multi-granularity feature fusion encoder and adaptive bidirectional decoder to generate descriptions in two directions. Finally, it is passed to the scoring and ranking module, which selects the description with the higher score from the two generated descriptions as the final output.

[0013] Furthermore, in the MSCOCO2014 image description dataset used as the training dataset, each data point contains an image and five manually annotated sentences. During the training phase, the end-to-end image description generation method based on the enhanced attention mechanism uses two description sentences and corresponding images as one data point. One description sentence is selected from the five manually annotated sentences, and the other is a sentence selected from the remaining four manually annotated sentences and processed in reverse order. The length of the two sentences needs to be padded. Finally, each image has five description pairs, and each description pair consists of a forward description and a reverse description. The image serves as the input for the training phase, and each description serves as the label for the training phase.

[0014] Furthermore, the image feature extraction layer is implemented as follows:

[0015] The VinVL model object detection network was used to extract coarse-grained features C = {c1, c2, c3, ..., c} from each image in the MSCOCO2014 image description dataset. n The ResNet101 convolutional neural network is used to extract fine-grained features F = {f1, f2, f3, ..., f} from each image in the MSCOCO2014 image description dataset. n}, where the feature dimensions of C and F are d1 and d2 respectively, mapping the coarse and fine granular features to the same dimensional space, such that C o and Fo The feature dimension is d0:

[0016] C o =Liner(C)

[0017] F o =Liner(F)

[0018] Where n is the number of images in the dataset, d1 represents the feature dimension obtained by the VinVL model, d2 represents the feature dimension obtained by the ResNet101 convolutional neural network, and d0 represents the feature dimension obtained after linear mapping.

[0019] Furthermore, the implementation process of the multi-granularity feature fusion encoder is as follows:

[0020] The multi-granularity feature fusion module of the multi-granularity feature fusion encoder includes a multi-head self-attention mechanism (MHSA) and a multi-head cross-attention mechanism (MSCA) set in parallel. It fuses the coarse-grained and fine-grained features of each image to obtain the multi-granularity feature F. CF This can be expressed as a formula:

[0021] F CF =F MHCA (C o ,F o )+F MHSA (C o )

[0022] The number of heads is set to 8;

[0023] Dense Global Attention (DGAM) employs broadcast mean and self-attention methods to integrate global features into the multi-granular features of the image to obtain x, which can be expressed by the formula:

[0024] x = DGAM(F CF ) = (Broadcasting(Average(F CF ))+Attention(F CF ))*0.5

[0025] The feedforward network layer can be expressed by the formula: where X is the output of the feedforward network layer:

[0026] X = FFLayer(x)

[0027] Multi-granularity feature fusion, dense global attention mechanism, residual connection between feedforward network layers, dropout to prevent overfitting, and regularization Norm to accelerate model convergence:

[0028] X′=X+dropout(Norm(X))

[0029] Where X' represents the features of the image after being encoded by a multi-granularity feature fusion encoder.

[0030] Furthermore, the adaptive bidirectional decoder expression is as follows:

[0031] Caption = Decoder(X′, z)

[0032] Where X' is the image feature after being encoded by a multi-granularity feature fusion encoder, and the image feature X' obtained from the encoded segment interacts with the text information z at the decoding end; z is the human description sentence corresponding to an image in the MSCOCO dataset, and the text information feature obtained after description sentence encoding and position encoding. z contains description information of two word orders, with dimensions of (batch_size, 2, sentence_len, dim), where 2 represents two word orders of sentences, batch_size is the input batch, sentence_len is the sentence length, and dim is the feature dimension;

[0033] The Adaptive Bidirectional Mask Attention (ABMA) module comprises two parallel mask attention mechanism modules. These two parallel mask attention mechanism modules interact with an Adaptive Fusion module. The adaptive fusion module processes the feature expressions obtained from the two mask modules as follows:

[0034] z′=ABMA(z)

[0035] ABMA(z)=AdaptiveFusion(MHMSA L (z,z,z),MHMSA R (z,flip(z),flip(z)))

[0036] Where z' represents the feature after ABMA processing, flip represents the flipping operation, and the expression for the AdaptiveFusion module is:

[0037] r t ,z t =σ(W g [M L M R ])

[0038] AF = AdaptiveFusion(M) L M R ) = r t ⊙M L +z t ⊙M R

[0039] Where ML M R The output features of the two masked attention mechanisms are... Here, σ is the learnable parameter, σ is the sigmoid function used for normalization and weight calculation, ⊙ is the bitwise product, and r is the weight. t ,z t The result is obtained by concatenating the two mask attention mechanisms according to the last dimension and then normalizing the result.

[0040] The masked attention mechanism uses the multi-head method MHMSA, as shown in the following expression:

[0041]

[0042] Where, d K It is a scaling factor, a mapping parameter matrix. The number of heads is set to 8 to be learnable parameters, and Q, K, and V represent the features after mapping by the mapping parameter matrix.

[0043] The number of heads in the multi-head cross-attention module is set to 8, and the expression is:

[0044]

[0045] The feedforward network layer can be expressed by the following formula:

[0046] Z′=FFLayer(Z)

[0047] The adaptive bidirectional masked attention module, cross-attention module, and feedforward network layers are connected using residual connections, and dropout is set to prevent overfitting. Regularization Norm is set to speed up the model's convergence.

[0048] Caption=Z′+dropout(Norm(Z′)).

[0049] Furthermore, the linear transformation layer consists of a linear layer and a softmax layer. The input caption dimension of the linear layer is (batch_size, 2, sentence_len, dim), and the output dimension of the linear layer is (batch_size, 2, sentence_len, vocab_size), where 2 represents sentences with two word orders, and vocab_size is the size of the vocabulary.

[0050] Furthermore, the specific method for the scoring and ranking layer is as follows:

[0051] During the inference phase of the scoring and ranking layer, the layer first sorts the two sentences obtained from the decoder in order. Using a length penalty mechanism, it calculates the log probability of each sentence at each position, sums them up, and returns the sentence with the higher score as the final output. The expression is as follows:

[0052]

[0053] in These are descriptive statements representing two directions.

[0054] Furthermore, the specific method for outputting image descriptions through the constructed image description generation model is as follows:

[0055] Step 3.1: Perform feature encoding; assuming the input is an image I, first use the image feature extraction layer to extract its coarse-grained and fine-grained features (C). o ,F o Secondly, the coarse-grained feature (C) is used to further refine the granularity characteristics. o ,F o X is fed into a multi-granularity feature fusion encoder as the feature to be encoded to obtain X;

[0056] Step 3.2: Output the target features; align X with the cross-attention mechanism in the adaptive bidirectional decoder and the description sentence encoding; the description information consists of two different word orders of image I, and the description information is encoded and then the position-encoded description features z are added to the description information Caption in the adaptive bidirectional decoder.

[0057] Step 3.3: Output the final statement; input the final description information Caption from the adaptive bidirectional decoder into the linear transform layer mapping lexicon to generate the final description statement, calculate the loss between the final description statement and the real label, and backpropagate to optimize the image description generation model.

[0058] Furthermore, the training process for the gradient backpropagation correction of the image description generation model's weights adopts a two-stage training approach. The specific methods for the loss functions in the two training stages are as follows:

[0059] The loss function for the first training phase is cross-entropy loss, used to quickly update the weights. The expression is:

[0060]

[0061] in This represents the calculated loss from left to right in the description statements. This represents the computational loss for describing statements from right to left, where T represents the sentence length and p... θ Describe the probability of the output word;

[0062] The second training phase uses a more refined loss function, expressed as follows:

[0063]

[0064] Where R(·) represents optimization of CIDEr score, and T represents sentence length.

[0065] Furthermore, the specific process of the image description generation method in the inference stage is as follows: After training is completed, the model parameters are frozen and the inference stage is entered. An image is input into the image description generation model. After forward computation, the text generator outputs the word IDs corresponding to the two words with the highest probabilities each time. After searching the dictionary, the image descriptions of the two inferred word orders can be obtained. Then, the scoring and ranking layer outputs the description sentence with the highest score.

[0066] Beneficial Effects: This invention addresses the shortcomings of existing technologies by proposing a method for generating descriptions through the fusion of multi-granularity features and bidirectional autoregression. By employing a multi-granularity image information fusion method, it captures and effectively fuses information of different granularities in the image being described. Furthermore, an adaptive bidirectional mask attention method captures past and future information of the linguistic description. Moreover, due to the acquisition of complete image features, this invention possesses stronger image semantic understanding capabilities, and because it utilizes an adaptive bidirectional decoding method, it more closely aligns with human language descriptive habits, exhibiting good interpretability. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the end-to-end image description generation method model based on the enhanced attention mechanism of the present invention; wherein each module in the encoder-decoder needs to go through residual connections;

[0068] Figure 2 This is a schematic diagram of the dense global attention mechanism array in an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of the adaptive bidirectional mask attention mechanism in an embodiment of the present invention. Detailed Implementation

[0070] The embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0071] like Figure 1As shown, an end-to-end image description generation method based on an enhanced attention mechanism includes an image feature extraction layer, a multi-granularity feature fusion encoder, an adaptive bidirectional decoder, a linear transformation layer, and a scoring and ranking layer. The image feature extraction layer consists of two pre-trained image feature extractors connected in parallel. One pre-trained image feature extractor captures coarse-grained features of the image, and the other pre-trained image feature extractor captures fine-grained features. The multi-granularity feature fusion encoder consists of a multi-granularity feature fusion module, a dense global attention mechanism, a feedforward network layer, and a residual layer. The adaptive bidirectional decoder consists of an adaptive bidirectional mask attention mechanism, a cross-attention mechanism, a feedforward network layer, and a residual layer. The linear transformation layer consists of a linear layer and a softmax classifier, which performs dimensionality transformation on the output of the adaptive bidirectional decoder and outputs bidirectional autoregressive descriptive words. Finally, the scoring and ranking layer receives the two descriptive statements obtained from the linear transformation layer, ranks the two descriptive statements in order, and selects the statement with the highest score as the final output. To preserve effective features and avoid the vanishing gradient problem, each module in the multi-granularity feature fusion encoder and the adaptive bidirectional decoder is cascaded using residual connections. Dropout is also employed in each module of both modules to prevent overfitting.

[0072] An end-to-end image captioning generation method based on an enhanced attention mechanism includes the following steps:

[0073] S1. Extract features from the official dataset, using a pre-trained image feature extraction model to extract coarse-grained and fine-grained features from the images respectively;

[0074] S2. Input the coarse-grained image features and fine-grained image features captured in S1 into the multi-granularity feature fusion encoder to perform feature fusion encoding, thereby obtaining multi-granularity image features with better image semantic representation capabilities.

[0075] S3. The multi-granularity image features and the corresponding bidirectional descriptive features obtained in S2 are used as input to the adaptive bidirectional decoder. The bidirectional descriptive features are obtained by encoding descriptions in two directions and adding positional encoding. The final output is image descriptions with two different word orders.

[0076] S4. Input the descriptive features of the two different word orders generated in S3 into the linear transformation layer, transform the dimensions and map them to the vocabulary, and output the final two word order descriptive sentences through the softmax layer. Calculate the loss with the real labels and backpropagate to optimize the model.

[0077] S5. In the inference stage, the two word order description statements obtained in S4 are arranged in order, and the generation scores of the two description statements are calculated using the scoring and ranking layer. The statement with the higher score is selected as the natural language description output by the final model.

[0078] Furthermore, in S1, a pre-trained VinVL model is used to extract region features from the dataset, which serve as coarse-grained features of the image C = {c1, c2, c3, ..., c n The pre-trained ResNet101 model is used to extract grid features from the image, which are then used as fine-grained features F = {f1, f2, f3, ..., f}. n},in To ensure that the two granularity features operate in the same dimensional space, this invention maps the two granularity features to the same dimensional space respectively, so that...

[0079] C o =Liner(C)

[0080] F o =Liner(F)

[0081] The multi-granularity feature fusion encoder in S2 consists of a multi-head self-attention mechanism module (MHSA) and a multi-head cross-attention mechanism module (MSCA). To obtain feature information from multiple channels, the attention mechanism employs a multi-head approach. The formula for the multi-head attention mechanism is as follows:

[0082]

[0083] Where, d K It is a scaling factor, a mapping parameter matrix. and Attention head count and Set it to 8.

[0084] And the coarse-grained features and fine-grained features are combined additively:

[0085] F CF =F MHCA +F MHSA

[0086] To fully capture global features and achieve high-performance representation through feature fusion, such as Figure 2As shown, a dense global attention mechanism (DGMA) is designed to capture global features with almost no computational cost. The dense local attention mechanism consists of a multi-head self-attention mechanism without linear transformation and broadcast mean calculation. The multi-head self-attention mechanism is used to retain the currently focused feature, which is dense. The broadcast mean calculation is used to focus on global features. Finally, the two types of features are added together and averaged to obtain the dense global information.

[0087]

[0088] F2(X3)=Broadcasting(Average(X3))

[0089] DGAM(X1,X2,X3)=(F1(X1,X2,X3)+F2(X3))*0.5

[0090] Among them, X {1，2，3} It is a feature obtained by fusing coarse and fine granularities, d x It's a scaling factor. Broadcasting performs a broadcast mechanism on the last dimension of the features, while Average calculates the average value on the last dimension of the features. The number of attention heads. x Set it to 8.

[0091] Then, a feedforward neural network is used to ensure that the dimensions are within a unified dimensional space:

[0092] X = FFLayer(x)

[0093] like Figure 3 As shown, the S3 adaptive bidirectional masked attention (ABMA) module consists of two masked attention mechanism modules and an adaptive fusion module, AdaptiveFusion. Here, z represents the input feature with dimensions (batch_size, 2, sentence_len, dim), where 2 represents two sentences in different order, batch_size is the input batch size, sentence_len is the sentence length, and dim is the feature dimension. For simplicity, the first sentence is represented as z1, and the second sentence as z2. The expression is:

[0094] z′=ABMA(z)

[0095] ML = MHMSA L (z,z,z)

[0096] MR = MHMSA R (z,flip(z),flip(z))

[0097] ABMA(z)=AdaptiveFusion(ML,MR)

[0098] The expression for the multi-head mask attention mechanism is as follows:

[0099]

[0100] Where M represents the mask matrix of the multi-head attention mechanism, and when calculating the self-attention score, M is an upper triangular matrix. ij =0 indicates that the character at position i cannot be considered in relation to the character at position j. When mask ij =1 indicates that the character at position i in the text sequence can focus on the character at position j;

[0101] Here, flip represents the flipping operation, and the expression for the AdaptiveFusion module is:

[0102] r t ,z t =σ(W g [ML; MR])

[0103] z′=AdaptiveFusion(ML,MR)=r t ⊙ML+z t ⊙MR

[0104] in, Here are the learnable parameters, σ is the Sigmoid function used for normalization and weight calculation, and ⊙ is the bitwise product;

[0105] Furthermore, the scoring and ranking layer in the inference phase first sorts the two sentences in order, uses a length penalty mechanism to calculate the log probability of each sentence at each position, and sums them up. The sentence with the higher score is returned as the final output.

[0106]

[0107] in These are descriptive statements representing two directions.

[0108] This invention provides an end-to-end image description generation method based on an enhanced attention mechanism. It includes an image feature extraction layer, a multi-granularity feature fusion encoder, an adaptive bidirectional decoder, a linear transformation layer, and a scoring and ranking layer. The image feature extraction layer consists of two pre-trained image feature extractors connected in parallel, one capturing coarse-grained features and the other capturing fine-grained features. The multi-granularity feature fusion encoder comprises a multi-granularity feature fusion module, a dense global attention mechanism, a feedforward network layer, and a residual layer. The adaptive bidirectional decoder consists of an adaptive bidirectional mask attention mechanism, a cross-attention mechanism, a feedforward network layer, and a residual layer. The linear transformation layer consists of a linear layer and a softmax classifier, which performs dimensionality transformation on the output of the adaptive bidirectional decoder and outputs bidirectional autoregressive descriptors. Finally, the scoring and ranking layer receives the two descriptions obtained from the linear transformation layer, ranks them in order, and selects the highest-scoring description as the final output. To preserve effective features and avoid the vanishing gradient problem, each module in the multi-granularity feature fusion encoder and the adaptive bidirectional decoder is cascaded using residual connections. Dropout is also employed in each module of both modules to prevent overfitting.

[0109] Example:

[0110] An end-to-end image caption generation method based on an enhanced attention mechanism, with training divided into two stages, includes the following steps:

[0111] Step 1: Feature extraction is performed on the official image description dataset of MSCOCO2014. For the image part, pre-trained VinVL is used to capture coarse-grained features of the image, and pre-trained ResNet101 is used to capture fine-grained features of the image. For the text description part, this invention filters out words that appear less than five times, and finally forms a vocabulary containing 9487 words, while retaining image descriptions of up to 16 words.

[0112] Step 2: Input the two image features into the multi-granularity feature fusion encoder for semantic fusion;

[0113] Step 3: This invention uses two descriptive statements and corresponding images as one data set. One descriptive statement is selected from five manually annotated sentences, and the other is a non-repeating sentence selected from five manually annotated sentences and processed in reverse order. The lengths of the two sentences need to be padded. The descriptive statements in both directions are encoded using the same method and added to the positional codes of different directions respectively. The final descriptive statement encoding with positional code information is sent to the adaptive bidirectional decoder. The semantic features fused in Step 2 are also sent to the adaptive bidirectional decoder.

[0114] Step 4: The decoding features generated by the adaptive bidirectional decoder are processed by a linear transform layer to generate a decoding sequence;

[0115] Step 5: In the first training phase, cross-entropy loss is used to calculate the loss value between the generated sequence and the true value. In the second phase, self-criticism training is used to optimize the CIDEr score.

[0116] Furthermore, in step 2, the dimension of the linear mapping is set to 2048, mapping the two features to the same dimensional space. The number of multi-granularity feature fusion encoders is set to 6, and the hidden layer dimension in the multi-granularity feature fusion encoder is set to 512.

[0117] In step 3, the number of adaptive bidirectional decoders is set to 6, and the hidden layer dimension of the adaptive bidirectional decoders is set to 512.

[0118] In steps 2 and 3, residual connections and dropout are used between each module, as shown in the following formula:

[0119] X = X + dropout(LayerNorm(X))

[0120] Where X is the input information, the dropout rate is set to 0.1, and LayerNorm is used for layer normalization to improve training stability and allow the model to converge quickly;

[0121] In step 5, the training hyperparameters are set as follows: the optimizer is selected as Adma; the first training phase has 15 training iterations, a learning rate of 5e-4, a batch size of 10, and 5000 validation images; the second phase starts from the end of the first phase and trains for 5 iterations, with a learning rate of 1e-5, a batch size of 10, and 5000 validation images.

[0122] The loss function for the first training phase is:

[0123]

[0124] in This represents the calculated loss from left to right in the description statements. This represents the computational loss for describing statements from right to left, where T represents the sentence length;

[0125] The loss function for the second training phase is:

[0126]

[0127] Where R(·) represents optimization of CIDEr score, and T represents sentence length.

[0128] To demonstrate the effectiveness of this invention, a comparative experiment was conducted with the proposed method and mainstream image description models. An ablation experiment was also performed on the proposed module. The evaluation metrics used were mainstream descriptive metrics, including the following metrics as shown in Table 1:

[0129] Table 1. Introduction to Evaluation Indicators

[0130]

[0131]

[0132] The experimental results are shown in Tables 2 and 3:

[0133] Table 2 Comparison of experimental results of our method and the comparative model on the MSCOCO dataset.

[0134]

[0135] To ensure fair comparison, all models were trained using a single model setup, and both cross-entropy loss and CIDEr score were used for optimization. Experimental results show that our method and the comparison model exhibit high performance across BLEU, METEOR, ROUGE, CIDEr, and SPICE, achieving optimal results in most metrics.

[0136] Table 3 shows the ablation experimental results of this method on the MSCOCO dataset.

[0137]

[0138] Ablation experiments clearly demonstrate that employing a multi-granularity feature fusion method effectively improves the description generation performance of the baseline model. This is because multi-granularity features capture more image details without losing the original coarse-grained features. A global dense attention mechanism effectively fuses global image features, resulting in smoother features. An adaptive bidirectional mask attention mechanism effectively enhances the decoder's understanding of past and future information in the description, thereby generating more human-like natural language descriptions. This improves the model's expressive power while also enhancing its interpretability.

Claims

1. An end-to-end image description generation method based on an enhanced attention mechanism, characterized in that, The steps are as follows: Step 1: Construct an end-to-end image description generation model with an enhanced attention mechanism based on the transformer architecture. The image description generation model includes a sequentially connected feature extraction layer, a multi-granularity feature fusion encoder, an adaptive bidirectional decoder, a linear transformation layer, and a scoring and ranking layer. The feature extraction layer includes the VinVL model and the ResNet101 model. The multi-granularity feature fusion encoder includes a sequentially connected multi-granularity feature fusion, a dense global attention mechanism, and a feedforward network layer. The adaptive bidirectional decoder includes a sequentially connected adaptive bidirectional mask attention, a cross attention mechanism, and a feedforward network layer. The granularity feature fusion encoder is connected to the cross attention mechanism, and the adaptive bidirectional mask attention is connected to description statement encoding and position encoding. Step 2: Select images from the MSCOCO2014 image dataset, and select two description statements from the description statements that come with the selected images to generate a training set. One of the two description statements is a sequential description, and the other is a reverse description. Segment and filter the image description statements in the MSCOCO2014 image dataset to generate a vocabulary. Assign pre-trained parameters to the VinVL model and ResNet101 model of the image feature extraction layer. Step 3: Input the images in the training set into the image feature extraction layer to obtain the unencoded original image features. Input the original image features into the multi-granularity feature fusion encoder and use the multi-granularity feature fusion module to encode the images: fuse the original image features to obtain the image fusion features, use the dense global attention mechanism to add the global features of the image to the fusion features, and then pass through the feedforward network layer to obtain the encoded image features. Step 4: Input the two descriptions corresponding to the image in the input image feature extraction layer into the description encoding and position encoding, and input the encoded description information into the adaptive bidirectional decoder. After the adaptive bidirectional mask attention mechanism, the bidirectional description decoding features are obtained. The image features obtained in Step 3 are input into the multi-head cross attention mechanism to interact with the bidirectional description decoding features to obtain the decoding information. The decoding information is input into the feedforward network layer to output the final decoding features. Step 5: The final decoded features generated by the adaptive bidirectional decoder are mapped to the vocabulary through a linear transformation layer to obtain the final description. The final description includes two sentences: a sequential description and a reverse description. The loss is calculated by comparing the two sentences of description of the original input image with the two sentences of description. The gradient is backpropagated to correct the weights of the image description generation model until the set training is completed. Step 6: After the image description generation model is trained, it enters the inference stage. At this time, the image description generation model does not change the model weights. The image to be described is passed to the image feature extraction layer of the trained image description generation model to obtain the original image features. Then, it is passed to the multi-granularity feature fusion encoder and adaptive bidirectional decoder to generate descriptions in two directions. Finally, it is passed to the scoring and ranking module, which selects the description with the higher score from the two generated descriptions as the final output.

2. The end-to-end image description generation method based on enhanced attention mechanism according to claim 1, characterized in that, In the MSCOCO2014 image description dataset used as the training dataset, each data point contains an image and five manually annotated sentences. During the training phase, the end-to-end image description generation method based on the enhanced attention mechanism uses two description sentences and corresponding images as one data point. One description sentence is selected from the five manually annotated sentences, and the other is a sentence selected from the remaining four manually annotated sentences and processed in reverse order. The length of the two sentences needs to be padded. Finally, each image has five description pairs, and each description pair consists of a forward description and a reverse description. The image serves as the input for the training phase, and each description serves as the label for the training phase.

3. The end-to-end image description generation method based on enhanced attention mechanism according to claim 1, characterized in that, The image feature extraction layer is implemented as follows: The VinVL model object detection network was used to extract coarse-grained features C = {c1, c2, c3, ..., c} from each image in the MSCOCO2014 image description dataset. n The ResNet101 convolutional neural network is used to extract fine-grained features F = {f1, f2, f3, ..., f} from each image in the MSCOCO2014 image description dataset. n }, where the feature dimensions of C and F are d1 and d2 respectively, mapping the coarse and fine granular features to the same dimensional space, such that C o and F o The feature dimension is d0: C o =Liner(C) F o =Liner(F) Where n is the number of images in the dataset, d1 represents the feature dimension obtained by the VinVL model, d2 represents the feature dimension obtained by the ResNet101 convolutional neural network, and d0 represents the feature dimension obtained after linear mapping.

4. The end-to-end image description generation method based on enhanced attention mechanism according to claim 3, characterized in that, The implementation process of the multi-granularity feature fusion encoder is as follows: The multi-granularity feature fusion module of the multi-granularity feature fusion encoder includes a multi-head self-attention mechanism (MHSA) and a multi-head cross-attention mechanism (MSCA) set in parallel. It fuses the coarse-grained and fine-grained features of each image to obtain the multi-granularity feature F. CF This can be expressed as a formula: F CF =F MHCA (C o ,F o )+F MHSA (C o ) The number of heads is set to 8; Dense Global Attention (DGAM) employs broadcast mean and self-attention methods to integrate global features into the multi-granular features of the image to obtain x, which can be expressed by the formula: x=DGAM(F CF )=(Broadcasting(Average(F CF ))+Attention(F CF ))*0.5 The feedforward network layer can be expressed by the formula: where X is the output of the feedforward network layer: X = FFLayer(x) Multi-granularity feature fusion, dense global attention mechanism, residual connection between feedforward network layers, dropout to prevent overfitting, and regularization Norm to accelerate model convergence: X' = ​​X + dropout(Norm(X)) Where X' represents the features of the image after being encoded by a multi-granularity feature fusion encoder.

5. The end-to-end image description generation method based on enhanced attention mechanism according to claim 4, characterized in that, The adaptive bidirectional decoder expression is as follows: Caption = Decoder(X', z) Where X' is the image feature after being encoded by a multi-granularity feature fusion encoder, and the image feature X' obtained from the encoded segment interacts with the text information z at the decoding end; z is the human description sentence corresponding to an image in the MSCOCO dataset, and the text information feature obtained after description sentence encoding and position encoding. z contains description information of two word orders, with dimensions of (batch_size, 2, sentence_len, dim), where 2 represents two word orders of sentences, batch_size is the input batch, sentence_len is the sentence length, and dim is the feature dimension; The Adaptive Bidirectional Mask Attention (ABMA) module comprises two parallel mask attention mechanism modules. These two parallel mask attention mechanism modules interact with an Adaptive Fusion module. The adaptive fusion module processes the feature expressions obtained from the two mask modules as follows: z' = ABMA(z) ABMA(z)=AdaptiveFusion(MHMSA L (z,z,z),MHMSA R (z,flip(z),flip(z))) Where z' represents the feature after ABMA processing, flip represents the flipping operation, and the expression for the AdaptiveFusion module is: r t ,z t =σ(W g [M L ;M R ]) AF=AdaptiveFusion(M L ,M R )=r t ⊙M L +z t ⊙M R Where M L M R The output features of the two masked attention mechanisms are... Here, σ is the learnable parameter, σ is the sigmoid function used for normalization and weight calculation, ⊙ is the bitwise product, and r is the weight. t ,z t The result is obtained by concatenating the two mask attention mechanisms according to the last dimension and then normalizing the result. The masked attention mechanism uses the multi-head method MHMSA, as shown in the following expression: Where, d K It is a scaling factor, a mapping parameter matrix. The number of heads is set to 8 to be learnable parameters, and Q, K, and V represent the features after mapping by the mapping parameter matrix. The number of heads in the multi-head cross-attention module is set to 8, and the expression is: Z=F MHCA (X,z’) The feedforward network layer can be expressed by the following formula: Z' = FFLayer(Z) The adaptive bidirectional masked attention module, cross-attention module, and feedforward network layers are connected using residual connections, and dropout is set to prevent overfitting. Regularization Norm is set to speed up the model's convergence. Caption=Z'+dropout(Norm(Z')).

6. The end-to-end image description generation method based on enhanced attention mechanism according to claim 5, characterized in that: The linear transformation layer consists of a linear layer and a softmax layer. The input caption dimension of the linear layer is (batch_size, 2, sentence_len, dim), and the output dimension of the linear layer is (batch_size, 2, sentence_len, vocab_size), where 2 represents sentences with two word orders, and vocab_size is the size of the vocabulary.

7. The end-to-end image description generation method based on enhanced attention mechanism according to claim 1, characterized in that, The specific method for the scoring and ranking layer is as follows: During the inference phase of the scoring and ranking layer, the layer first sorts the two sentences obtained from the decoder in order. Using a length penalty mechanism, it calculates the log probability of each sentence at each position, sums them up, and returns the sentence with the higher score as the final output. The expression is as follows: in These are descriptive statements representing two directions.

8. The end-to-end image description generation method based on enhanced attention mechanism according to claim 6, characterized in that, The specific method for outputting image descriptions through the constructed image description generation model is as follows: Step 3.1: Perform feature encoding; assuming the input is an image I, first use the image feature extraction layer to extract its coarse-grained and fine-grained features (C). o ,F o Secondly, the coarse-grained feature (C) is used to further refine the granularity characteristics. o ,F o X is fed into a multi-granularity feature fusion encoder as the feature to be encoded to obtain X; Step 3.2: Output the target features; align X with the cross-attention mechanism in the adaptive bidirectional decoder and the encoding of the descriptive statement; The description information consists of two different word orders for image I. The description information is encoded and then the position-encoded description features z are added to each description information and used as input to the final description information Caption of the adaptive bidirectional decoder. Step 3.3: Output the final statement; The final caption information from the adaptive bidirectional decoder is input into the linear transform layer to map the vocabulary, generating the final description statement. The loss is calculated by comparing the final description statement with the ground truth label, and then backpropagation is used to optimize the image caption generation model.

9. The end-to-end image description generation method based on enhanced attention mechanism according to claim 1, characterized in that, The training process for the gradient backpropagation-corrected image description generation model's weights employs a two-stage training approach. The specific loss functions for the two training stages are as follows: The loss function for the first training phase is cross-entropy loss, used to quickly update the weights. The expression is: in This represents the calculated loss from left to right in the description statements. This represents the computational loss for describing statements from right to left, where T represents the sentence length and p... θ Describe the probability of the output word; The second training phase uses a more refined loss function, expressed as follows: Where R(·) represents optimization of CIDEr score, and T represents sentence length.

10. The end-to-end image description generation method based on enhanced attention mechanism according to claim 1, characterized in that, The specific process of the image description generation method in the inference stage is as follows: After training, the model parameters are frozen, and the inference phase begins. An image is input into the image description generation model. After forward computation, the text generator outputs the word IDs corresponding to the two words with the highest probabilities each time. After searching the dictionary, the image descriptions of the two inferred word orders are obtained. Then, the scoring and ranking layer outputs the description with the highest score.

Images