Adaptive feature reconstruction image description generation method based on Transform

Abstract

The invention discloses an adaptive feature reconstruction image description generation method based on Transform. The method comprises the following steps: 1) extracting grid features of an input image; 2) respectively feeding the extracted grid features into a feature reconstruction module and a channel attention module; 3) fusing the features after feature reconstruction with features output by a channel attention module, and inputting a feature image into an encoder by using residual connection for visual feature encoding; 4) feeding the output features of the encoder and the extracted language features into the encoder for decoding to obtain implicit information representation; and 5) fusing the visual features and the language features to predict the generated words. According to the method, the adaptive feature is adopted to reconstruct the Transform model, more complete visual features are learned through the feature reconstruction module and the self-attention codec fusing the channel weight, and meanwhile, the feature screening capability is enhanced, so that the matching of the image subtitles is more accurate.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular, relates to a Transformer-based adaptive feature reconstruction image description generation method. Background Art

[0002] Image description generation is a research field that combines computer vision and natural language processing. Its goal is to generate a natural language text describing a scene by inputting a scene image and processing it by computer. This task requires the computer to learn the correspondence between images and text, and to match the visual information in the image with the text information, while meeting the requirements of grammatical correctness and content readability. Image description generation has a wide range of applications. It can be used to help the blind recognize objects, assist people to quickly familiarize themselves with the surrounding environment, and realize the pre-task of text retrieval images.

[0003] With the development of science and technology and the improvement of people's living standards, a large amount of video and image information can be obtained. Among them, scene images contain rich information, which makes people feel intuitive, and the text information is accurate and can accurately capture the required information. Therefore, how to convert scene images into the required text information is a key issue in image description generation. However, there are many difficulties and challenges in image description generation, such as incorrect or inaccurate descriptions. This is because image description generation requires computers to process image and text information at the same time, and requires a deep understanding of the content in the image, which requires computers to have strong visual understanding and natural language processing capabilities.

[0004] To solve these problems, researchers have proposed many different methods and techniques, such as using deep learning models to learn the mapping relationship between images and text, using attention mechanisms to improve the model's attention to important areas in the image, and using semantic analysis techniques to improve the accuracy of the generated text information. Although some progress has been made, image description generation is still a challenging research field that requires continued in-depth research and exploration. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a Transformer-based adaptive feature reconstruction image description generation method, which solves the problems raised in the above-mentioned background technology.

[0006] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0007] A method for generating image description based on adaptive feature reconstruction of Transformer, comprising the following steps:

[0008] 1) Use ResNext-101 as the backbone network to extract the grid features of the input image;

[0009] 2) Feed the extracted grid features into the feature reconstruction module and the channel attention module respectively;

[0010] 3) The reconstructed features are fused with the features output by the channel attention module. After weighing the importance of different features, the feature image is input into the Transformer encoder that fuses the channel weights for visual feature encoding using a residual connection.

[0011] 4) The encoder output features and the language features extracted using the BERT pre-training model are fed into the Transformer encoder with fused channel weights for decoding to obtain implicit information representation;

[0012] 5) The visual features output by the encoder, the language features extracted by the BERT pre-trained model, and the implicit information through the masked multi-head attention mechanism and the multi-head attention mechanism are fused to predict the generated words.

[0013] Step 1) The specific process is:

[0014] ResNext-101 is used as the backbone network for grid feature extraction. When extracting features from an image, each image is first divided into 7×7 grids, and then a pair of two-dimensional relative positions are calculated for each grid.

[0015]

[0016] In the formula, and Represents the center coordinates of grid i, and the coordinate position of the upper left corner of grid i is The coordinate position of the lower right corner of grid i is The relative coordinate position δ of the two grids i and j ij for:

[0017]

[0018] Step 2) specifically includes the following steps:

[0019] 201) feeding the extracted mesh features into a feature reconstruction module;

[0020] 202) The extracted grid features are fed into the channel attention module.

[0021] The specific process of step 201) is:

[0022] The feature reconstruction module consists of shift convolution, dilated convolution and grouped convolution. The grid features extracted by the ResNext-101 backbone network are fed into the feature reconstruction module to reconstruct feature information. The shift convolution is used to achieve cross-channel information fusion. The dilated convolution expands the receptive field from 3×3 to 7×7. The grouped convolution divides the number of channels into 4 groups for convolution. The feature reconstruction process can be defined as:

[0023]

[0024] R=(c shift +c empty +c group )F g (5)

[0025] In the formula, F g is the relative geometric feature vector obtained, FC is fully connected, ReLU is the activation function, represents the learned weight, R is the feature reconstruction vector result, c shift represents shift convolution, c empty represents the dilated convolution, c group Represents grouped convolution.

[0026] The specific process of step 202) is:

[0027] The channel attention module performs global average pooling at the channel level without reducing the dimension, and captures local cross-channel interaction information through each channel and k adjacent channels; the grid features are extracted through 1×1 convolution at the channel level, and then pass through the Sigmoid function layer, and finally the residual connection output features. The channel attention module can be defined as:

[0028] A=f×sigmoid(c 1×1，k (f)) (6)

[0029] In formula (6), c 1×1，k represents a 1×1 convolution with a kernel size of k, f is the input feature, and A represents the output feature of the channel attention module.

[0030] The specific process of step 3) is:

[0031] The reconstructed features are fused with the features output by the channel attention module. After weighing the importance of different features, the feature image is input into the Transformer encoder of the fused channel weights for visual feature encoding using residual connection. The fusion process is defined as:

[0032] I input =(A+R)×μ+I temp (7)

[0033] In formula (7), μ is the balance coefficient between the feature reconstruction module and the channel attention module, which is used to control the importance of different features and its value is 0.2; input Represents the input of the Transformer encoder, I temp Represents the value before residual connection.

[0034] The specific process of step 4) is:

[0035] The output features of the designed channel attention module are fed into the Transformer to form a self-attention encoder-decoder that integrates channel weights, thereby enhancing the ability of feature screening;

[0036] The core of the encoder consists of matrices Q, K, and V, where Q is a combination of query vectors, K is a combination of key vectors, and V is a combination of value vectors; in the encoder, the original image features are first converted to 1D information, then embedded by a fully connected layer, followed by a ReLU and a dropout layer, and the embedded features are input to the first encoder layer of the Transformer model. The encoder consists of multiple identical layers stacked together, each layer consists of a multi-head self-attention mechanism layer and a feedforward network sublayer, and the output is residually connected and softmax normalized. Scaling the attention mechanism by dot product:

[0037]

[0038] In formula (8), d k is the dimension of V, D is the obtained weight matrix, and temp represents the value before the residual connection. The number of self-attention heads is set to 8. After the multi-head attention mechanism, 8 weight matrices are obtained. Then the 8 weight matrices are concatenated and multiplied with an additional weight matrix to obtain an output vector of the same size as the input word vector to obtain the output of the self-attention layer:

[0039] I out =concate(D i )M W (9)

[0040] In formula (9), D i represents the weight matrix of the i-th head, M W Represents an additional weight matrix, and concate represents concatenation of the weight matrix.

[0041] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0042] (1) In the present invention, the representation ability and multimodal fusion ability of the Transformer-based image description model are further improved by designing a feature reconstruction module and a channel attention module. Specifically, the feature reconstruction module can reconstruct and fuse the feature information in the image, thereby improving the model's representation ability for complex features in the image. The channel attention module can filter the visual features in the image and redistribute word weights, thereby achieving information exchange and fusion between different channels and improving the model's attention to important features in the image;

[0043] In general, the Transformer-based image description model is a model with better representation ability, language model ability, multimodal fusion ability and interpretability, and has obvious advantages over existing image description generation technologies. In the present invention, by designing a feature reconstruction module and a channel attention module, the representation ability and multimodal fusion ability of the Transformer-based image description model are further improved, thereby achieving more efficient and accurate image description generation;

[0044] (2) The feature reconstruction module is a module used to reconstruct and fuse image feature information. Compared with existing image feature extraction technologies, it has the advantages of expanding the receptive field, reducing the number of parameters, realizing cross-channel information fusion, and improving model generalization capabilities. In the image description generation task, the grid features extracted by ResNext-101 can be fed into the feature reconstruction module for feature reconstruction and fusion, thereby improving the accuracy and fluency of the final generated image description;

[0045] (3) The adaptive channel attention module is a module used to filter and redistribute word weights for visual features in an image. Compared with existing image description generation technologies, it has the advantages of increasing the model's attention to important features, reducing the impact of the model on noise and irrelevant features, achieving information exchange and fusion between different channels, and improving the model's generalization ability. In the image description generation task, the grid features extracted by ResNext-101 can be fed into the adaptive channel attention module for filtering and redistributing word weights, thereby improving the quality of the final generated image description.

[0046] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0048] Figure 1An adaptive feature reconstruction image description generation model constructed for the present invention;

[0049] Figure 2 A feature reconstruction module designed for the present invention;

[0050] Figure 3 It is the channel attention module of the present invention.

[0051] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] The present invention will now be described in further detail with reference to the accompanying drawings.

[0053] See also Figure 1-3 As shown, in this embodiment, a method for generating image description based on adaptive feature reconstruction of Transformer is provided, comprising the following steps:

[0054] 1) Image preprocessing: First, the input image is preprocessed, including adjusting the image to the same size, normalizing the image, etc., to facilitate subsequent feature extraction.

[0055] 2) Feature extraction: Then, ResNext-101 is used as the backbone network to extract features from the preprocessed image and obtain grid features.

[0056] 3) Feature reconstruction and channel attention: The extracted grid features are fed into the feature reconstruction module and the channel attention module respectively for feature reconstruction and channel attention operations. The feature reconstruction module uses convolution kernels of different scales to extract features from the image, expand the receptive field, reduce the number of parameters, realize cross-channel information fusion, and improve the generalization ability of the model. The channel attention module selects visual features to redistribute word weights, improve the model's attention to important features, reduce the model's impact on noise and irrelevant features, realize information exchange and fusion between different channels, and improve the generalization ability of the model.

[0057] 4) Feature fusion and encoding: The features output by the feature reconstruction module and the channel attention module are fused. After weighing and controlling the importance of different features, the feature image is input into the Transformer encoder with fused channel weights using residual connection for visual feature encoding. At the same time, the input text is encoded using the BERT pre-trained model to obtain language features.

[0058] 5) Decoding and prediction: The visual features output by the encoder, the language features extracted by the BERT pre-trained model, and the implicit information obtained by the masked multi-head attention mechanism and the multi-head attention mechanism are combined to predict the generated words. Specifically, the visual features output by the encoder and the language features extracted by the BERT pre-trained model are first concatenated as the input of the decoder. Then, the masked multi-head attention mechanism and the multi-head attention mechanism are used in the decoder to encode and decode the input sequence to obtain the implicit information representation. Finally, the implicit information representation is input into a linear layer for prediction and generation of the final image description.

[0059] 6) Model training and evaluation: During the model training process, the cross entropy loss function is used to optimize and train the model. During the model evaluation process, the generated image descriptions are evaluated and analyzed using evaluation indicators such as BLEU, ROUGE, and CIDEr.

[0060] The model uses cross entropy loss to optimize the model training, and then uses the self-critical method to perform reinforcement learning training. The reward mechanism of reinforcement learning uses CIDEr's score function to minimize its negative expected value:

[0061]

[0062] In the formula, θ is the model parameter, E GT is the description word sequence of Ground Truth, and T represents the length of the word sequence. The expected gradient of a single sample can be approximated as:

[0063]

[0064] In formula (15), reward(·) represents the reward function of CIDEr. n is the number of sampling sequences. Specifically, it includes the following steps:

[0065] 1) Use ResNext-101 as the backbone network to extract the grid features of the input image;

[0066] In the image description generation task, ResNext-101 is used as the backbone network to extract the grid features of the input image. Specifically, each input image is first divided into 7×7 grids, and each grid is the same size. Then, for each grid, the position of its center coordinates and upper left corner coordinates is calculated. The center coordinates can be obtained by calculating the average of the upper left corner coordinates and the lower right corner coordinates of the grid.

[0067] After calculating the coordinate position of each grid, we then perform feature extraction on each grid. Specifically, we use the pixel value of each grid as input and forward propagate through the ResNext-101 network to obtain the feature vector of each grid.

[0068] After obtaining the feature vector of each grid, these feature vectors are then used as input to calculate the attention weight of each grid through an attention mechanism. Specifically, the feature vector of each grid is dot-producted with a learnable query vector, and the dot product value is normalized to the range of [0,1] through the softmax function to obtain the attention weight of each grid. Then, the feature vector of each grid and its corresponding attention weight are weighted averaged to obtain a feature vector representing the entire image. After obtaining the feature vector representing the entire image, it is then used as input and decoded through a sequence-to-sequence model to obtain the final image description.

[0069] 2) Feed the extracted grid features into the feature reconstruction module and the channel attention module respectively;

[0070] 201) The feature reconstruction module is a module for reconstructing and fusing image feature information, which consists of convolution operations such as shift convolution, dilated convolution, and grouped convolution. The extracted grid features are fed into the feature reconstruction module to reconstruct and fuse the feature information. In the feature reconstruction module, shift convolution can be used to achieve cross-channel information fusion.

[0071] Specifically, shift convolution can move the feature information of one channel to another channel, thereby realizing information exchange and fusion between different channels. Dilated convolution can be used to expand the receptive field from 3×3 to 7×7. Dilated convolution is a convolution operation that sets some elements in the convolution kernel to 0 to expand the receptive field. Compared with traditional convolution operations, dilated convolution can expand the receptive field and improve the representation ability of the model without increasing the number of parameters. Grouped convolution can divide the number of channels into 4 groups for convolution. Grouped convolution is a convolution operation that divides the number of channels into multiple groups, and each group performs convolution operations independently. Compared with traditional convolution operations, grouped convolution can reduce the number of parameters to a certain extent and improve the computational efficiency of the model. ;

[0072] 202) The channel attention module is a module used to capture local cross-channel interaction information in an image. It usually performs global average pooling at the channel level without reducing the dimensionality to aggregate and characterize the feature information of each channel;

[0073] The extracted grid features are fed into the channel attention module to aggregate and characterize the feature information. First, a 1×1 convolution operation with a kernel size of k is performed on the input grid features at the channel level to aggregate and characterize the feature information of each channel and its k adjacent channels. This operation can capture local cross-channel interaction information in the image and improve the representation ability of the model;

[0074] After aggregating the local cross-channel interaction information, the aggregated channel features are then activated by the Sigmoid function layer to normalize and compress the channel attention vector. Finally, the channel attention vector is element-wise multiplied with the input grid features and a residual connection is performed to re-weight and fuse the feature information.

[0075] 3) The reconstructed features are fused with the features output by the channel attention module. After weighing the importance of different features, the feature image is input into the Transformer encoder that fuses the channel weights for visual feature encoding using a residual connection.

[0076] The feature reconstruction module and channel attention module are modules used to reconstruct and aggregate image feature information. After feature reconstruction and channel attention operations, the reconstructed features can be fused with the features output by the channel attention module to measure and control the importance of different features.

[0077] Specifically, the features output by the feature reconstruction module and the features output by the channel attention module are element-wise added, and then multiplied by the balance coefficient that measures the importance of different features to achieve re-weighting and fusion of feature information. Then, the fused features can be residually connected to retain the original feature information and further improve the representation ability of the model. After that, the fused feature image is input into the Transformer encoder for visual feature encoding.

[0078] 4) The encoder output features and the language features extracted using the BERT pre-training model are fed into the Transformer encoder with fused channel weights for decoding to obtain the implicit information representation, specifically:

[0079] The output features of the designed channel attention module are fed into the Transformer to form a self-attention encoder-decoder that integrates channel weights, thereby enhancing the ability of feature screening.

[0080] The core of the encoder consists of matrices Q, K, and V, where Q is the combination of query vectors, K is the combination of key vectors, and V is the combination of value vectors. In the encoder, the original image features are first converted to 1D information, then embedded by a fully connected layer, followed by a ReLU and a dropout layer, and the embedded features are input to the first encoder layer of the Transformer model. The encoder consists of multiple identical stacked layers, each of which consists of a multi-head self-attention mechanism layer and a feedforward network sublayer. The output is residually connected and softmax normalized. Scaling the attention mechanism by dot product:

[0081]

[0082] In formula (8), d k is the dimension of V, D is the obtained weight matrix, and temp represents the value before the residual connection. The number of self-attention heads is set to 8. After the multi-head attention mechanism, 8 weight matrices are obtained. Then the 8 weight matrices are concatenated and multiplied with an additional weight matrix to obtain an output vector of the same size as the input word vector to obtain the output of the self-attention layer:

[0083] I out =concate(D i )M W (9)

[0084] In formula (9), D i represents the weight matrix of the i-th head, M W Represents an additional weight matrix, and concate represents concatenation of the weight matrix.

[0085] The BERT pre-trained language model is used to extract and process language features. Each layer of the decoder consists of a masked multi-head attention module and an encoder-decoder attention module. The masked multi-head attention module is used to hide future information generated by the model to avoid information interference.

[0086] In the encoder, Q is the attribute of the decoder, K and V are the attributes of the encoder, and are the final output of the encoder. The decoder fuses the language signal, visual signal, and implicit information during prediction to predict the generated words. The decoder processes the sequence features of words through word embedding and combines them into the word sequence position encoding. It encodes the sentence sequence to be generated according to the word vector, adds its position vector as the sequence input feature E, and obtains the weighted sequence feature after passing through the masked multi-head attention layer:

[0087] W E =Norm(E+MaskHead(E,E,E)) (10)

[0088] In formula (10), MaskHead represents masked multi-head attention, and Norm is the normalization operation. The multi-head attention module performs weighted attention between image and text modalities on the fusion output result of the multimodal encoder and the text input, and obtains the image description attention weighted matrix W guided by image information. I :

[0089] W I =Norm(W E +head(W E ,result,result)) (11)

[0090] In formula (11), head represents multi-head attention. After passing through multiple layers of decoding layers, a multi-head attention layer and a feed-forward connection layer are used to fuse the encoded output with the sequence features of the last layer, and finally a log_softmax layer is passed to generate the final predicted probability distribution:

[0091] P E =log_softmaxFFN(W I ) (12)

[0092] In formula (12), FFN represents the feed-forward connection layer;

[0093] 5) The visual features output by the encoder, the language features extracted by the BERT pre-trained model, and the implicit information through the masked multi-head attention mechanism and the multi-head attention mechanism are fused to predict the generated words.

[0094] Embodiment 1:

[0095] The present invention uses the Microsoft COCO 2014 data set to verify the model performance. The data set has a total of 164,062 images, of which 82,783 are training images, 40,504 are verification images, and 40,775 are test images, each of which has 5 corresponding English titles. The data set segmentation method proposed by Karpath et al. was used for testing. The operating system is Linux, and the deep learning framework uses Pytorch; the hardware configuration for training and testing is: Intel i7-11700K CPU processor, and the GPU uses Gigabyte AORUS RTX 3080Ti graphics card. The Adam optimizer is used to train the model. The initial learning rate of the model is set to 2e-4, the termination learning rate is 0, the training parameters are at least 20 epochs and at most 25 epochs, and after the training, reinforcement learning is used for fine-tuning, the learning rate is set to 1e-5, and the batch size of the network receivable data volume is set to 50.

[0096] The model evaluation adopts the objective quantitative scoring methods commonly used in image description generation, including BLEU1-4, ROUGE_L, METEOR, CIDEr and SPICE. The BLEU value analyzes the overlap between the N-tuples in the evaluated sentence and the Ground Truth, but BLEU has limitations in the accuracy of sentence expression. The ROUGE value was first used to evaluate the quality of text summarization models, among which ROUGE-L was used to evaluate image descriptions, calculating the length of the longest common subsequence between the evaluated sentence and Ground Truth, but this evaluation indicator tends to ignore the influence of the alternative longest subsequence and shorter subsequences. METEOR is an improvement on BLEU, using WordNet to calculate specific sequence matching, synonyms, roots and affixes, and matching relationships between interpretations, but the disadvantage is that it requires external knowledge sources. CIDEr is specifically proposed for image description tasks, calculating the cosine similarity between GT (Ground Truth) and the evaluated sentence to measure the quality of image annotation. SPICE is an indicator for image description tasks, which can achieve the purpose of measuring the coverage of the generated description for the entities contained in the image and the relationships between entities, so it is widely used in image description. The higher the scores of evaluation indicators such as BLEU1-4, ROUGE_L, METEOR, CIDEr, and SPICE, the better the performance of the model for image description.

[0097] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. A Transformer-based adaptive feature reconstruction image description generation method, characterized in that: The steps include: 1) Use ResNext-101 as the backbone network to extract the grid features of the input image; 2) Feed the extracted grid features into the feature reconstruction module and the channel attention module respectively; 3) The reconstructed features are fused with the features output by the channel attention module. After weighing the importance of different features, the feature image is input into the Transformer encoder that fuses the channel weights for visual feature encoding using a residual connection. 4) The encoder output features and the language features extracted using the BERT pre-training model are fed into the Transformer encoder with fused channel weights for decoding to obtain implicit information representation; 5) The visual features output by the encoder, the language features extracted by the BERT pre-trained model, and the implicit information through the masked multi-head attention mechanism and the multi-head attention mechanism are fused to predict the generated words.

2. The method for generating image description based on adaptive feature reconstruction of Transformer according to claim 1, characterized in that: Step 1) The specific process is: ResNext-101 is used as the backbone network for grid feature extraction. When extracting features from an image, each image is first divided into 7×7 grids, and then a pair of two-dimensional relative positions are calculated for each grid. In the formula, and Represents the center coordinates of grid i, and the coordinate position of the upper left corner of grid i is The coordinate position of the lower right corner of grid i is The relative coordinate position δ of the two grids i and j ij for:

3. The method for generating image description based on adaptive feature reconstruction of Transformer according to claim 1, characterized in that: Step 2) specifically includes the following steps: 201) feeding the extracted mesh features into a feature reconstruction module; 202) The extracted grid features are fed into the channel attention module.

4. The method for generating image description based on adaptive feature reconstruction of Transformer according to claim 3, characterized in that: The specific process of step 201) is: The feature reconstruction module consists of shift convolution, dilated convolution and grouped convolution. The grid features extracted by the ResNext-101 backbone network are fed into the feature reconstruction module to reconstruct feature information. The shift convolution is used to achieve cross-channel information fusion. The dilated convolution expands the receptive field from 3×3 to 7×7. The grouped convolution divides the number of channels into 4 groups for convolution. The feature reconstruction process can be defined as: R=(c shift +c empty +c group )F g (5) In the formula, F g is the relative geometric feature vector obtained, FC is fully connected, ReLU is the activation function, represents the learned weight, R is the feature reconstruction vector result, c shift represents shift convolution, c empty represents the dilated convolution, c group Represents grouped convolution.

5. The method for generating image description based on adaptive feature reconstruction of Transformer according to claim 3, characterized in that: The specific process of step 202) is: The channel attention module performs global average pooling at the channel level without reducing the dimension, and captures local cross-channel interaction information through each channel and k adjacent channels; the grid features are extracted through 1×1 convolution at the channel level, and then pass through the Sigmoid function layer, and finally the residual connection output features. The channel attention module can be defined as: A=f×sigmoid(c 1×1，k (f)) (6) In formula (6), c 1×1，k represents a 1×1 convolution with a kernel size of k, f is the input feature, and A represents the output feature of the channel attention module.

6. The method for generating image description based on adaptive feature reconstruction of Transformer according to claim 1, characterized in that: The specific process of step 3) is: The reconstructed features are fused with the features output by the channel attention module. After weighing the importance of different features, the feature image is input into the Transformer encoder of the fused channel weights for visual feature encoding using residual connection. The fusion process is defined as: I input =(A+R)×μ+I temp (7) In formula (7), μ is the balance coefficient between the feature reconstruction module and the channel attention module, which is used to control the importance of different features and its value is 0.2; input Represents the input of the Transformer encoder, I temp Represents the value before residual connection.

7. The method for generating image description based on adaptive feature reconstruction of Transformer according to claim 2, characterized in that: The specific process of step 4) is: The output features of the designed channel attention module are fed into the Transformer to form a self-attention encoder-decoder that integrates channel weights, thereby enhancing the ability of feature screening; The core of the encoder consists of matrices Q, K, and V, where Q is a combination of query vectors, K is a combination of key vectors, and V is a combination of value vectors; in the encoder, the original image features are first converted to 1D information, then embedded by a fully connected layer, followed by a ReLU and a dropout layer, and the embedded features are input to the first encoder layer of the Transformer model. The encoder consists of multiple identical layers stacked together, each layer consists of a multi-head self-attention mechanism layer and a feedforward network sublayer, and the output is residually connected and softmax normalized. Scaling the attention mechanism by dot product: In formula (8), d k is the dimension of V, D is the obtained weight matrix, and temp represents the value before the residual connection. The number of self-attention heads is set to 8. After the multi-head attention mechanism, 8 weight matrices are obtained. Then the 8 weight matrices are concatenated and multiplied with an additional weight matrix to obtain an output vector of the same size as the input word vector to obtain the output of the self-attention layer: I out =concate(D i )M W (9) In formula (9), D i represents the weight matrix of the i-th head, M W Represents an additional weight matrix, and concate represents concatenation of the weight matrix.

Images