A low-illumination image description method based on branch prediction

Abstract

The application discloses a low-illumination image description method based on branch prediction, comprising the following steps: 1) processing an image description data set; 2) establishing an image description word library; 3) using CLIP to encode image and text data; 4) obtaining an image word pool through a semantic filter from the features obtained by encoding; 5) enhancing and branch predicting the image to obtain a group of attention feature maps optimized for different subjects; 6) obtaining text features with high matching degrees with the feature maps by combining each attention feature map with the word pool; and 7) decoding the obtained text features and image features to generate a description sentence. The application reduces the dependence of the model on low-illumination image features by using the branch prediction method, and can effectively improve the performance of the image description model in a low-illumination environment.

Description

Technical Field

[0001] This invention relates to image captioning technology and low-light image processing technology, specifically a low-light image captioning method based on branch prediction. Background Technology

[0002] Image description is the process of describing the content of an image using text. Its goal is to express the key information and features of the image through words, enabling people to understand its content even if they cannot directly observe the image. Image description typically involves identifying elements such as objects, scenes, and actions in the image, and using natural language to describe their appearance, relationships, and contextual information.

[0003] Image captioning is typically achieved through a combination of computer vision and natural language processing. Computer vision techniques are used to detect and recognize objects and features in images, such as object detection, object recognition, and scene understanding. Natural language processing techniques then transform the information in the image into human-readable text descriptions. In practice, image captioning often utilizes deep learning techniques, particularly Convolutional Neural Networks (CNNs) for image processing and Recurrent Neural Networks (RNNs) for generating descriptive sentences. During training, the model receives an image as input and outputs a description corresponding to the image.

[0004] Image description has applications in many fields. Early applications included image annotation, image search, and accessibility technologies. With the development of image description technology, it has become a key technology for transforming image recognition into intelligent services in other fields. It provides people with a more intuitive and comprehensive way to provide intelligent services through text (which can be converted to speech), such as autonomous driving systems that can provide voice-guided updates on the road ahead. It also lays the foundation for better intelligent services from computers.

[0005] Low-light images are images captured in dim light, resulting in a blurry and dim appearance. Common low-light images include nighttime landscapes and photographs taken in low-light indoor environments. The main characteristics of low-light images include low contrast, blurred details, color distortion, and high noise levels.

[0006] For low-light image description tasks, existing research often uses an augmentation-then-description approach. This involves first converting the low-light image to normal illumination, and then using a description model to obtain a descriptive statement. Unlike computer vision tasks such as detection and segmentation, image description tasks focus more on matching the image domain with the text domain. This two-stage augmentation-then-description approach leads to a significant shift in the original image domain, resulting in missed detections and false detections, thus reducing the ability to describe low-light images. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a low-light image description method based on branch prediction. This method avoids the domain shift problem caused by enhancing the original image, thereby reducing the impact of feature loss in low-light images on the image description model and improving the performance of the low-light image description model.

[0008] The technical solution to achieve the objective of this invention is:

[0009] A low-light image description method based on branch prediction includes the following steps:

[0010] 1) Processing Image Description Datasets: Processing image description datasets reduces image brightness to obtain low-light image datasets, including:

[0011] 1-1) The publicly available Visual Genome dataset is used. The Visual Genome dataset not only provides annotation information for descriptive sentences in image description tasks, but also provides annotation information for the Region Graph and Scene Graph of the image. Different visual elements and relationships in the image are annotated. Training the model with the Visual Genome dataset can achieve better performance in capturing visual elements and their relationships.

[0012] 1-2) Apply ForkGAN to the images to perform day-night transformation, reduce the brightness of the images in the dataset, and obtain the low-light dataset Visual Genome-Night;

[0013] 1-3) The Visual Genome-Night dataset is divided into training and validation sets. The dataset contains 108,077 images. The training and validation sets are divided in an 8:2 ratio. 80% of the dataset is the training set, which contains 86,462 images; 20% of the dataset is the validation set, which contains 21,615 images.

[0014] 2) Establish an image descriptive lexicon: Extract and filter the descriptive text from the dataset to obtain an image descriptive lexicon, including:

[0015] 2-1) Extract the descriptive text of all images in the Visual Genome dataset, merge all text into a large text file, and split the large text file into words using spaces, commas, and periods, and calculate the frequency of word occurrences.

[0016] 2-2) Filter words by removing words that appear less than 3 times, and remove distractors such as punctuation marks, numbers, and irrelevant stopwords.

[0017] 2-3) Normalize the vocabulary by converting all words in the vocabulary to lowercase to reduce the impact of word variants on model training and avoid misclassifying words with different capitalizations as different words. At the same time, map all variants of related words to a stem, such as normalizing variants of “played”, “playing”, and “playful” to “play”.

[0018] 2-4) Integrate the remaining words into one file to form the final vocabulary;

[0019] 3) Constructing an image encoder and a text encoder to encode text and images: CLIP is used to encode image and text data, including:

[0020] 3-1) Preprocess the training images by scaling them to 512*512 pixels to ensure that the size of the input data is consistent with the expected input size of the model and to avoid size mismatch errors. At the same time, important details in the small images are preserved. Preprocess the text information by combining related words in the vocabulary into r short sentence templates.

[0021] 3-2) The CLIP-I image encoder and CLIP-T text encoder are constructed using the CLIP-ViT-B / 32 model, and the pre-trained weights of CLIP are loaded.

[0022] 3-3) Encoding the input image and text: The CLIP-I image encoder receives the image input and outputs the encoded image features V. c The CLIP-T text encoder receives text input and outputs encoded text features T. c ;

[0023] 4) Construct a semantic filter module: This module encodes the feature V obtained in step 3-3). c and T c A semantic filter is used to obtain a word pool for the image, including:

[0024] 4-1) For the input image features V c and text features Tc A fully connected linear and ReLU activation layer with 512 input and output channels is used to perform a learnable mapping, giving the input features stronger representational power, while simultaneously transforming the image features V... c and text features T c Mapped to the same dimension;

[0025] 4-2) Calculate the image features V obtained after passing through a fully connected (linear) layer and a ReLU activation layer. c and text features T c The cosine similarity between them is used to obtain the corresponding cosine similarity matrix M. sim ;

[0026] 4-3) For the cosine similarity matrix M sim A softmax layer is used to map the image to a probability vector P with respect to the vocabulary. t Then, based on the set threshold T=0.2, the word pool elements with a probability higher than the threshold T are selected, and the elements in the image and their relationships are mapped to the corresponding elements in the word pool, i.e., the word pool corresponding to the image.

[0027] 5) Constructing the enhancement module and branch attention module: The image is enhanced and branched to obtain a set of attention feature maps optimized for different subjects. These are used to capture different elements and their relationships in the image, specifically:

[0028] 5-1) Constructing the auxiliary enhancement module: First, the image is histogram equalized to initially adjust the overall brightness of the image. Then, 2x upsampling and 2x downsampling operations are used to obtain two images equivalent to 4 times and 1 / 4 of the original image size. These images are combined with the original image to form an image pyramid. The images of different sizes in the image pyramid are processed by 3 layers of 3*3 convolution and ReLU pooling to extract features. Then, the extracted features are downsampled to 1 / 4 of the original image size and summed pixel by pixel to obtain the enhanced features, which are used as the input K of Masked Multi-Head Attention in the self-attention branch module.

[0029] 5-2) The input image first passes through three convolutional layers. Convolutional layers 1 and 2 do not change the size of the feature map and use residual connections to preserve image feature information. Convolutional layer 3 reduces the width and height of the feature map to half of its original size and uses max pooling to downsample the original image. The downsampled feature map is then added to the feature map obtained from convolutional layer 3. Convolutional layer 1 uses a 3x3 kernel, a padding of 1, 3 input channels, and 64 output channels. Convolutional layer 2 uses a kernel of... The first convolutional layer is a 3x3 convolution with a pad of 1 and 64 input and output channels. The second convolutional layer is a 2x2 convolution with a stride of 2, which reduces the width and height of the feature map to half of its original size. The third convolutional layer uses residual connections to add the original image to the outputs of the first and second convolutional layers pixel by pixel. The input image is downsampled using max pooling with a 2x2 kernel and a stride of 2. The downsampled feature map is then added pixel by pixel to the feature map obtained from the third convolutional layer.

[0030] 5-3) Calculate the attention weight map of the preliminary feature map obtained in step 5-2) using a multi-head self-attention module;

[0031] 5-4) Constructing a branch module to separate the main body from the attention map obtained in step 5-3): First, the feature map is downsampled by 7*7, i.e., the feature map is reduced in dimensionality to obtain a small-sized attention map. Then, a 5*5 convolution kernel with a fixed parameter of 1 is used to calculate the attention weights of each region on the small-sized attention map. Then, Batch Normalization is used to normalize the attention weights of each region to obtain the weight matrix W. Then, a scaling matrix is ​​initialized. The scaling matrix has the same size as the weight matrix W but the initial value is all 1. The top-k values ​​and positions in the weight matrix are determined and recorded in list L. k is a hyperparameter that controls the number of branches. Finally, k iterations are performed to obtain k feature map branches that focus on different main bodies: In the first iteration, the maximum value filtering, deconvolution and upsampling operations are skipped and the input feature map of the branch module is directly fed into the self-attention to obtain the feature map about the main body O1. The specific operations of the other nth iteration are as follows: ① Find the maximum values ​​Q1 and Q2 in the current weight matrix; ② Initialize the scaling matrix, then modify its values, changing the value at position Q1 to Q2 / Q1, and the value at position Q2 to Q1 / Q2, and multiply the scaling matrix element-wise with the weight matrix to obtain a new weight matrix; ③ Restore the modified scaling matrix to the same size as the input feature map by performing a 5*5 fixed-parameter deconvolution and a 7*7 upsampling; ④ Multiply the input feature map and the obtained scaling matrix element-wise; ⑤ Feed the results into self-attention to obtain information about the subject O. n Feature map;

[0032] 5-5) Constructing the branch detail enhancement module involves feeding the obtained main feature maps into dilated convolutional layers Dil_Conv with kernels of 3*3, 5*5, and 7*7 respectively, and then stitching the obtained image features together. The ability of dilated convolution to capture contextual information is used to enhance the model's ability to distinguish the main subject.

[0033] 6) For each attention feature map, a semantic aggregator is used in conjunction with the word pool to obtain text features with high matching degree with the feature map. Specifically, the semantic aggregator module rematches the word pool retrieved from the original image and the feature maps of k branches with the word pool obtained by the semantic filter, and uses an attention mechanism to aggregate the feature words related to branch i to obtain the word vector D for branch i. i The specific operation is as follows: the feature maps of the input k branches are encoded using CLIP-I, then mapped to the same dimension as the word pool using a linear layer. The feature maps are then used as the query and key for multi-head cross-attention, and the word vectors from the word pool are used as the value to obtain the word vector D belonging to branch i. i ;

[0034] 7) Decode the obtained text and image features to generate descriptive sentences: Construct a joint sentence decoding module to generate text descriptions corresponding to the images. This module receives the k feature maps obtained in step 5-4), the k word vectors obtained in step 6), and the proportion matrix L obtained in step 5-4), and outputs a complete descriptive sentence, including:

[0035] 7-1) First, assign weights to the corresponding feature maps and word vectors according to list L. In step 5), the weight matrix is ​​used to adjust the main body in each feature map to the same attention weight as the region with the largest weight in the original attention map. Therefore, when performing fusion, the original weights recorded in list L are used to restore the weight of each feature map in the original image.

[0036] 7-2) Concatenate the k feature maps after weighting and concatenate the k word vectors after weighting;

[0037] 7-3) Masked Multi-Head Attention is used to generate the descriptive words one by one, and the output of the next time step is predicted by masking the information of the next time step, thereby generating a complete description.

[0038] This technical solution adopts a branch prediction method for low-light image description. By using an attention mechanism to establish different branches for different subjects, the influence of missing features in low-light images on the subjects is reduced, and the model's ability to understand different subjects in low-light images is improved. Then, the outputs of different branches are fused according to the weights of different subjects in the image, and then sentence decoding is performed to generate the corresponding description. In this way, it can show good performance in low-light image description tasks.

[0039] This method combines an auxiliary enhancement module and a branch attention module to separate different subjects in low-light images, avoiding missed detections. Then, a branch detail enhancement module enhances the features of different branches to avoid false detections. Finally, the different subjects are fused to generate descriptive sentences. This avoids the domain shift problem caused by enhancing the original image, thereby reducing the impact of feature loss in low-light images on the image description model and improving the performance of the low-light image description model. Attached Figure Description

[0040] Figure 1 This is a structural diagram of the branch prediction image description model in the embodiment;

[0041] Figure 2 This is a structural diagram of the semantic filter module in the embodiment;

[0042] Figure 3 This is a structural diagram of the branch attention module in the embodiment;

[0043] Figure 4 This is a structural diagram of the branch module in the embodiment;

[0044] Figure 5 This is a structural diagram of the semantic aggregator module in the embodiment;

[0045] Figure 6 This is a structural diagram of the joint sentence decoder module in the embodiment;

[0046] Figure 7 This is a structural diagram of the auxiliary enhancement module in the embodiment;

[0047] Figure 8 This is a structural diagram of the branch detail enhancement module in the embodiment. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0049] Example:

[0050] Reference Figure 1 A low-light image description method based on branch prediction includes the following steps:

[0051] 1) Processing Image Description Datasets: Processing image description datasets reduces image brightness to obtain low-light image datasets, including:

[0052] 1-1) The publicly available Visual Genome dataset is used. The Visual Genome dataset not only provides annotation information for descriptive sentences in image description tasks, but also provides annotation information for the Region Graph and Scene Graph of the image. Different visual elements and relationships in the image are annotated. Training the model with the Visual Genome dataset can achieve better performance in capturing visual elements and their relationships.

[0053] 1-2) Apply ForkGAN to the images to perform day-night transformation, reduce the brightness of the images in the dataset, and obtain the low-light dataset Visual Genome-Night;

[0054] 1-3) The Visual Genome-Night dataset is divided into training and validation sets. The dataset contains 108,077 images. The training and validation sets are divided in an 8:2 ratio. 80% of the dataset is the training set, which contains 86,462 images; 20% of the dataset is the validation set, which contains 21,615 images.

[0055] 2) Establish an image descriptive lexicon: Extract and filter the descriptive text from the dataset to obtain an image descriptive lexicon, including:

[0056] 2-1) Extract the descriptive text of all images in the Visual Genome dataset, merge all text into a large text file, and split the large text file into words using spaces, commas, and periods, and calculate the frequency of word occurrences.

[0057] 2-2) Filter words by removing words that appear less than 3 times, and remove distractors such as punctuation marks, numbers, and irrelevant stopwords.

[0058] 2-3) Normalize the vocabulary by converting all words in the vocabulary to lowercase to reduce the impact of word variants on model training, avoid misclassifying words with different capitalization forms as different words, and map all variants of related words to a stem.

[0059] 2-4) Integrate the remaining words into one file to form the final vocabulary;

[0060] 3) Constructing an image encoder and a text encoder to encode text and images: CLIP is used to encode image and text data, including:

[0061] 3-1) Preprocess the training images by scaling them to 512*512 pixels to ensure that the size of the input data is consistent with the expected input size of the model and to avoid size mismatch errors. At the same time, important details in the small images are preserved. Preprocess the text information by combining related words in the vocabulary into r short sentence templates.

[0062] 3-2) The CLIP-I image encoder and CLIP-T text encoder are constructed using the CLIP-ViT-B / 32 model, and the pre-trained weights of CLIP are loaded.

[0063] 3-3) Encoding the input image and text: The CLIP-I image encoder receives the image input and outputs the encoded image features V. c The CLIP-T text encoder receives text input and outputs encoded text features T. c ;

[0064] 4) such as Figure 2 As shown, a semantic filter module is constructed: the feature T encoded in step 3) is processed... c and V c A semantic filter is used to obtain a word pool for the image, including:

[0065] 4-1) For the input image features V c and text features T c A fully connected linear and ReLU activation layer with 512 input and output channels is used to perform a learnable mapping, giving the input features stronger representational power, while simultaneously transforming the image features V... c and text features T c Mapped to the same dimension, in this example, the input and output dimensions of the linear layer for the image feature part are 512, and the input and output dimensions of the linear layer for the text feature part are r.

[0066] 4-2) Calculate the image features V obtained after passing through a fully connected (linear) layer and a ReLU activation layer. c and text features T c The cosine similarity between them is used to obtain the corresponding cosine similarity matrix M. sim ;

[0067] 4-3) For the cosine similarity matrix M sim A softmax layer is used to map the image to a probability vector P with respect to the vocabulary. t Then, based on the set threshold T=0.2, the word pool elements with a probability higher than the threshold T are selected, and the elements in the image and their relationships are mapped to the corresponding elements in the word pool, i.e., the word pool corresponding to the image.

[0068] 5) such as Figure 3 As shown, an enhancement module and a branch attention module are constructed: Image enhancement and branch prediction are performed to obtain a set of attention feature maps optimized for different subjects, which are used to capture different elements and their relationships in the image, specifically:

[0069] 5-1) For example Figure 7 As shown, the auxiliary enhancement module first performs histogram equalization on the image to initially adjust the overall brightness of the image. Then, it performs 2x upsampling and 2x downsampling operations to obtain two images equivalent to 4 times and 1 / 4 of the original image size. These images are combined with the original image to form an image pyramid. The images of different sizes in the image pyramid are processed by 3 layers of 3*3 convolution and ReLU pooling to extract features. Then, the extracted features are downsampled to 1 / 4 of the original image size and summed pixel by pixel to obtain the enhanced features, which are used as the input K of Masked Multi-Head Attention in the self-attention branch module.

[0070] 5-2) The input image first passes through three convolutional layers. Convolutional layers 1 and 2 do not change the size of the feature map and use residual connections to preserve image feature information. Convolutional layer 3 reduces the width and height of the feature map to half of its original size and uses max pooling to downsample the original image. The downsampled feature map is then added to the feature map obtained from convolutional layer 3. Convolutional layer 1 uses a 3x3 kernel, a padding of 1, 3 input channels, and 64 output channels. Convolutional layer 2 uses a kernel of... The first convolutional layer is a 3x3 convolution with a pad of 1 and 64 input and output channels. The second convolutional layer is a 2x2 convolution with a stride of 2, which reduces the width and height of the feature map to half of its original size. The third convolutional layer uses residual connections to add the original image to the outputs of the first and second convolutional layers pixel by pixel. The input image is downsampled using max pooling with a 2x2 kernel and a stride of 2. The downsampled feature map is then added pixel by pixel to the feature map obtained from the third convolutional layer.

[0071] 5-3) Calculate the attention weight map of the preliminary feature map obtained in step 5-2) using a multi-head self-attention module;

[0072] 5-4) For example Figure 4As shown, the branching module performs subject separation on the attention map obtained in step 5-3): First, the feature map is downsampled by 7*7, i.e., the feature map is reduced in dimensionality to obtain a small-sized attention map. Then, a 5*5 convolutional kernel with a fixed parameter of 1 is used to calculate the attention weights of each region on the small-sized attention map. Then, BatchNormalization is used to normalize the attention weights of each region to obtain the weight matrix W. Then, a scaling matrix is ​​initialized. The scaling matrix has the same size as the weight matrix W but the initial value is all 1. The top-k values ​​and positions in the weight matrix are determined and recorded in list L. k is a hyperparameter that controls the number of branches. Finally, k iterations are performed to obtain k feature map branches that focus on different subjects: the first iteration is skipped. The maximum value filtering, deconvolution, and upsampling operations directly feed the input feature map of the branch module into the self-attention to obtain the feature map about the subject O1. The specific operations of the other nth iteration are: ① Find the maximum values ​​Q1 and Q2 in the current weight matrix; ② Initialize the scaling matrix, then modify the values ​​of the scaling matrix, changing the value at position Q1 to Q2 / Q1, and the value at position Q2 to Q1 / Q2, and multiply the scaling matrix element-wise with the weight matrix to obtain a new weight matrix; ③ Restore the modified scaling matrix to the same size as the input feature map by using a 5*5 fixed-parameter deconvolution and 7*7 upsampling; ④ Multiply the input feature map and the obtained scaling matrix element-wise; ⑤ Feed it into the self-attention to obtain the feature map about the subject O1. n Feature map;

[0073] 5-7) such as Figure 8 As shown, the branch detail enhancement module feeds the obtained main feature maps into dilated convolutional layers Dil_Conv with kernels of 3*3, 5*5, and 7*7 respectively. Then, the obtained image features are stitched together. The ability of dilated convolution to capture contextual information is used to enhance the model's ability to distinguish the main subject.

[0074] 6) For example Figure 5 As shown, for each attention feature map, a semantic aggregator is used in conjunction with the word pool to obtain text features with high matching degree with the feature map. Specifically, the semantic aggregator module rematches the word pool and feature maps of k branches retrieved from the original image with the word pool obtained by the semantic filter, and uses an attention mechanism to aggregate the feature words related to branch i to obtain the word vector D about branch i. iThe specific operation is as follows: the feature maps of the input k branches are encoded using CLIP-I, then mapped to the same dimension as the word pool using a linear layer. The feature maps are then used as the query and key for multi-head cross-attention, and the word vectors from the word pool are used as the value to obtain the word vector D belonging to branch i. i ;

[0075] 7) Decode the obtained text and image features to generate descriptive sentences: Construct a joint sentence decoding module to generate text descriptions corresponding to the images. This module receives the k feature maps obtained in step 5-4), the k word vectors obtained in step 6), and the proportion matrix L obtained in step 5-4), and outputs a complete descriptive sentence, including:

[0076] 7-1) As Figure 6 As shown, firstly, weights are assigned to the corresponding feature maps and word vectors according to list L. In step 5), a weight matrix is ​​used to adjust the main body in each feature map to the same attention weight as the region with the largest weight in the original attention map. Therefore, when performing fusion, the original weights recorded in list L are used to restore the weight of each feature map in the original image.

[0077] 7-2) Concatenate the k feature maps after weighting and concatenate the k word vectors after weighting;

[0078] 7-3) Masked Multi-Head Attention is used to generate the descriptive words one by one, and the output of the next time step is predicted by masking the information of the next time step, thereby generating a complete description.

Claims

1. A low-illumination image description method based on branch prediction, characterized in that, Comprising the following steps: 1) processing image description data set: processing image description data set, reducing the brightness of image to obtain low-illumination image data set, comprising: 1-1) using the public Visual Genome data set, the Visual Genome data set not only provides the annotation information of the description sentence in the image description task, but also provides the Region Graph and Scene Graph annotation information of the image, which labels different visual elements and relationships in the image; 1-2) using ForkGAN to convert day and night for the image, reducing the brightness of the image in the data set, obtaining low-illumination data set Visual Genome-Night; 1-3) dividing the training set and the validation set for the VisualGenome-Night data set, the data set has a total of 108077 images, and the training set and the validation set are divided according to the ratio of 8:2, 80% of the data set is the training set, a total of 86462 images; 20% of the data set is the validation set, a total of 21615 images; 2) establishing an image description vocabulary: extracting the description text in the data set and obtaining an image description vocabulary through screening, comprising: 2-1) extracting the description text of all images in the VisualGenome data set, merging all the texts into a large text file, and dividing the large text file into words according to the space, comma and period symbols, calculating the frequency of word occurrence; 2-2) screening the words, including removing words with a frequency lower than 3 according to the frequency of occurrence, and removing interference items including punctuation marks, numbers and irrelevant words stopwords; 2-3) normalizing the vocabulary, including converting all words in the vocabulary to lowercase, and mapping related word variants to a stem; 2-4) integrating the remaining words into a file to form the final vocabulary; 3) constructing image encoder and text encoder to encode text and image: using CLIP to encode image and text data, comprising: 3-1) preprocessing the training image, including uniformly scaling the size of the training image to 512*512 pixels to ensure that the size of the input data is consistent with the expected input size of the model, while preserving important detail information in small size images, and preprocessing the text information is to combine related words in the vocabulary into r short sentence templates; 3-2) using CLIP-ViT-B / 32 model to construct CLIP-I image encoder and CLIP-T text encoder, and loading the pre-training weight of CLIP; 3-3) Encoding the input image and text includes a CLIP-I image encoder receiving an image input and outputting an encoded image feature V c , and a CLIP-T text encoder receiving a text input and outputting an encoded text feature T c ; 4) Constructing the semantic filter module: encode the features V obtained in step 3-3) to obtain semantic filter c and T c Obtaining the word pool of the image using the semantic filter, comprising: 4-1) to the input image feature V c and the text feature T c A fully connected layer linear with 512 input and output channels is used to perform a learnable mapping, while mapping the image feature V c and the text feature T c to the same dimension; 4-2) Calculate the cosine similarity between the image features V obtained by passing through the full connection layer linear and the Relu activation layer, and the text features T, to obtain the corresponding cosine similarity matrix M c ; and the text features T c ; Calculate the cosine similarity between the image features V obtained by passing through the full connection layer linear and the Relu activation layer, and the text features T, to obtain the corresponding cosine similarity matrix M sim ; 4-3) Cosine similarity matrix M sim The softmax layer is used to map the image to a probability vector P t Then, according to the set threshold T=0.2, the elements in the word pool with a probability higher than the threshold T are screened out, and the corresponding elements in the word pool to which the elements and their relationships in the image are mapped are obtained as the word pool corresponding to the image. 5) constructing enhancement module and branch attention module: enhancing and branch predicting the image to obtain a group of attention feature maps optimized for different subjects, which are used to capture different elements and their relationships in the image, specifically: 5-1) Constructing an auxiliary enhancement module First, the image is histogram equalized to preliminarily adjust the overall brightness of the image, and then 2 times up-sampling and 2 times down-sampling operations are adopted to obtain two images equivalent to 4 times and 1 / 4 of the original image size, which form an image pyramid with the original image. The images of different sizes in the image pyramid are subjected to feature extraction through 3 layers of 3*3 convolution and RELU pooling operation, and then the extracted features are down-sampled to 1 / 4 of the original image size and subjected to pixel-by-pixel addition operation to obtain enhanced features as the input K of Masked Multi-Head Attention in the self-attention branch module; 5-2) The input image is first subjected to three convolutional layers. Convolutional layer 1 and convolutional layer 2 do not change the size of the feature map, while residual connection is adopted to retain image feature information. Convolutional layer 3 reduces the length and width of the feature map to 1 / 2 of the original size, while maximum pooling operation is adopted to down-sample the original image. The down-sampled feature map is added to the feature map obtained by convolutional layer 3, wherein convolutional layer 1 is a convolution operation with a kernel of 3*3, a pad of 1, an input channel of 3, and an output channel of 64; convolutional layer 2 is a convolution operation with a kernel of 3*3, a pad of 1, and an input channel and an output channel of 64; convolutional layer 3 is a convolution operation with a kernel of 2*2 and a step of 2, which reduces the length and width of the feature map to 1 / 2 of the original size; the original image is added to the outputs of convolutional layer 1 and convolutional layer 2 through residual connection in a pixel-by-pixel manner. The input image is down-sampled by maximum pooling operation with a pooling kernel of 2*2 and a step of 2, and the down-sampled feature map is added to the feature map obtained by convolutional layer 3 in a pixel-by-pixel manner; 5-3) The attention weight map is calculated for the preliminary feature map obtained in step 5-2) through the multi-head self-attention module; 5-4) Constructing the branch module to separate the attention map obtained in step 5-3) into subjects: first, downsize the feature map by 7*7 to reduce the dimension of the feature map to obtain an attention map of small size, then calculate the attention weight of each region using a fixed parameter convolution kernel of 5*5 with a value of 1 on the small size attention map, and then normalize the attention weight of each region using Batch Normalization to obtain the weight matrix W; then initialize a scale matrix, the size of the scale matrix is the same as the size of the weight matrix W but the value is initially all 1, and sequentially determine the top-k value and position in the weight matrix and record them in the list L, k is a hyperparameter that controls the number of branches; finally, k cycles are performed to obtain k feature map branches that focus on different subjects: the first cycle skips the maximum value screening, deconvolution and upsampling operations and directly inputs the input feature map of the branch module into the self-attention to obtain the feature map about subject O1, and the specific operation of the nth cycle is: ① find the maximum values Q1 and Q2 in the current weight matrix; ② initialize the scale matrix, then modify the values of the scale matrix, modify the value at the position of Q1 to Q2 / Q1, modify the value at the position of Q2 to Q1 / Q2, and multiply the scale matrix and the weight matrix element by element to obtain a new weight matrix; ③ restore the modified scale matrix to the same size as the input feature map through 5*5 fixed parameter deconvolution and 7*7 upsampling; ④ multiply the input feature map and the obtained scale matrix element by element; ⑤ input into the self-attention to obtain the feature map about subject O n ​ 5-5) The branch detail enhancement module is constructed by feeding the obtained main feature map into the dilated convolution layer Dil_Conv with a convolution kernel of 3*3, 5*5, and 7*7, and then splicing the obtained image features; 6) The semantic aggregator is used to combine each attention feature map with the word pool to obtain text features with high matching degree with the feature map, specifically: the semantic aggregator module re-matches the word pool retrieved from the original image with the k-branch feature map with the word pool obtained by using the semantic filter, and uses the attention mechanism to aggregate the feature words related to branch i to obtain the word vector D of branch i i , Specifically: encode the input k-branch feature map through CLIP-I, then map it to the same dimension as the word pool using a linear layer, and then use the feature map as the query and key of the multi-head cross-attention Mutil Headcross-attention, and use the word vector of the word pool as the value to obtain the word vector D belonging to branch i i ; 7) Decoding the obtained text features and image features to generate a description sentence: a joint sentence decoding module is constructed to generate a text description corresponding to the image. The module receives the k feature maps obtained in step 5-4), the k word vectors obtained in step 6), and the proportion matrix obtained in step 5-4), and outputs a complete description sentence, including: 7-1) First, assign weights to the corresponding feature maps and word vectors according to the list L. In step 5), the weight matrix is used to adjust the main body in each feature map to the same attention weight as the maximum weight region in the original attention map. Therefore, when fusing, the original weight recorded by the list L is used to restore the weight size of the feature map in the original image; 7-2) Splice the k feature maps after assigning weights, and splice the k word vectors after assigning weights; 7-3) Generating words of the description one by one using Masked Multi-Head Attention, predicting the output of the next time step using the information masked from the next time step, thereby generating the complete description.

Images