Image dense description method based on image saliency and deep attention network

Abstract

The application provides an image dense description method based on image saliency and a deep attention network, and belongs to the cross technical application field of computer vision and natural language processing.S1, a focus area and a focus area mask of an image are obtained based on image saliency analysis;S2, the image is encoded to obtain an image feature vector;S3, the image feature vector and the focus area mask are input into a decoder to generate a description sentence.In training, the model uses a rectangular area used in traditional dense description as a mask, calculates a cross entropy loss of an inference result in cooperation with a description corpus, updates parameters by using a gradient descent method, finds an optimal model, and enables the optimal model to understand and describe a specific area.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing technology, and particularly relates to a method for dense image description based on image saliency and deep attention networks. Background Technology

[0002] Image understanding is an innate human ability to perceive the world. With the development of cutting-edge artificial intelligence technologies, more and more scholars hope that machines can possess such image understanding capabilities. Image description can intuitively reflect a machine's image understanding level, and it has wide-ranging potential applications in image retrieval, social media, e-commerce, and other fields. Therefore, research on image description is of great significance. Dense image description is similar to existing image description methods, both using an image as input. While image description generates a summary description of the image, dense description breaks down the image and provides phrase descriptions for each part.

[0003] Research on dense image description tasks is still insufficient and largely in the exploratory stage. However, for general image description, there is already a wealth of research, with a basic model framework involving pre-trained convolutional neural networks (such as ResNet) to extract image feature vectors, followed by recurrent neural networks (such as LSTM) to generate descriptive content word-by-word based on these feature vectors. Previous dense image description methods, building upon the encoding and decoding structures of single-sentence descriptions, all started with image target detection, using rectangular detection boxes to obtain candidate description regions. This approach may lose some information about the relationship between the target and the background or between the target itself, and may also result in excessive repetition and complexity in the dense description. For example, the dense description method proposed by Johnson J et al. in 2016, as shown in the references below, while providing a relatively comprehensive description, suffers from significant information redundancy.

[0004] References: Johnson, J., Karpathy, A., & Fei-Fei, L. (2016). Densecap: Fullyconvolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp.4565-4574). Summary of the Invention

[0005] To overcome the aforementioned problems, this invention proposes a dense image description method based on image saliency and deep attention networks. It utilizes image saliency analysis to obtain image regions of interest with arbitrary shapes; employs a pre-trained convolutional neural network to acquire complete image feature vectors; and leverages hard attention to limit the image regions focused on by soft attention during feature decoding using the obtained masks, enabling the decoder to effectively obtain local information of the corresponding regions. This approach blurs the concept of specific instance objects, allowing the description model to focus more on the meaning of the image itself rather than the features of objects. This invention combines the intensity of human responses to image content to obtain important regions, using an attention-based encoder-decoder (using ResNet-101 for the encoder and an attention-based LSTM for the decoder) to encode the image and decode based on the regions of interest, thereby obtaining sufficient yet concise descriptive content.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A dense image description method based on image saliency and deep attention networks includes the following steps:

[0008] S1. Obtain the region of interest and its mask based on image saliency analysis;

[0009] S2. Encode the image to obtain the image feature vector;

[0010] S3. Input the image feature vector and the region of interest mask into the decoder to generate a description statement.

[0011] Furthermore, step S1 includes:

[0012] S11. Obtain the saliency intensity map of the image through image saliency analysis;

[0013] S12. Binarize the saliency intensity map according to different preset thresholds to obtain the region of interest in the image;

[0014] S13. Generate a mask for the region of interest;

[0015] S14. The significance intensity is calculated from weak to strong. The cross-union ratio (CUI) of the region of interest mask is calculated. If the CUI is greater than a certain threshold, the region of interest with strong significance intensity is discarded. Otherwise, the region of interest with strong significance intensity and its mask are saved.

[0016] Preferably, the crossover ratio threshold is 0.2.

[0017] Furthermore, in S12, the connected regions in the region of interest of the obtained image are segmented by morphological dilation and erosion to form the region of interest.

[0018] Furthermore, in step S2, encoding is performed using ResNet.

[0019] Furthermore, in step S3, the decoder predicts the probability distribution of the current word, while simultaneously introducing soft attention and hard attention into the decoder to restrict the image region that the decoder can see, and then gradually generates descriptive words, specifically including:

[0020] S31. Set weights for regions of interest and regions of non-interest;

[0021] S32. The latent variable H of the previous time step of the LSTM and the image feature vector are concatenated through a fully connected layer to form Soft Attention;

[0022] S33. The attention region mask with weights set in S31 is used as the input of global attention intensity, and the product of the soft attention in S32 is used as the attention weight of the image feature vector. The weighted image feature vector AWE is then input into LSTM.

[0023] Furthermore, in step S3, the decoder is LSTM.

[0024] Furthermore, in S31, the background weight of the attention region mask is adjusted to be greater than 0, and the attention region weight is 1.

[0025] Preferably, the background weight is 0.2.

[0026] Furthermore, training methods for the encoder and decoder include:

[0027] The description words before the current time step use the corpus content. The probability distribution of the description words is obtained through steps S2 and S3. If the top 5 words in the distribution contain the target words in the corpus, then the inference of the word is considered to be correct.

[0028] The original corpus and the inferred descriptive statements are compared, and cross-entropy is calculated as the loss function. The encoder and decoder parameters are updated using stochastic gradient descent. In the first round, only the decoder is updated, and in subsequent rounds, both the decoder and encoder are updated until convergence.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] (1) Image saliency intuitively reflects the focus and order of people when reading images, blurs the concept of instance objects, strengthens local information of images, and avoids the problem of existing technologies ignoring the relationship between objects and environment as well as between objects.

[0031] (2) Reasonably select the areas of interest, retain effective content while avoiding information redundancy, and effectively summarize and describe the content of similar objects nearby.

[0032] (3) The present invention can effectively describe irregular areas, breaking the limitation of existing technologies that can only describe the content of rectangular areas. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating an image density description method according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the process for processing saliency images at a certain threshold according to an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the specific structure of a decoder according to an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram showing the result of one embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0039] The overall idea of ​​this invention is as follows: First, obtain the saliency intensity map of the image, and obtain masks of the region of interest under different attention intensities based on different thresholds. Second, use ResNet-101 to obtain feature vectors in the image. Finally, based on the obtained feature vectors, under the constraint of the region of interest mask, an LSTM network combined with an attention mechanism decodes and gradually generates descriptive words. Since there is no labeled data for irregular regions, we use traditional rectangular boxes to generate the region of interest mask, calculate the cross-entropy loss by comparing it with the labeled corpus, and use the gradient descent method to update the parameters of the encoder and decoder to find the optimal model that can describe the specific region represented by the mask.

[0040] The proposed image dense description method based on image saliency and deep attention networks in this invention, such as... Figure 1 As shown, it includes:

[0041] S1. Obtain the region of interest and its mask based on image saliency analysis;

[0042] S2. Encode the image to obtain the image feature vector;

[0043] S3. Input the image feature vector and the region of interest mask into the decoder to generate a description statement.

[0044] Step S1 includes:

[0045] S11. Obtain the saliency intensity map of the image through image saliency analysis. There are various methods for obtaining the saliency intensity map, such as the deep learning approach of EML-NET, and more traditional methods such as the ITTI algorithm, residual spectrum algorithm (SR), FT algorithm, HC algorithm, etc. This invention chooses to obtain it through the ITTI visual saliency model (the ITTI method mainly uses Gaussian sampling to construct Gaussian pyramids for the image's color, brightness, and orientation, then uses the Gaussian pyramids to calculate the image's brightness feature map, color feature map, and orientation feature map, and finally combines the feature maps at different scales to obtain the brightness, color, and orientation saliency maps, which are then added together to obtain the final visual saliency map). The saliency intensity map is denoted as SAL.

[0046] S12. The saliency intensity map is binarized according to different preset thresholds to obtain the region of interest in the image. Preferably, the connected regions are segmented by morphological dilation and erosion to form the region of interest. Figure 2 The image shows the result of binarizing and dilating / eroding a saliency image at a certain threshold. For each threshold, a set of regions of interest can be generated.

[0047] S13. Generate a mask for the region of interest by setting the pixel value of the region of interest to 1 and the pixel value of other regions to 0.

[0048] S14. Calculate the Cross-Union Ratio (CUI) for the masks of regions of interest, from weakest to strongest salience. Specifically, for each threshold-corresponding region mask, calculate its CUI with the saved masks. If the CUI is greater than a certain threshold, it indicates excessive region overlap, and the region of interest is discarded; otherwise, its mask is saved. The CUI threshold is adjustable. Setting it too high will result in too many masks being retained, increasing redundancy; conversely, setting it too low will lead to a large number of valid regions of interest being discarded, resulting in severe information loss. A threshold of 0.2 is preferred.

[0049] In one embodiment, SAL grayscale values ​​are uniformly mapped to the range [0,1] for normalization.

[0050] In one embodiment, a set of thresholds T = {T1, T2, ..., Tn | Ti = (i – 1) / n} is set, for example, n is set to 16. Figure 2As shown, the saliency intensity map of the image is binarized based on this set of thresholds. Then, morphological erosion is used to separate the contiguous regions, and morphological dilation is used to restore the shrunken regions. Each connected region obtained in this way is considered a valid region of interest. For example, T1 = 0 is a global value; under this set of thresholds, it indicates that the entire image is of interest, mainly describing the scene of the entire image. To avoid redundant information, details are not described when the threshold is small, such as... Figure 1 The example of the mask and description content at the bottom center shows that when the threshold is 0, focusing on the entire image, it is described as "photo was taken in the daytime". As the threshold increases, the area retained in the region of interest decreases, but at the same time, it has higher visual saliency, and the information retained becomes more specific, resulting in the description "woman playing tennis". When the threshold is Tn, the small region with the highest saliency is obtained, representing the image detail, "a yellow tennis ball". To reduce information redundancy and improve the effectiveness of the description, regions with an intersection-union ratio (IU) greater than a certain IU threshold are removed from the obtained effective regions of interest. Then, a mask is generated for the finally retained regions of interest, which is stored as a mask in the form of a binary image (0 represents the background region, 1 represents the region of interest) and used in subsequent steps.

[0051] By leveraging image saliency, this invention intuitively reflects the focus and order of a person's attention when reading an image, blurring the concept of instance objects, strengthening local image information, and avoiding the problem of existing technologies neglecting the relationship between objects and their environment, as well as between objects themselves. Following this method, the present invention can generate irregular regions of interest (while existing technologies generate rectangular regions of interest), and because it uses ITTI visual saliency to assist in region division, it is more closely related to the underlying visual characteristics of humans, representing a novel and interpretable division method. While retaining effective content, it avoids information redundancy and effectively groups adjacent, similar object content into the same descriptive region for description.

[0052] In step S2, this invention does not extract specific image object instances, but directly uses image features for subsequent calculations. ResNet-101 can be used as the encoder, removing the classification head to obtain the image feature vector. To ensure a full grasp of local image information, the dimension of the output vector can be increased. For example, in image tasks, ResNet is typically used to obtain a 14x14x2048-dimensional feature vector, which is similar to dividing the image into 14x14 regions. Increasing the number of regions to 32x32 effectively preserves more detailed information from smaller regions. Similarly, encoders such as VGG and Inception V4 can also be used.

[0053] In step 3, the decoder predicts the probability distribution of the current word and introduces both soft and hard attention into the decoder to limit the image area that the decoder unit can see, and then gradually generates descriptive words. In each loop, the decoder generates a probability distribution of descriptive words based on the corresponding key area input by the mask, and passes the latent variables C and H to the next time step.

[0054] The decoder can use RNN, LSTM, Transformer, etc. In one embodiment, LSTM is chosen to implement decoding, such as... Figure 3 The diagram shows the decoder structure and operation process after the introduction of attention, including:

[0055] S31. Set weights for regions of interest and regions of non-interest. Input the final retained region of interest mask from step 1 into an attention-enabled LSTM network. The mask itself is a binary image. When inputting into the LSTM, adjust the background weight from 0 to a number greater than 0, such as 0.2 (this value is adjustable; too small a value will ignore the influence of background information, and too large a value will lose focus and the focus of the description will be lost). The weight of the region of interest remains at 1.

[0056] S32. Introduce Soft Attention, which involves applying the latent variable H from the previous time step of the LSTM to the image feature vector Z. f After passing through a fully connected layer, the data are concatenated to form a Soft Attention layer, which associates descriptive words with image feature vectors.

[0057] S33. Introduce Hard Attention, which means using the attention region mask with weights set in S31 as the input to the global attention intensity, and multiplying it with the Soft Attention described in S32 to generate the LSTM attention weights for the entire image feature vector, which then act together on the image feature vector Z. f The weighted image feature vector AWE is generated and then input into LSTM.

[0058] Unlike traditional methods, the input to LSTM is a weighted image feature vector AWE (Attention-weighted-encoding):

[0059] ATT = relu(FC1(Z) f )+FC2(h t ))

[0060] α = softmax(FC3(ATT))

[0061] AWE = Z f *α*mask

[0062] Z f It is an image feature vector, h tIt is an LSTM hidden variable, ATT encoder and AT&T decoder These are fully connected layers mapped to the attention vector space, designed according to the feature vector dimension and the LSTM latent variable dimension, respectively. `mask` is the attention region mask obtained in step S1, and `AWE` is the weighted image feature vector, which is the feature vector content referenced by the LSTM at the current time step. The LSTM output vector is the probability distribution of the predicted words at the current position. The probability distribution obtained during training is used to calculate the loss function, and the probability distribution with the highest probability distribution during inference is the inference result. These are then concatenated to obtain the descriptive sentence.

[0063] The attention-based decoder input region of interest mask designed in this invention can adapt to regions of interest of any shape, thus effectively describing irregular regions and breaking the limitation of existing technologies that can only describe rectangular regions.

[0064] This invention also includes training the encoder and decoder. During training, the descriptive words before the current time step are used from the corpus content. The probability distribution of the descriptive words is obtained through steps S2 and S3. If the top 5 words in the distribution contain the target vocabulary from the corpus, the inference for that word is considered correct. Following this method, the original corpus and the inferred descriptive sentences are compared, and cross-entropy is calculated as the loss function. Stochastic gradient descent is used to update the model parameters (only the decoder is updated in the first round, and the encoder is also updated in subsequent rounds for fine-tuning) until convergence. In one embodiment, the specific training method includes:

[0065] An end-to-end training model is adopted, using a pre-trained ResNet-101 with the classification head removed as the encoder. To facilitate convergence, the decoder (LSTM+Attention) parameters are initialized with a uniform distribution U(-0.1, 0.1), the fully connected biases of the word vectors are initialized to 0, and the node parameters in the LSTM are ignored with a 50% probability to improve the model's generalization ability and prevent overfitting. The descriptive words are obtained through step S3. If the top 5 probability distributions of the predicted descriptive words contain the target vocabulary from the corpus, the inference is considered correct. Based on the above method, the original corpus and the inferred descriptive sentences are compared, the cross-entropy is calculated as the loss function, and the stochastic gradient descent method is used to update the model parameters until convergence. In the first epoch, only the decoder is trained with a learning rate of 4e-4 and a batch size of 256, giving the decoder initial descriptive capabilities. After the first epoch, the encoder is fine-tuned with a learning rate of 1e-5, while the decoder's learning rate is reduced to 4e-5, and the batch size is reduced to 128 (larger batch sizes allow for faster training, more balanced samples, and enable the encoder and decoder to quickly acquire basic functionality; smaller batch sizes subsequently help improve network model performance). If there is no improvement after four consecutive epochs, the learning rate is reduced to 0.8. Each batch includes an image resized to 256x256 and a mask corresponding to the target region bounding box (resized to 32x32 to fit the model). Training is performed using an NVIDIA RTX A6000, and starting from the second epoch, the average training time per batch is 0.938 seconds.

[0066] Train on the Visual Genome dataset for 20 epochs. If the loss of the codec on the validation set does not decrease for 8 consecutive epochs, stop training and save the model.

[0067] like Figure 4 The following is an example of the application of this invention. Step S1 involves inputting the image to be described into the ITTI model and performing binarization, dilation, and erosion operations based on different thresholds. A series of associated region masks are obtained through intersection-union-comparison (IUCN) filtering. For ease of understanding, the mask regions are drawn with different colored boxes in the image. Then, steps S2 and S3 are executed, generating the original image feature vector. Each time, one mask and the original image feature vector are input together into an attention-based encoder-decoder. During decoding, the descriptive word with the highest probability distribution is output at each time step until the sentence-end marker is decoded, resulting in a complete descriptive sentence. The dense description result of the entire image is shown below. Figure 4 The corresponding color of the attention area box yields the corresponding color description:

[0068] Area 1. Photo taken during the day.

[0069] Area 2, A window on a building.

[0070] Area 3. Cars parked on the street.

[0071] Area 4: Motorcycles on the road.

[0072] Area 5. The roof is white.

[0073] Area 6: The sky is blue.

[0074] Area 7, Red and white awning.

[0075] Area 8. A window on a building.

[0076] Area 9: The sky is blue.

[0077] Area 10, Window on the building.

[0078] Area 11: Man riding a bike.

[0079] Area 12, Silver car parked on street.

[0080] Area 13: The car is black.

[0081] Area 14, A black metal fence.

[0082] This invention uses underlying human visual factors for reasonable segmentation, while also breaking the limitation of traditional methods that can only handle rectangular region descriptions. Furthermore, it provides rich descriptive content with minimal information redundancy.

[0083] Those skilled in the art will understand that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for dense image description based on image saliency and deep attention networks, characterized in that, Includes the following steps: S1. Obtain the region of interest and its mask based on image saliency analysis; S2. Encode the image to obtain the image feature vector; S3. Input the image feature vector and the region of interest mask into the decoder to generate a description statement; Step S1 includes: S11. Obtain the saliency intensity map of the image through image saliency analysis; S12. Binarize the saliency intensity map according to different preset thresholds to obtain the region of interest in the image; S13. Generate a mask for the region of interest; S14. From weak to strong saliency, calculate the cross-union ratio (CURRR) for the mask of the region of interest. If the CURRR is greater than a certain threshold, discard the region of interest with strong saliency; otherwise, save the region of interest with strong saliency and its mask. In step S3, the decoder predicts the probability distribution of the current word, and simultaneously introduces soft attention and hard attention into the decoder to limit the image area that the decoder can see, and then gradually generates descriptive words, specifically including: S31. Set weights for regions of interest and regions of non-interest. S32. The latent variable H of the previous time step of the LSTM and the image feature vector are concatenated through a fully connected layer to form Soft Attention; S33. The attention region mask with weights set in S31 is used as the input of global attention intensity, and the product of the soft attention in S32 is used as the attention weight of the image feature vector. The weighted image feature vector AWE is then input into LSTM.

2. The image dense description method according to claim 1, characterized in that, The crossover-union ratio threshold is 0.

2.

3. The image dense description method according to claim 1, characterized in that, In S12, the connected regions in the region of interest of the obtained image are segmented by morphological dilation and erosion to form the region of interest.

4. The image dense description method according to claim 1, characterized in that, In step S2, encoding is performed using ResNet.

5. The image dense description method according to claim 1, characterized in that, In step S3, the decoder is LSTM.

6. The image dense description method according to claim 1, characterized in that, In S31, the background weight of the attention region mask is adjusted to be greater than 0, and the attention region weight is 1.

7. The image dense description method according to claim 6, characterized in that, The background weight is 0.

2.

8. The image dense description method according to claim 1, characterized in that, Training methods for encoders and decoders include: The description words before the current time step use the corpus content. The probability distribution of the description words is obtained through steps S2 and S3. If the top 5 words in the distribution contain the target words in the corpus, then the inference of the word is considered to be correct. The original corpus and the inferred descriptive statements are compared, and cross-entropy is calculated as the loss function. The encoder and decoder parameters are updated using stochastic gradient descent. In the first round, only the decoder is updated, and in subsequent rounds, both the decoder and encoder are updated until convergence.

Images