Image description method for enhancing bidirectional image feature interaction

Abstract

The invention relates to an image description method for enhancing bidirectional image feature interaction, and belongs to the field of multi-modal image description. According to the method, a bidirectional interaction encoder is introduced, two image features are made to complement each other in a cross interaction mode, and self visual information is enriched. In the decoder, a cross alignment module is provided, the cross alignment module enables text features to interact according to a region feature-grid feature sequence and a grid feature-region feature sequence to obtain two new text features, and the similarity between the two text features is explored. In this way, the semantic gap problem between the regional features and the grid features is converted into the similarity problem between the two text features.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal image description, and in particular relates to an image description method for enhancing bidirectional image feature interaction. Background Art

[0002] In life, we have access to a lot of data, including images, text, audio, and so on. A single type of data may not be fully analyzed, so we need to use multiple types of data. For humans, processing multiple types of data at the same time is not difficult, but it is very time-consuming. Image description technology can process image and text data, and can assist humans in processing data quickly. This technology can help the visually impaired live a better life. For example, SeeingAI developed by Microsoft can describe the target image using only a mobile phone. Moreover, image description can also help retrieval tools to classify faster, and has been widely used in many social software such as Facebook. In addition, image description can also be applied to medical images and remote sensing technology. For example, in the medical field, image description can help doctors analyze more comprehensively and improve the accuracy of diagnosis. Just provide the patient's medical image to automatically generate relevant pathological descriptions.

[0003] The performance of image description models is related to the richness of image features. At present, in order to improve the richness of image features, regional features and grid features that combine image features are generally used. However, although many methods have emerged to fuse these two features, these methods either have a simple fusion method or use one image feature to enrich the other, and ignore the semantic gap between the two image features. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the prior art and provide an image description method that enhances the interaction of two-way image features. The method introduces a two-way interactive encoder, which uses a cross-interaction method to make the two image features complement each other and enrich their own visual information. In the decoder, a cross-alignment module is proposed, which makes the text features interact in the order of "regional feature-grid feature" and "grid feature-regional feature" to obtain two new text features, and explores the similarity between the two text features. This converts the semantic gap problem between the region feature and the grid feature into the similarity problem between the two text features.

[0005] To achieve the above-mentioned purpose, the technical solution of the present invention is: an image description method for enhancing bidirectional image feature interaction, introducing a bidirectional interactive encoder, using a cross-interaction method to make the two image features complement each other and enrich their own visual information; in the decoder, a cross-alignment module is proposed, so that the text features interact in the order of "region feature-grid feature" and "grid feature-region feature" to obtain two new text features, and explore the similarity between the two new text features.

[0006] In one embodiment of the present invention, the method is based on a Transformer model, the number of bidirectional interactive encoders is 3, and the number of decoders is 2.

[0007] In one embodiment of the present invention, the input of the bidirectional interactive encoder includes the region features and mesh features Where N R Indicates the number of regions, N G Indicates the number of grids; the regional features and grid features are first enhanced by the bidirectional interactive encoder and then passed to the decoder. The semantic gap between the regional features and the grid features is alleviated by the cross alignment module in the decoder.

[0008] In one embodiment of the present invention, the region features are extracted by VinVL, and the grid features are extracted by the visual encoder of CLIP.

[0009] In one embodiment of the present invention, the bidirectional interactive encoder is composed of three identical encoding blocks, each of which is composed of a self-attention module and a cross-attention module.

[0010] In one embodiment of the present invention, the bidirectional interactive encoder is specifically implemented as follows:

[0011] In the bidirectional interactive encoder, the grid features and the region features will pass through the self-attention module and the cross-attention module in turn. The self-attention module is used to strengthen the important information of the image features themselves, and the cross-attention module is used to make up for the shortcomings of the image features themselves. For the l-th encoding block:

[0012] G l+1 =LayerNorm(MSA(G l ,G l ,G l )+G l ) (1)

[0013] R l+1 =LayerNorm(MSA(R l ,R l ,R l )+R l ) (2)

[0014] Among them, G represents grid features, R represents regional features, MSA represents multi-head self-attention module, and LayerNorm represents layer normalization. After the self-attention module, the information between each other will be exchanged through the cross-attention module to make up for the shortcomings of regional features lacking global features and the lack of spatial information of grid features. Finally, in order to ensure the integrity of the original features as much as possible, the features before and after enhancement are defined as follows:

[0015] G' l+1 =LayerNorm(MCA(G l ,R l ,R l )+G l ) (3)

[0016] R' l+1 =LayerNorm(MCA(R l ,G l ,G l ),R l ) (4)

[0017]

[0018] Among them, G represents grid features, R represents regional features, MCA represents multi-head cross attention module, LayerNorm represents layer normalization, and FFN represents feedforward network module; after two-way interaction, grid features and regional features have initially acquired their missing information.

[0019] In one embodiment of the present invention, the cross alignment module consists of two parts: a cross part and an alignment part, which are specifically implemented as follows:

[0020] The crossover part realizes the purpose of text features interacting in the order of "regional features-grid features" and "grid features-regional features", and the alignment part is to improve the feature similarity; although the regional features and grid features come from the same image, they have different feature extraction methods and there is a semantic gap between the two features. The S g and S r will also be different; therefore, through the cross section, S is indirectly solved g and S r The difference between them; for block l:

[0021]

[0022] Among them, G' represents the grid feature after self-attention, R' represents the regional feature after self-attention, S' grepresents the mesh features after interaction, S r ' represents the regional features after interaction, MCA represents the multi-head cross attention module, and LayerNorm represents layer normalization. and These are the two new text features obtained; based on the previous assumptions, and should be very similar; therefore, using the alignment part to improve and similarity;

[0023] In the alignment part, a text consistency loss is designed, including a diagonal loss and a KL loss; in the alignment part, there is a loss consisting of and The calculated cross-correlation matrix, and the closer the diagonal is to 1, the and The more similar; In addition, using KL makes and have similar characteristic distributions.

[0024] In one embodiment of the present invention, cross entropy loss is also used during the training process; due to the diagonal loss L diag , KL loss L KL , cross entropy loss loss ce The training focus is different and has different weights; based on this, the cross entropy training phase is split; first, at the beginning of training, a warm-up phase is performed, in which the cross entropy loss, diagonal loss, and KL loss all have a learnable parameter as a weight; then, after the warm-up phase, the weight of the cross entropy loss is fixed to 1, and the remaining losses still have learnable parameters, which are defined as follows:

[0025] L TC =L diag +L kl (11)

[0026]

[0027]

[0028] Among them, loss diag 、loss kl The diagonal loss L diag , KL loss L kl , batch represents the training batch, L TC represents text consistency loss, α, β, γ represent learnable parameters, ⊙ represents the calculation method, and epoch<n represents the warm-up phase.

[0029] In one embodiment of the present invention, α⊙loss ce The representation is as follows:

[0030]

[0031] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0032] Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention introduces a bidirectional interactive encoder, which uses a cross-interaction method to make the two image features complement each other and enrich their own visual information. In the decoder, a cross-alignment module is proposed. This module enables the text features to interact in the order of "region feature-grid feature" and "grid feature-region feature" to obtain two new text features, and explores the similarity between the two text features. This converts the semantic gap problem between the region feature and the grid feature into a similarity problem between the two text features. A large number of experiments on the MS COCO dataset have proved that the proposed model has achieved competitive results on the Karpathy segmentation test set. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a flow chart of the method of the present invention.

[0034] Figure 2 This is the multi-head self-attention and multi-head cross attention of the present invention.

[0035] Figure 3 This is the cross alignment module of the present invention.

[0036] Figure 4 Visualization of text descriptions generated for the method of the present invention and related comparisons. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0040] The present invention provides an image description method for enhancing bidirectional image feature interaction. A bidirectional interactive encoder is introduced to make two image features complement each other in a cross-interaction manner to enrich the visual information. In the decoder, a cross-alignment module is proposed to make text features interact in the order of "region feature-grid feature" and "grid feature-region feature" to obtain two new text features, and explore the similarity between the two new text features.

[0041] The following is the specific implementation process of the present invention.

[0042] 1. Network structure

[0043] like Figure 1 As shown in the figure, the method of the present invention is based on the Transformer model, the number of encoders is 3, and the number of decoders is 2. The regional feature has rich spatial information, but lacks global information, while the grid feature has rich global information. The fusion of the two features can effectively make up for their respective shortcomings. The input of the entire model includes the regional feature and mesh features Where N R Indicates the number of regions, N G Indicates the number of grids. The region features are extracted by VinVL, while the grid features are extracted by the visual encoder of CLIP. The region features and grid features are first enhanced by the bidirectional interactive encoder and then passed to the decoder. The cross alignment module in the decoder alleviates the semantic gap between the two image features and improves the fusion effect.

[0044] 2. Bidirectional interactive encoder

[0045] The bidirectional interactive encoder consists of three identical blocks, each of which is composed of a self-attention module and a cross-attention module. Among them, the multi-head cross-attention module is composed of a multi-head cross-attention module and a feedforward network module. Figure 2 shown.

[0046] In the bidirectional interactive encoder, the grid features and the region features will pass through the self-attention module and the cross-attention module in turn. The self-attention module is used to strengthen the important information of the image features themselves, and the cross-attention module is used to make up for the shortcomings of the image features themselves. Take the first encoding block as an example:

[0047] G l+1 =LayerNorm(MSA(G l ,G l ,G l )+G l ) (1)

[0048] R l+1 =LayerNorm(MSA(R l ,R l ,R l )+R l ) (2)

[0049] Among them, G represents grid features, R represents regional features, MSA represents multi-head self-attention module, and LayerNorm represents layer normalization. As mentioned before, after self-attention, the information of each other will be exchanged through the cross-attention module to make up for the shortcomings of regional features lacking global features and grid features lacking spatial information. Finally, in order to ensure the integrity of the original features as much as possible, the features before and after enhancement are defined as follows:

[0050] G' l+1 =LayerNorm(MCA(G l ,R l ,R l )+G l ) (3)

[0051] R' l+1 =LayerNorm(MCA(R l ,G l ,G l ),R l ) (4)

[0052]

[0053]

[0054] Among them, G represents grid features, R represents regional features, MCA represents multi-head cross attention module, LayerNorm represents layer normalization, and FFN represents feedforward network layer. After multiple bidirectional interaction blocks, grid features and regional features have initially acquired their missing information. Moreover, after interaction, they can provide some help in alleviating the semantic gap.

[0055] 3. Cross Alignment Module

[0056] Grid features and region features are different representations of the same image. They are not only complementary but also homologous. Therefore, the present invention believes that if the text features are interacted in the order of "region feature-grid feature" and "grid feature-region feature", the feature representations obtained should have a very high similarity. Based on this assumption, the present invention method designs a cross alignment module. Figure 3 As shown, the cross alignment module consists of two parts: a cross part and an alignment part.

[0057] The crossover part is mainly used to realize the purpose of text features interacting in the order of "region feature-grid feature" and "grid feature-region feature", while the alignment part is used to improve feature similarity. Although region features and grid features come from the same image, they have different ways of extracting features, so there is a semantic gap between the two features. g and S r Therefore, through the cross section, the present invention can indirectly solve S g and S r The difference between them. Take block 1 as an example:

[0058]

[0059] Among them, G' represents the grid feature after self-attention, R' represents the regional feature after self-attention, S' g represents the mesh features after interaction, S r ' represents the regional features after interaction, MCA represents the multi-head cross attention module, and LayerNorm represents layer normalization. and These are the two new text features obtained; based on the previous assumptions, and should be very similar; therefore, using the alignment part to improve and The similarity.

[0060] In the alignment part, the present invention designs a text consistency loss. The loss includes a diagonal loss and a KL loss. Figure 3 As shown, in the alignment part there is a and The calculated cross-correlation matrix, and the closer the diagonal is to 1, the and In addition, the present invention uses KL to make and have similar feature distributions. During the training process, the present invention will also use cross-entropy loss. However, the training focuses of the three losses are different, and they should have different weights. Based on this, the present invention splits the cross-entropy training stage. First, at the beginning of training, the present invention first performs a warm-up stage, in which the cross-entropy loss, diagonal loss, and KL loss all have a learnable parameter as a weight. Then, after the warm-up stage, the weight of the cross-entropy loss is fixed to 1, and the remaining losses still have learnable parameters, which are specifically defined as follows:

[0061] L TC =L diag +L kl (11)

[0062]

[0063] Among them, loss diag 、loss kl The diagonal loss L diag , KL loss L kl , batch represents the training batch, L TC represents text consistency loss, α, β, γ represent learnable parameters, ⊙ represents the calculation method, epoch<n represents the warm-up phase, and α⊙loss ce For example:

[0064]

[0065] In order to fairly verify the effectiveness of the BIANet proposed in the present invention in the image description task, the present invention conducted experiments on the MS COCO dataset which is popular in the image description task. MS COCO is a dataset widely used in image description tasks, which contains a total of 123,287 images. The training set has 82,783 images, the validation set has 40,504 images, and the test set has 40,775 images, and each image also carries 5 different related text descriptions. In addition, the present invention adopts the allocation method proposed by Karpathy in the experiment. MS COCO is allocated according to 5,000 images as the validation set, 5,000 images as the test set, and the remaining images as the training set. For the convenience of training, the 5 text descriptions carried by each image will be converted into lowercase, and words with a frequency of less than 5 will be filtered out.

[0066] 1. Qualitative comparison

[0067] Figure 4 Some qualitative results are provided to demonstrate the performance of the method of the present invention. Figure 4 The underlined part of the word indicates the incorrect word. Figure 4As can be seen from the figure, the base model already has good results and the generated text description can accurately express the content of the image. However, compared with the base model, this method has fewer errors and can generate more detailed text descriptions.

[0068] 2. Quantitative comparison

[0069] Table 1. Performance comparison on the MS COCO test set under the “Karpathy” allocation. All values ​​are expressed in percentage (%).

[0070]

[0071]

[0072] Table 2 Comparison with pre-trained large models on the MS COCO test set

[0073] BLEU-1 BLEU-4 METEOR ROUGR CIDEr SPICE ViTCAP - 41.2 30.1 - 138.1 24.1 OSCAR - 40.5 29.7 60.1 137.6 22.8 VinVL - 40.9 30.9 - 140.4 25.1 BLIP - 40.4 - - 136.7 - Ours 83.5 41.8 30.8 60.8 142.7 24.6

[0074] In order to more intuitively compare the method of the present invention with other algorithms, the present invention not only compares the method of the present invention with the model trained from scratch on the MS COCO test set, but also compares it with some pre-trained large models. BLEU, METEOR, ROUGE, CIDEr and SPICE are used as evaluation indicators of the present invention. Tables 1 and 2 respectively show the quantitative evaluations obtained under the two test conditions, and the optimal values ​​in each column of data are bolded. In Table 1, it can be found that except for BLEU-4, the method of the present invention has achieved the optimal value for all other indicators. In Table 2, compared with the pre-trained large model, the method of the present invention has achieved the maximum value in each indicator. In summary, the method of the present invention can not only generate detailed text descriptions, but also has excellent performance.

[0075] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0076] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0077] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. An image description method for enhancing bidirectional image feature interaction, characterized in that: A bidirectional interactive encoder is introduced to make the two image features complement each other in a cross-interactive way and enrich the visual information. In the decoder, a cross-alignment module is proposed to make the text features interact in the order of "region feature-grid feature" and "grid feature-region feature" to obtain two new text features, and explore the similarity between the two new text features.

2. The image description method for enhancing bidirectional image feature interaction according to claim 1, characterized in that: The method is based on the Transformer model, the number of bidirectional interactive encoders is 3, and the number of decoders is 2.

3. The image description method for enhancing bidirectional image feature interaction according to claim 1, characterized in that: The input of the bidirectional interactive encoder includes regional features and mesh features Where N R Indicates the number of regions, N G Indicates the number of grids; The regional features and grid features are first enhanced by the bidirectional interactive encoder and then passed to the decoder. The semantic gap between the regional features and the grid features is alleviated by the cross alignment module in the decoder.

4. The image description method for enhancing bidirectional image feature interaction according to claim 3, characterized in that: The region features are extracted by VinVL, and the mesh features are extracted by the visual encoder of CLIP.

5. The image description method for enhancing bidirectional image feature interaction according to claim 3, characterized in that: The bidirectional interactive encoder consists of three identical encoding blocks, each of which is composed of a self-attention module and a cross-attention module.

6. The image description method for enhancing bidirectional image feature interaction according to claim 5, characterized in that: The specific implementation of the two-way interactive encoder is as follows: In the bidirectional interactive encoder, the grid features and the region features will pass through the self-attention module and the cross-attention module in turn. The self-attention module is used to strengthen the important information of the image features themselves, and the cross-attention module is used to make up for the shortcomings of the image features themselves. For the l-th encoding block: G l+1 =LayerNorm(MSA(G l ,G l ,G l )+G l ) (1) R l+1 =LayerNorm(MSA(R l ,R l ,R l )+R l ) (2) Among them, G represents grid features, R represents regional features, MSA represents multi-head self-attention module, and LayerNorm represents layer normalization. After the self-attention module, the information between each other will be exchanged through the cross-attention module to make up for the shortcomings of regional features lacking global features and the lack of spatial information of grid features. Finally, in order to ensure the integrity of the original features as much as possible, the features before and after enhancement are defined as follows: G' l+1 =LayerNorm(MCA(G l ,R l ,R l )+G l ) (3) R' l+1 =LayerNorm(MCA(R l ,G l ,G l ),R l ) (4) Among them, G represents grid features, R represents regional features, MCA represents multi-head cross attention module, LayerNorm represents layer normalization, and FFN represents feedforward network module; after two-way interaction, grid features and regional features have initially acquired their missing information.

7. The image description method for enhancing bidirectional image feature interaction according to claim 6, characterized in that: The cross alignment module consists of two parts: the cross part and the alignment part. The specific implementation is as follows: The crossover part realizes the purpose of text features interacting in the order of "regional features-grid features" and "grid features-regional features", and the alignment part is to improve the feature similarity; although the regional features and grid features come from the same image, they have different ways of extracting features, and there is a semantic gap between the two features. The S g and S r will also be different; therefore, through the cross section, S is indirectly solved g and S r The difference between them; for block l: Among them, G' represents the grid feature after self-attention, R' represents the regional feature after self-attention, S' g represents the mesh features after interaction, S r ' represents the regional features after interaction, MCA represents the multi-head cross attention module, and LayerNorm represents layer normalization. and These are the two new text features obtained; based on the previous assumptions, and should be very similar; therefore, using the alignment part to improve and similarity; In the alignment part, a text consistency loss is designed, including a diagonal loss and a KL loss; in the alignment part, there is a loss consisting of and The calculated cross-correlation matrix, and the closer the diagonal is to 1, the and The more similar; In addition, using KL makes and have similar characteristic distributions.

8. The image description method for enhancing bidirectional image feature interaction according to claim 7, characterized in that: During the training process, cross entropy loss is also used; due to the diagonal loss L diag , KL loss L kl , cross entropy loss loss ce The training focus is different and has different weights; based on this, the cross entropy training phase is split; first, at the beginning of training, a warm-up phase is performed, in which the cross entropy loss, diagonal loss, and KL loss all have a learnable parameter as a weight; then, after the warm-up phase, the weight of the cross entropy loss is fixed to 1, and the remaining losses still have learnable parameters, which are defined as follows: L TC =L diag +L kl (11) Among them, loss diag 、loss kl The diagonal loss L diag , KL loss L kl , batch represents the training batch, L TC represents text consistency loss, α, β, γ represent learnable parameters, ⊙ represents the calculation method, and epoch<n represents the warm-up phase.

9. The image description method for enhancing bidirectional image feature interaction according to claim 8, characterized in that: α⊙loss ce The representation is as follows:

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 9 can be implemented.

Images