Image description generation method and device based on multi-level interaction of visual features and text features

Abstract

The present invention discloses a method and device for generating image description based on multi-level interaction of visual features and text features. The method comprises the following steps: obtaining an image to be described, extracting regional features and grid features of the image respectively, inputting the features into a feature enhancement encoder based on a Transformer architecture for feature extraction, and fusing the features to obtain the final enhanced image features; cropping the obtained image to be described into multiple sub-images of different sizes, and retrieving each sub-image to obtain text descriptions of the sub-images of different sizes; mapping the text descriptions of each sub-image and the enhanced image features to the same semantic space and performing multi-level interaction to calibrate the semantic information and visual features based on multi-layer semantic guidance to obtain calibrated global features; and decoding the calibrated global features to obtain the final image description result. The present invention can accurately understand the image context scene and generate accurate and rich descriptions.

Description

Technical Field

[0001] The present invention relates to the field of image description technology, and in particular to a method and device for generating image description based on multi-level interaction of visual features and text features. Background Art

[0002] The goal of image description is to understand the objects in the image and their relationships with each other in different image regions, and then selectively process these relationships based on their semantic relationship with each generated word. Inspired by machine translation models, most current image description generation methods generally adopt an encoder-decoder architecture, for example, using a CNN (convolutional neural network) as an encoder to extract visual features and then using an RNN (recurrent neural network) as a decoder to generate descriptions word by word.

[0003] The Transformer model is a neural network model based on the self-attention mechanism. Compared with traditional recurrent neural network models, the Transformer model has better parallel performance and shorter training time. Transformer-based models have become the mainstream image description method. However, traditional Transformer-based image description methods have the following problems:

[0004] 1. Inability to effectively capture contextual scene features. Because the model typically extracts image features directly through the self-attention mechanism, contextual scene information is not always explicitly taken into account in the attention weight calculation. However, the features of objects in the image are usually associated with scene features. Therefore, when object features are not prominent enough, contextual scene features cannot be effectively captured, resulting in an inability to correctly predict the object category. For example, a cat may appear in the grass, while a car may be parked on the side of the road. If the model cannot effectively understand the scene features, when the object features are not prominent, it will be difficult for the model to correctly infer the object category.

[0005] 2. Lack of interaction of global information. Rich feature information and the interaction between multimodal (visual and language) features are crucial for image description tasks. However, in traditional description methods, the interaction between text and image only occurs in the decoder, resulting in the model being unable to accurately understand certain complex scenes in the image or distinguish similar objects, making it difficult to map visual features well to corresponding semantic concepts. In addition, because the interaction between multimodal features only occurs in the cross-attention between generated words and visual features, there is a lack of interaction in global context information. In addition, there is a lack of clear text information as an aid in the image encoding process, which in turn limits the reasoning ability to a certain extent and makes the description results relatively simple. Summary of the Invention

[0006] The technical problem to be solved by the present invention is: In response to the technical problems existing in the prior art, the present invention provides an image description generation method and device based on multi-level interaction between visual features and text features, combined with feature enhancement and multi-level interaction between visual features and text features, which can accurately understand the image context scene and generate accurate and rich descriptions.

[0007] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0008] A method for generating image descriptions based on multi-level interaction of visual features and text features, comprising the following steps:

[0009] Acquiring an image to be described, extracting regional features and grid features of the image from the acquired image to be described, respectively, inputting the extracted regional features and grid features into a feature enhancement encoder based on a Transformer architecture for feature extraction, wherein the feature enhancement encoder is an encoder based on the Transformer architecture and uses a visual feature enhancement module to replace the multi-head attention layer in the Transformer architecture, and fusing the outputs of the regional features and grid features obtained by the feature enhancement encoder to obtain a final enhanced image feature;

[0010] The obtained image to be described is cropped into multiple sub-images of different sizes, and each sub-image is retrieved to obtain text descriptions of the sub-images of different sizes;

[0011] Mapping each sub-image text description and the enhanced image features to the same semantic space and performing multi-level interaction to achieve calibration of semantic information and visual features based on multi-layer semantic guidance to obtain a calibrated global feature, wherein when performing multi-level interaction, features of different semantic levels are obtained by calculating each sub-image text description and the enhanced image features, and the features of each semantic level are interactively integrated layer by layer to finally obtain the calibrated global feature;

[0012] The calibrated global features are decoded to obtain a final image description result.

[0013] Furthermore, the step of inputting the extracted regional features and grid features into a feature enhancement encoder based on a Transformer architecture for feature extraction includes:

[0014] The extracted regional features and grid features are respectively input into the visual feature enhancement module for visual feature enhancement. The visual feature enhancement module includes an MLP branch, a self-attention branch, and a fusion module. The MLP branch learns the global scene features in the input features, the attention branch captures the object relationship of the image in the input features, and the fusion module fuses the output results of the MLP branch and the self-attention branch to generate the final enhanced feature output;

[0015] The enhanced features obtained by passing the regional features and the grid features through the visual feature enhancement module are respectively passed through the feedforward neural network and the AddNorm module to obtain a feature output sequence.

[0016] Furthermore, in the MLP branch, the input features are divided into multiple parts along the feature dimension, each feature part after division is normalized and then input into the linear layer respectively, and the outputs of each feature part after passing through the linear layer are connected and output as the result of the MLP branch. The calculation expression is:

[0017] h i =W i iNorm(X i )+b i

[0018] O bottom =Concat(h1,h2,......h H )

[0019] Among them, h i represents the output of the i-th feature part after the linear layer, H represents the number of input feature divisions, and W i 、b i They represent the weight and bias corresponding to the i-th feature part, Norm represents the normalization operation, and Concat represents the connection operation.

[0020] Furthermore, the self-attention branch adopts a single-head self-attention mechanism, and the calculation expression is:

[0021] Q=X in W Q , K=X in W K , V=X in W V

[0022]

[0023] Among them, W Q , W K , W V are respectively the learnable mapping matrices, Otop is the output of the self-attention branch, X in Represents the input features.

[0024] Furthermore, the step of mapping the text description of each sub-image and the enhanced image features into the same semantic space and performing multi-level interaction includes:

[0025] Linearly projecting the text descriptions of the sub-images of different sizes respectively so that the resulting text descriptions and the enhanced image feature maps are aligned in the same semantic space;

[0026] The results of linear projection of the text description of each sub-image are respectively subjected to inner product operation with the enhanced image features, and after layer normalization, features at different semantic levels are obtained;

[0027] The features of each semantic level are interactively integrated layer by layer to obtain the calibrated global features, wherein the features of the first layer and the second layer are spliced ​​and linearly transformed to obtain integrated features, and the currently obtained integrated features are spliced ​​and linearly transformed with the features of the next layer to obtain new integrated features. This is executed in sequence until the last layer is spliced ​​and linearly transformed to obtain integrated features, and finally all the obtained integrated features are spliced ​​and linearly transformed with the features of the first layer again to obtain the final integrated features as the calibrated global features.

[0028] Furthermore, after the acquired image to be described is cropped into 1, 5, and 9 sub-images, the text description set T is obtained. sub , where the text description set T sub Including the first text description T obtained by dividing the corresponding sub-image into 1 one , the second text description T obtained by dividing into 5 sub-images five And the third text description T obtained by dividing into 9 sub-images nine , the features of each semantic level are calculated according to the following formula:

[0029] F one =σ(LN(T one W one ⊙X))

[0030] F five =σ(LN(T five W jive ⊙X))

[0031] F nine =σ(LN(T nine W nine ⊙X))

[0032] Among them, W one , W five, W nine are respectively the learnable mapping matrices, LN represents layer normalization, ⊙ represents the inner product operation element by element, σ represents the ReLU activation function, and F one 、F five 、F nine Respectively represent the first text description T one , second text description T five And the third text description T nine The corresponding features of different semantic levels;

[0033] The features of each semantic level are concatenated and linearly transformed in sequence according to the following formula:

[0034] C1=Linear[F one , F fice ]

[0035] C2=Linear[F nine , C1]

[0036] FM=[C1,C2,F one ]

[0037] F HSGM =Linear(FM)

[0038] Among them, [,] represents the splicing operation, Linear represents the linear transformation, and C1 is the one With F five The integrated feature obtained after integration, C2 is F five The integrated feature obtained after integration with C1, FM is the integration of C1, C2 and F one The result after splicing, F HSGM is the calibrated global feature obtained by passing FM through the linear layer.

[0039] Furthermore, the calibrated global features are input into a decoder based on a Transformer model to decode and generate a final image description result, the steps including:

[0040] Input each word into the masked self-attention module to model the relationship between words, and output the result after masked self-attention of each word;

[0041] Providing the output of the masked self-attention module and the calibrated global features to a multi-head attention interaction module to model the cross-modal relationship between the calibrated global features and the results of the masked self-attention of each word;

[0042] The output of the multi-head attention interaction module is sequentially passed through the feedforward neural network, residual and layer normalization to obtain the decoded output sequence;

[0043] The decoded output sequence is linearly transformed and classified by the Softmax function to generate the final image description result.

[0044] An image description generation device based on multi-level interaction of visual features and text features, comprising:

[0045] An encoding module is used to obtain an image to be described, extract regional features and grid features of the image from the image to be described, and input the extracted regional features and grid features into a feature enhancement encoder based on the Transformer architecture for feature extraction. The feature enhancement encoder uses a visual feature enhancement module to replace the multi-head attention layer in the Transformer architecture, and fuses the outputs of the regional features and grid features obtained by the feature enhancement encoder to obtain the final enhanced image features;

[0046] A retrieval module is used to crop the acquired image to be described into multiple sub-images of different sizes, and retrieve each sub-image to obtain text descriptions of the sub-images of different sizes;

[0047] a multi-level semantic guidance interaction module, configured to map the text descriptions of each sub-image and the enhanced image features to the same semantic space and perform multi-level interaction, thereby calibrating semantic information and visual features based on multi-level semantic guidance to obtain calibrated global features. During the multi-level interaction, features at different semantic levels are calculated by respectively calculating the text descriptions of each sub-image and the enhanced image features, and the features at each semantic level are interactively integrated layer by layer to ultimately obtain the calibrated global features.

[0048] The decoding module is used to decode the calibrated global features to obtain a final image description result.

[0049] Furthermore, the multi-level semantic guidance interaction module includes:

[0050] Multiple interaction branches, each of which includes a linear layer, an inner product operation unit, and a layer normalization unit connected in sequence, each of which receives a sub-image text description and outputs an interaction feature, and in each of the interaction branches, linearly projects each sub-image text description through the linear layer so that each result text description is aligned with each sub-image text description, and the inner product operation unit receives the output of the linear layer and the enhanced image feature for inner product operation, and then obtains features interacting with different semantic levels after layer normalization by the layer normalization unit;

[0051] An integration unit is used to gradually integrate the features interacting with different semantic levels to obtain calibrated global features; the integration unit includes multiple integration sub-units, each integration sub-unit includes a splicing circuit and a linear layer connected in sequence, the input end of the first integration sub-unit is respectively connected to the output ends of the first and second interaction branches, the output end of the first integration sub-unit and the output end of the third interaction branch are respectively connected to the input end of the second integration sub-unit, the input end of the next integration sub-unit is respectively connected to the output end of the previous integration sub-unit and the output end of the next interaction branch, and each integration sub-unit is further connected to the first integration sub-unit through an integration sub-unit.

[0052] A computer device includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0053] Compared with the prior art, the advantages of the present invention are:

[0054] 1. The present invention can improve the accuracy of visual feature extraction and effectively capture contextual scene features by enhancing image features. At the same time, combined with multi-level layer-by-layer interaction of visual and textual information, it can gradually integrate visual and textual information, fully integrate visual and textual information, improve the matching degree between visual features and textual features, and thus generate more accurate and richer descriptions.

[0055] 2. The present invention further adopts a multi-level semantic guidance interaction module MSGM to conduct multi-level interaction between visual features and text features, forming an image-text interaction mechanism with semantic guidance. The progressive fusion method can gradually integrate the description of the sub-image from coarse to fine into the visual features, and gradually connect the generated more detailed text concepts with the corresponding image features, promote the alignment between the two modalities, and improve the matching degree between visual features and text features, thereby further improving the accuracy of image description.

[0056] 3. The present invention further enhances image features by adopting a visual feature enhancement module, using an attention part to focus on object features, and a multi-layer perceptron (MLP) part to extract global scene information. It can focus on the most relevant space and help the model understand which features in the current image are most important, so that more accurate visual features can be extracted. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 3 is a schematic diagram comparing the frameworks of a traditional image description generation method and the image description generation method of the present invention.

[0058] Figure 2It is a schematic diagram of the architecture of the image description generation device based on multi-level interaction of visual features and text features of the present invention.

[0059] Figure 3 This is a schematic diagram of the implementation flow of the image description generation method based on multi-level interaction of visual features and text features in this embodiment.

[0060] Figure 4 Schematic diagram of the structure of the visual feature enhancement module (VFEM) in this embodiment.

[0061] Figure 5 Schematic diagram of the structure of the multi-level semantic guidance interaction module (MSGM) in this embodiment. DETAILED DESCRIPTION

[0062] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0063] See also Figure 1 ,in Figure 1 The above part corresponds to a traditional image description method based on the Transformer model. This method performs pre-training feature extraction on the input image, extracts regional features and grid features respectively, and directly inputs the regional features and grid features into the image description generation model. The image description generation model is a Transformer model. The Transformer model extracts and fuses regional features and grid features based on the attention mechanism, and generates the final image description after decoding. However, since the Transformer model directly extracts image features through the self-attention mechanism, when the object features are not prominent enough, it cannot effectively capture the contextual scene features, resulting in the inability to correctly predict the object category. Moreover, since the interaction between text and image only occurs through the cross-attention between words and visual features in the decoder, the global context information lacks interaction, resulting in the model may not be able to accurately understand certain complex scenes in the image or distinguish similar objects, and the description results are relatively simple. Figure 1 Taking the example in the figure as an example, the image description generation model using the Transformer model cannot accurately understand the context at this time (dogs are usually not on trucks) and cannot distinguish between "dog" and "cow" from the overhead view, resulting in the generation of the wrong word "dog".

[0064] Figure 1The middle and lower parts correspond to the image description generation method based on the multi-level interaction of visual features and text features of the present invention. The present invention first performs pre-training feature extraction on the input image, extracts regional features and grid features respectively, and performs feature enhancement to form enhanced image features. At the same time, the input image is cropped into multiple sub-images of different sizes, and each sub-image is retrieved to obtain text descriptions of sub-images of different sizes. The enhanced image features and the text descriptions of each sub-image are input into the image description generation model. The image description generation model maps the text descriptions of each sub-image to the same semantic space with the enhanced image features and performs multi-level interaction to calibrate the global features to obtain calibrated global features. After decoding, the final image description result is obtained. Due to the feature enhancement of the image features, the extraction accuracy of the visual features can be improved, and the context scene features can be effectively captured. At the same time, combined with the multi-level layer-by-layer interaction of visual and textual information, the visual and textual information can be gradually integrated step by step, the visual and textual information can be fully integrated, and the matching degree between the visual features and the textual features can be improved, so that a more accurate and richer description can be generated.

[0065] like Figure 2 The figure shows a schematic diagram of the structure of an image description generation device applicable to the present invention. The image description device includes an encoding module, a retrieval module, a multi-level semantic guidance interaction module (MSGM), and a decoding module. The encoding module includes a feature extraction unit, a feature enhancement encoder, and an addition unit. The feature extraction unit is used to extract the regional features of the image from the input image. Mesh Features N G and N R are the number of corresponding features, n is the serial number of the feature, and they are provided to the feature enhancement encoder Nx respectively. The feature enhancement encoder Nx is based on the Transformer architecture and uses the visual feature enhancement module (VFEM) to replace the multi-head attention layer in the original Transformer architecture. The feature enhancement encoder Nx can not only extract features but also enhance features. The addition unit is used to add regional features I G , Mesh Features I R The outputs of the feature enhancement encoders are added and fused to obtain the final enhanced image feature X, which is provided to the multi-level semantic guidance interaction module (MSGM). The retrieval module is used to crop the input image into multiple sub-images of different sizes, and retrieve the corresponding text description for each sub-image to form a text description set T sub , and provide it to the multi-level semantic guidance interaction module (MSGM). The multi-level semantic guidance interaction module (MSGM) receives the enhanced image feature X output by the encoding module and the text description set T output by the retrieval module. sub, by taking the text description set T sub The text description of each sub-image is mapped to the same semantic space as the enhanced image feature X and interacts with it at multiple levels. Based on multi-layer semantic guidance, the semantic information and visual features are calibrated to obtain the calibrated global feature F, which is provided to the decoding module. The decoding module decodes and outputs the image description result.

[0066] like Figure 3 As shown, the steps of the image description generation method based on multi-level interaction of visual features and text features in this embodiment include:

[0067] Step S01. Obtain an image to be described, extract the regional features and grid features of the image from the acquired image to be described, and input the extracted regional features and grid features into a feature enhancement encoder based on the Transformer architecture for feature extraction. The feature enhancement encoder is an encoder based on the Transformer architecture and uses a visual feature enhancement module (VFEM) to replace the multi-head attention layer in the Transformer architecture. The outputs of the regional features and grid features obtained by the feature enhancement encoder are fused to obtain the final enhanced image features.

[0068] In this embodiment, the image to be described is input into the pre-trained VinVL (object attribute detection model) model, and the pre-trained VinVL model extracts regional features from the input image to be described. Regional features describe the attributes or quantitative indicators of a specific region in an image. They are unique properties exhibited within a specific image region. Regional features can include color features, texture features, shape features, etc., and can be extracted and analyzed through deep learning methods. The image to be described is input into the frozen CLIP (Contrastive Language-Image Pre-training)-I model, which freezes the specified parameters in the CLIP-I model and extracts the grid features of the image. Grid features are obtained by dividing the image into several regular grid structures and then extracting features from each grid unit using a convolutional neural network. This feature extraction method can capture the local information of the image.

[0069] The CLIP model is a text-image pre-training multimodal learning model based on contrastive learning. It uses large-scale image and text data for pre-training, mapping images and text into the same embedding space. CLIP is trained using paired text-image data, meaning each image has a matching text description, allowing it to learn cross-modal semantic relationships between text and images. CLIP consists of two parts: a text encoder and an image encoder. The TextEncoder is primarily responsible for extracting text features and can use the language Transformer model used in natural language processing tasks. The image encoder is responsible for extracting visual features from images and can use a CNN model or a visual Transformer.

[0070] In this embodiment, the regional feature I is extracted G and mesh feature I R Then it is provided to the feature enhancement encoder Nx. The feature enhancement encoder Nx is constructed by using the visual feature enhancement module (VFEM) to replace the multi-head attention layer in the original Transformer architecture on the basis of the Transformer architecture encoder. The rest of the parts are the same as the Transformer encoder. That is, in addition to the visual feature enhancement module (VFEM), the feature enhancement encoder Nx also includes the feedforward neural network and ADD&Norm (residual connection and layer normalization) in the Transformer architecture. The visual feature enhancement module (VFEM) is connected to the feedforward neural network and ADD&Norm in sequence. The visual feature enhancement module (VFEM) receives the regional features I extracted through pre-training respectively. G and mesh feature I R After passing through the feedforward neural network and ADD&Norm, the output feature sequence is formed. Regional feature I G and mesh feature I R The outputs of the feature enhancement encoders Nx are then added together to achieve fusion and form enhanced image features X.

[0071] See also Figure 4 In this embodiment, the visual feature enhancement module (VFEM) includes an MLP branch (the lower branch in the figure), a self-attention branch (the upper branch in the figure), and a fusion module. The input ends of the MLP branch and the self-attention branch are respectively connected to the image features (regional features I G / Grid Features I R ), the output ends of the MLP branch and the self-attention branch are connected to the input end of the fusion module respectively, and the image feature O formed by the feature enhancement is output from the output end of the fusion module VFEM, where the MLP branch learns the global scene features in the input image features, the attention branch captures the object relationship of the image in the input image features, and the fusion module fuses the output result Obottom of the MLP branch with the output result Otop of the self-attention branch to generate the final enhanced feature O VFEM Output, enhanced feature O VFEM The output calculation expression can be expressed as:

[0072] O VEFM =O bottom (X in )+O top (X in ) (1)

[0073] Among them, O bottom and O top denote the outputs of the MLP branch and the self-attention branch, respectively. It is the image feature (region feature I) extracted by the pre-trained model from the input image to be described G / Grid Features I R ), n is the number of features, and d is the feature dimension.

[0074] This embodiment enhances image features by adopting the visual feature enhancement module of the above structure. The MLP branch can learn global scene features, and the self-attention branch can capture the object relationships in the image. The enhanced features formed after the final fusion can effectively enhance the visual representation ability.

[0075] Optionally, the MLP branch may specifically include a head division unit, a normalization layer, a linear layer, and a splicing unit connected once, wherein the head division unit converts the input image feature x in (Regional Characteristics I G / Grid Features I R ) is divided into multiple parts along the feature dimension, and each part has a corresponding weight and bias b i , each feature part after division is provided to the normalization layer for normalization and then input into the linear layer respectively. The outputs of each feature part after passing through the linear layer are connected and output as the result of the MLP branch. The calculation expression can be expressed as:

[0076]

[0077] Among them, h i represents the output of the feature part of the i-th partition after passing through the linear layer, H represents the number of input feature partitions, and W i 、b iThey represent the weight and bias corresponding to the i-th feature part, Norm represents the normalization operation, and Concat represents the connection operation.

[0078] The weight matrix W of the above-mentioned MLP part will be updated collaboratively according to different input images. When adjusting the dynamic parameters, it can not only fully consider the characteristics of the current image, but also integrate the feature information of previous images. Therefore, the above mechanism can make the model more adaptable and robust, thereby providing the model with scene features rich in global context information in different scenarios.

[0079] This embodiment enhances image features by adopting a visual feature enhancement module (VFE M). All inputs jointly update a weight matrix so that the weight matrix can implicitly represent the most informative features in the sample. The single-head self-attention focuses on the most relevant subspace, which can avoid attention distraction and help the model learn which features are most important to the output.

[0080] Optionally, to capture object-level features, the self-attention branch can adopt a single-head self-attention structure, which can focus on the most relevant space and help the model understand which features in the current image are most important. The calculation expression can be expressed as:

[0081] Q=X in W Q , K=X in W K , V=X in W V

[0082]

[0083] Among them, W Q , W K , W V are respectively the learnable mapping matrices, O top is the output of the self-attention branch.

[0084] In this embodiment, the steps of inputting the extracted regional features and grid features into a feature enhancement encoder based on a Transformer architecture for feature extraction include:

[0085] Step S101: The extracted regional features and grid features are respectively input into the visual feature enhancement module (VFEM) for visual feature enhancement. The MLP branch learns the global scene features in the input features, the attention branch captures the object relationship of the image in the input features, and the fusion module fuses the output results of the MLP branch and the self-attention branch to generate the final enhanced features. VFEM Output;

[0086] Step S102: The enhanced features obtained by passing the regional features and the grid features through the VFEM module are respectively passed through the feedforward neural network and the AddNorm module to obtain a feature output sequence.

[0087] The output of the feature enhancement encoder Nx in this embodiment can be specifically expressed as:

[0088] X R =AddNorm(Feedforward( VEFM (I R ))

[0089] X G =AddNorm(Feedforward( VEFM (I G )) (4)

[0090] Among them, AddNorm represents the combination of residual connection and normalization operation, x R and X G Represent the two visual features obtained by the encoding module respectively.

[0091] The final enhanced image feature X can be obtained by fusing the output of the feature enhancement encoder Nx according to the following formula:

[0092] X=X R +X G (5)

[0093] Step S02: Crop the acquired image to be described into multiple sub-images of different sizes, and retrieve each sub-image to obtain text descriptions of the sub-images of different sizes.

[0094] In this example, the image to be described is cropped into multiple sub-images of varying sizes. These sub-images are then fed into a frozen CLIP model for retrieval, generating text descriptions for the image sub-regions while retaining more detail of the input image. The CLIP model, trained on diverse, large-scale data, is a powerful cross-modal retrieval model. Using the CLIP model can enrich the model with richer multimodal information.

[0095] Specifically, the image to be described can be cropped into 1, 5, and 9 sub-regions respectively. These three cropping methods are represented by c1, c5, and c9 respectively. The image cropped into 1 sub-region corresponds to the original image. Each sub-image is input into the frozen CLIP model for retrieval to obtain the set of text descriptions of the sub-image T sub ={t i，j，k |i∈{1, 5, 9}, j∈{1, 2,…,c i}, k∈{1, 2, …, 12}}, where ti，j，k Represents a text description, the i subscript indicates whether it is from the original image, 5-crop or 9-crop, which corresponds to c1, c5, c9 respectively; j represents the sequence number of the sub-image under the corresponding cropping method; the k subscript represents the previous k retrievals. Therefore, the text description set T sub It contains 12 text descriptions corresponding to each sub-image under various cropping methods. sub The enhanced image features X from the post-processor and encoder are jointly input into the subsequent multi-level semantic guided interaction module MSGM, so that the text descriptions of sub-images of different sizes can be interacted with the enhanced image features x multiple times in MSGM.

[0096] It is understandable that the specific method of cropping (the number of cropping times, the number of cropped sub-regions, etc.) can be configured according to actual needs.

[0097] Step S03. Map the text description of each sub-image and the enhanced image features to the same semantic space and perform multi-level interaction to achieve calibration of semantic information and visual features based on multi-layer semantic guidance to obtain calibrated global features. When performing multi-level interaction, features of different semantic levels are obtained by calculating the text description of each sub-image and the enhanced image features respectively. The features of each semantic level are interactively integrated layer by layer to finally obtain the calibrated global features.

[0098] Since images and language modalities have different properties, it is difficult to explicitly align image features with text features. The training dataset of the CLIP model is a large amount of images and text collected from the Internet, which will lead to bias and noise, introduce more irrelevant semantic words, and as the image is divided into more and more sub-regions, the richness of the obtained visual detail information will also be significantly enhanced. This process is inevitably accompanied by the growth of noise information. To balance this contradiction, this embodiment uses the text description set T sub The enhanced image features X output by the encoder are then fed into the subsequent multi-level semantically guided interaction module MSGM. MSGM interacts the text descriptions of sub-images of different sizes with the enhanced image features x multiple times, gradually matching the visual features with the semantic information, thus achieving adaptive image-text interaction.

[0099] In this embodiment, the multi-level semantic guidance interaction module MSGM includes:

[0100] Multiple interaction branches, each interaction branch includes a linear layer, an inner product operation unit, and a layer normalization unit connected in sequence. Each interaction branch receives a sub-image text description and outputs an interaction feature. In each interaction branch, the sub-image text description is linearly projected through a linear layer to align each result text description with the sub-image text description. The inner product operation unit receives the output of the input linear layer and the enhanced image feature for inner product operation, and then performs layer normalization by the layer normalization unit to obtain features that interact with different semantic levels.

[0101] The integration unit is used to gradually integrate the features interacting with different semantic levels to obtain calibrated global features; the integration unit includes multiple integration sub-units, each integration sub-unit includes a splicing circuit and a linear layer connected in sequence, the input end of the first integration sub-unit is connected to the output end of the first and second interaction branches respectively, the output end of the first integration sub-unit and the output end of the third interaction branch are connected to the input end of the second integration sub-unit respectively, the input end of the next integration sub-unit is connected to the output end of the previous integration sub-unit and the output end of the next interaction branch respectively, and each integration sub-unit is connected to the first integration sub-unit through an integration sub-unit.

[0102] By adopting the multi-level semantically guided interaction module MSGM with the above structure, a semantically guided image-text interaction mechanism is provided. The description of the sub-image from coarse to fine is gradually integrated into the visual features in a progressive fusion manner, and the generated more detailed text concepts are gradually connected with the corresponding image features, which promotes the alignment between the two modalities and improves the matching degree between visual features and text features, thereby improving the accuracy of image description.

[0103] In this embodiment, the steps of performing multi-level interactions between the text description of each sub-image and the enhanced image features include:

[0104] Step S301. Linearly project the text descriptions of the sub-images of different sizes so that the resulting text descriptions are aligned with the text descriptions of the sub-images;

[0105] Step S302: Perform inner product operations on the results of linear projection of the text description of each sub-image with the enhanced image features, and obtain features that interact with different semantic levels after layer normalization;

[0106] Step S303. The features interacting with different semantic levels are gradually integrated to obtain calibrated global features, wherein the features of the first layer and the next layer are spliced ​​and linearly transformed to obtain integrated features, and the currently obtained integrated features are spliced ​​and linearly transformed with the features of the next layer to obtain new integrated features. This is performed sequentially until the splicing and linear transformation of the last layer are completed, and finally all the obtained integrated features are spliced ​​and linearly transformed with the features of the first layer again to obtain the final integrated features as the calibrated global features.

[0107] by Figure 5 For example, the text description set T is obtained by cropping the image to be described into 1, 5 and 9 sub-images. sub Middle T sub include and T one Corresponding to the first text description obtained by dividing into 1 sub-image, T five Corresponding to the second text description obtained by dividing into 5 sub-images, T nine The third text description is obtained by dividing the image into 9 sub-images, ns represents the number of features in the sub-region, and the three interactive branches receive T one 、T five and T nine Processing, T one 、T five and T nine After passing through the linear layer in sequence and performing inner product operations with the enhanced image feature X, they are normalized at different levels to form semantic features F at different levels. one 、F five and F nine The integration unit specifically includes three integration subunits, the first integration subunit integrates F one With F five After concatenation and linear layer, the first integration result C1 is obtained, and then the first integration result C1 is combined with F nine F five After the concatenation and linear layers, the second integration result C1 is obtained. Finally, the first integration result C1, the second integration result C1 and F one The integration result FM obtained after the splicing and linear layer is passed through the linear layer to obtain the final calibrated global feature F. one Represents the unsegmented image area, which is least affected by noise, and uses F one To calibrate the global feature representation, the text description closest to the reference description can be obtained.

[0108] Specifically, the features interacting with different semantic levels can be calculated as follows:

[0109]

[0110] Among them, W one , W five , W nine are respectively the learnable mapping matrices, LN represents layer normalization, ⊙ represents the inner product operation element by element, σ represents the ReLU activation function, and F one 、F five 、F nine Respectively represent the first text description T one , second text description T five And the third text description T nine Interactive features.

[0111] Then, according to the following formula, the features interacting with different semantic levels are sequentially concatenated and linearly transformed to obtain the final calibrated global features:

[0112]

[0113] Among them, [,] represents the concatenation operation, Linear represents the linear transformation, C1 and C2 are the output results after integrating the features, and F HSGM is the final calibrated global feature.

[0114] Step S04: Decode the calibrated global feature F to obtain the final image description result.

[0115] The decoding module is used to generate output subtitles word by word based on the features of the encoding module. In this embodiment, the decoding module uses a decoder xN based on the Transformer model, which mainly includes a masked self-attention operation and a multi-head attention interaction module. The masked self-attention is used to model the relationship between words, and the multi-head attention interaction is used to model the final global feature output F of the encoding end and the t-th word Y t Y after masked self-attention MSA The cross-modal relationship is expressed as follows:

[0116]

[0117] Among them, O De is the decoded output of decoder xN.

[0118] The decoded output of decoder xN is linearly transformed and classified by Softmax function to generate the final image description result Y t+1 .

[0119] Specifically, the steps of decoding and generating the image description result include:

[0120] Step S401. Each word Y t Input into the masked self-attention module to model the relationship between words, and output the result after masked self-attention of each word;

[0121] Step S402. Provide the output of the masked self-attention module and the calibrated global feature F to the multi-head attention interaction module to model the cross-modal relationship between the calibrated global feature and the results of each word after masked self-attention.

[0122] This embodiment adopts the above method and uses the enhanced visual features and cross-modal text multiple interaction network (Mul-VT) to perform multiple interactions between enhanced visual features and cross-modal text, which can generate more accurate and richer descriptions.

[0123] The image description generation device based on multi-level interaction of visual features and text features in this embodiment includes:

[0124] An encoder module is used to obtain an image to be described, extract the region features and grid features of the image from the image to be described, enhance and encode the extracted region features and grid features, and fuse the encoding results of the region features and grid features to obtain enhanced image features;

[0125] A retrieval module is used to crop the acquired image to be described into multiple sub-images of different sizes, and retrieve each sub-image to obtain text descriptions of the sub-images of different sizes;

[0126] A multi-level semantic guided interaction module is used to perform multi-level interactions between the text description of each sub-image and the enhanced image features to obtain calibrated global features;

[0127] The decoder module is used to decode the calibrated global features to obtain the final image description result.

[0128] In this embodiment, the multi-level semantic guidance interaction module includes:

[0129] Multiple interaction branches, each interaction branch includes a linear layer, an inner product operation unit, and a layer normalization unit connected in sequence. Each interaction branch receives a sub-image text description and outputs an interaction feature. In each interaction branch, the sub-image text description is linearly projected through a linear layer to align each result text description with the sub-image text description. The inner product operation unit receives the output of the input linear layer and the enhanced image feature for inner product operation, and then performs layer normalization by the layer normalization unit to obtain features that interact with different semantic levels.

[0130] The integration unit is used to gradually integrate the features interacting with different semantic levels to obtain calibrated global features; the integration unit includes multiple integration sub-units, each integration sub-unit includes a splicing circuit and a linear layer connected in sequence, the input end of the first integration sub-unit is connected to the output end of the first and second interaction branches respectively, the output end of the first integration sub-unit and the output end of the third interaction branch are connected to the input end of the second integration sub-unit respectively, the input end of the next integration sub-unit is connected to the output end of the previous integration sub-unit and the output end of the next interaction branch respectively, and each integration sub-unit is connected to the first integration sub-unit through an integration sub-unit.

[0131] The image description generation device based on multi-level interaction of visual features and text features in this embodiment corresponds one to one with the above-mentioned image description generation method based on multi-level interaction of visual features and text features, and will not be described in detail here.

[0132] This embodiment further provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0133] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing relevant programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0134] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed above with reference to the preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiment that do not depart from the technical solution of the present invention and are based on the technical essence of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for generating image description based on multi-level interaction of visual features and text features, characterized in that the steps include: Acquire an image to be described, extract regional features and grid features of the image from the acquired image to be described, and input the extracted regional features and grid features into a feature enhancement encoder based on a Transformer architecture for feature extraction. The feature enhancement encoder is an encoder based on a Transformer architecture and uses a visual feature enhancement module to replace the multi-head attention layer in the Transformer architecture. The outputs of the regional features and grid features obtained by the feature enhancement encoder are fused to obtain the final enhanced image features. The visual feature enhancement module includes an MLP branch, a self-attention branch, and a fusion module. The MLP branch learns the global scene features in the input features, the attention branch captures the object relationship of the image in the input features, and the fusion module fuses the output results of the MLP branch and the self-attention branch to generate the final enhanced feature output. The obtained image to be described is cropped into multiple sub-images of different sizes, and each sub-image is retrieved to obtain text descriptions of the sub-images of different sizes; Mapping each sub-image text description and the enhanced image features to the same semantic space and performing multi-level interaction to achieve calibration of semantic information and visual features based on multi-layer semantic guidance to obtain a calibrated global feature, wherein when performing multi-level interaction, features of different semantic levels are obtained by calculating each sub-image text description and the enhanced image features, and the features of each semantic level are interactively integrated layer by layer to finally obtain the calibrated global feature; The calibrated global features are decoded to obtain a final image description result.

2. The image description generation method based on multi-level interaction of visual features and text features according to claim 1 is characterized in that: Inputting the extracted regional features and grid features into a feature enhancement encoder based on a Transformer architecture for feature extraction includes: Inputting the extracted regional features and grid features into the visual feature enhancement module respectively to perform visual feature enhancement; The enhanced features obtained by passing the regional features and the grid features through the visual feature enhancement module are respectively passed through the feedforward neural network and the AddNorm module to obtain a feature output sequence.

3. The image description generation method based on multi-level interaction of visual features and text features according to claim 2 is characterized in that: In the MLP branch, the input features are divided into multiple parts along the feature dimension. Each feature part after division is normalized and then input into the linear layer respectively. The outputs of each feature part after passing through the linear layer are connected and output as the result of the MLP branch. The calculation expression is: in, Represents the output of the i-th feature part after passing through the linear layer, H represents the number of input feature partitions, 、 Represents the i-th feature part respectively The corresponding weights and biases, Norm represents the normalization operation, and Concat represents the connection operation. represents the output of the MLP branch.

4. The image description generation method based on multi-level interaction of visual features and text features according to claim 2 is characterized in that: The self-attention branch adopts a single-head self-attention mechanism, and the calculation expression is: in, , , are respectively the learnable mapping matrices, is the output of the self-attention branch, Represents the input features.

5. The image description generation method based on multi-level interaction of visual features and text features according to claim 1 is characterized in that: The step of mapping the text description of each sub-image and the enhanced image features into the same semantic space and performing multi-level interaction includes: Linearly projecting the text descriptions of the sub-images of different sizes so that the resulting text descriptions and the enhanced image feature maps are aligned in the same semantic space; The results of linear projection of the text description of each sub-image are respectively subjected to inner product operation with the enhanced image features, and after layer normalization, features at different semantic levels are obtained; The features of each semantic level are interactively integrated layer by layer to obtain the calibrated global features, wherein the features of the first layer and the second layer are spliced ​​and linearly transformed to obtain integrated features, and the currently obtained integrated features are spliced ​​and linearly transformed with the features of the next layer to obtain new integrated features. This is executed in sequence until the last layer is spliced ​​and linearly transformed to obtain integrated features, and finally all the obtained integrated features are spliced ​​and linearly transformed with the features of the first layer again to obtain the final integrated features as the calibrated global features.

6. The image description generation method based on multi-level interaction of visual features and text features according to claim 5 is characterized in that: When the acquired image to be described is cropped into 1, 5, and 9 sub-images, the text description set is obtained. , where the text description set Including the first text description obtained by dividing the corresponding sub-image into 1 , the second text description obtained by dividing into 5 sub-images And the third text description obtained by dividing into 9 sub-images , the features of each semantic level are calculated according to the following formula: in, , , are respectively the learnable mapping matrices, LN Representation layer normalization, Represents the inner product operation element by element, represents the Relu activation function, 、 、 Respectively represent the first text description , Second text description And the third text description Corresponding to the features of different semantic levels, Represents enhanced image features; The features of each semantic level are concatenated and linearly transformed in sequence according to the following formula: in, Represents the splicing operation, Represents a linear transformation, C1 is and The integrated features obtained after integration, C2 is The integrated features obtained after integration with C1, FM Is to combine C1, C2 and The result after splicing is For the general FM Calibrated global features obtained after the linear layer.

7. The image description generation method based on multi-level interaction of visual features and text features according to any one of claims 1 to 6, characterized in that: The calibrated global features are input into a decoder based on a Transformer model to decode and generate a final image description result, the steps including: Input each word into the masked self-attention module to model the relationship between words, and output the result after masked self-attention of each word; Providing the output of the masked self-attention module and the calibrated global features to a multi-head attention interaction module to model the cross-modal relationship between the calibrated global features and the results of the masked self-attention of each word; The output of the multi-head attention interaction module is sequentially passed through the feedforward neural network, residual and layer normalization to obtain the decoded output sequence; The decoded output sequence is linearly transformed and classified by the Softmax function to generate the final image description result.

8. An image description generation device based on multi-level interaction of visual features and text features, characterized in that: include: An encoding module is used to obtain an image to be described, extract regional features and grid features of the image from the obtained image to be described, and input the extracted regional features and grid features into a feature enhancement encoder based on a Transformer architecture for feature extraction. A visual feature enhancement module is used in the feature enhancement encoder to replace the multi-head attention layer in the Transformer architecture. The outputs of the regional features and grid features obtained by the feature enhancement encoder are fused to obtain the final enhanced image features. The visual feature enhancement module includes an MLP branch, a self-attention branch, and a fusion module. The MLP branch learns the global scene features in the input features, the attention branch captures the object relationship of the image in the input features, and the fusion module fuses the output results of the MLP branch and the self-attention branch to generate the final enhanced feature output; A retrieval module is used to crop the acquired image to be described into multiple sub-images of different sizes, and retrieve each sub-image to obtain text descriptions of the sub-images of different sizes; a multi-level semantic guidance interaction module, configured to map the text descriptions of each sub-image and the enhanced image features to the same semantic space and perform multi-level interaction, thereby calibrating semantic information and visual features based on multi-level semantic guidance to obtain calibrated global features. During the multi-level interaction, features at different semantic levels are calculated by respectively calculating the text descriptions of each sub-image and the enhanced image features, and the features at each semantic level are interactively integrated layer by layer to ultimately obtain the calibrated global features. The decoding module is used to decode the calibrated global features to obtain a final image description result.

9. The image description generation device based on multi-level interaction of visual features and text features according to claim 8, characterized in that: The multi-level semantic guidance interaction module includes: Multiple interaction branches, each of which includes a linear layer, an inner product operation unit, and a layer normalization unit connected in sequence, each of which receives a sub-image text description and outputs an interaction feature, and in each of the interaction branches, linearly projects each sub-image text description through the linear layer so that each resulting text description is aligned with the enhanced image feature, and the inner product operation unit receives the output of the linear layer and the enhanced image feature, performs inner product operation, and then obtains features that interact with different semantic levels after layer normalization by the layer normalization unit; An integration unit is used to gradually integrate the features of interactions at different semantic levels to obtain calibrated global features; the integration unit includes multiple integration sub-units, each integration sub-unit includes a splicing circuit and a linear layer connected in sequence, the input end of the first integration sub-unit is connected to the output ends of the first and second interaction branches respectively, the output end of the first integration sub-unit and the output end of the third interaction branch are connected to the input end of the second integration sub-unit respectively, the input end of the next integration sub-unit is connected to the output end of the previous integration sub-unit and the output end of the next interaction branch respectively, and each integration sub-unit is connected to the first integration sub-unit through an integration sub-unit.

10. A computer device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 7.

Images