Referential expression understanding method and system based on pixel-lexical association modeling

Abstract

The present invention provides a method and system for understanding referential expressions based on pixel-word association modeling. The method comprises the following steps: obtaining a text description and a corresponding original image; extracting text features from the text description; extracting pixel features from the original image using a convolutional encoder; and embedding a feature enhancement module based on pixel-word association after several convolutional layers of the convolutional encoder. The pixel-word association-based feature enhancement module calculates the correlation between visual pixels and text words, embeds the text features into the pixel features, and enhances the pixel features. The enhanced visual features and the original text features are input into a multi-stage cascade decoder, where the features are integrated using a cross-attention mechanism. A feed-forward neural network is then used to output the final detection results. By explicitly calculating the pixel-word correlation between visual features and language features, the method can better focus on language-related image regions, significantly improving the accuracy of object detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and system for understanding referential expressions based on pixel-vocabulary association modeling. Background Art

[0002] As a key research area in human-computer interaction, referential expression understanding has garnered widespread attention in recent years. With the continuous advancement of multimodal reasoning technology, the practical value of this task has become increasingly prominent. The core of referential expression understanding lies in effectively combining natural language descriptions with visual information, locating target objects in images by analyzing text instructions. This not only involves the coordinated processing of language and visual information, but also plays a vital role in downstream tasks such as visual question answering, image captioning, and visual-language navigation.

[0003] In the existing technology, the referential expression understanding task is usually handled based on the target detection framework. The two-stage method relies on pre-generated region candidate boxes or anchor boxes, but it often appears to be powerless when faced with complex text or flexible targets. In addition, the one-stage method extracts features through independent visual and text encoders, and then fuses and predicts them in a multimodal decoder. Although these methods improve processing efficiency, due to the relative independence of visual and text features, they may face overfitting problems when locating multiple objects with different text descriptions in the same image, resulting in unsatisfactory detection performance. With the deepening of research on the referential expression understanding task, the academic community has gradually realized the limitations of relying solely on independent branches to process visual and text features.

[0004] For referential expression comprehension tasks, existing methods often adopt a two-branch architecture, using independent visual encoders and text encoders to extract modality-specific features. Therefore, during training, the visual encoder relies on prior knowledge to extract potential foreground features. However, since the same image is often associated with multiple objects related to unique textual expressions, the independent visual feature extraction process limits the capabilities of the visual encoder. It can only train broad foreground features and lacks text-specific visual features. As a result, the visual encoder struggles to focus on the areas most relevant to the text description, resulting in significant differences between the extracted visual features and those required for multimodal reasoning. Consequently, independent visual encoders cannot fully meet the requirements of referential expression comprehension tasks.

[0005] Another category addresses the challenges through multi-language guided visual encoder structures. These methods can be roughly divided into two categories: one is feature-based methods, which directly process intermediate visual features based on language information, or incorporate language information into visual representations through cross-attention mechanisms; the other is structure-based methods, which dynamically modify the parameters or architecture of the visual encoder based on language features to extract language-related visual features. Although these methods have achieved significant performance improvements in visual feature enhancement, most methods still require complex designs, such as query-aware attention modules, dynamic weight generators, etc., which increase the difficulty of model training. In addition, the above methods fail to explicitly align the visual and linguistic features of local pixels and text words, which may lead to the neglect of descriptive vocabulary for specific target objects, thereby indiscriminately highlighting other image regions related to similar semantic concepts. Summary of the Invention

[0006] The present invention proposes a referential expression understanding method and system based on pixel-lexical association modeling to solve the technical problems that the existing technology lacks specific visual features for text or fails to clearly align the visual and linguistic features of local pixels with text words, resulting in difficulty in highlighting image areas related to specific semantics during recognition.

[0007] To solve the above technical problems, the present invention provides a method for understanding referential expressions based on pixel-word association modeling, comprising the following steps:

[0008] Step S1: Obtain text description and corresponding original image;

[0009] Step S2: extracting text features of the text description; extracting pixel features of the original image through a convolutional encoder, and embedding a feature enhancement module based on pixel-word association after several convolutional layers of the convolutional encoder; the feature enhancement module based on pixel-word association calculates the correlation between visual pixels and text words, and embeds the text features into the pixel features to enhance the pixel features;

[0010] Step S3: The enhanced visual features and original text features are input into a multi-stage cascade decoder, the features are integrated through a cross-attention mechanism, and a feed-forward neural network is used to output the final detection results.

[0011] Preferably, in step S2, the method for calculating the correlation between visual pixels and text words by the feature enhancement module based on pixel-word association includes: mapping pixel features and text features to a unified dimensional space for alignment through linear mapping, and calculating the association at the pixel-word level; generating word-level attention weights for different words in the input expression; and performing an operation based on attention pooling to convert the pixel-word level association into a pixel-sentence level correlation through the word-level attention weights.

[0012] Preferably, the word-level attention weight The expression is:

[0013] ;

[0014] Where, represents the nonlinear projection function; Indicates the first i vocabulary; N l Indicates the number of words.

[0015] Preferably, the expression of the pixel-sentence level correlation is:

[0016] ;

[0017] In the formula, represents the The association of words to pixels; is the sigmoid function.

[0018] Preferably, in step S2, a ResNet network is used to extract pixel features.

[0019] Preferably, a feature enhancement module is embedded after the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer of the ResNet network.

[0020] Preferably, a Transformer encoder is stacked after the ResNet network.

[0021] Preferably, a feature enhancement module is embedded before the linear projection of the Transformer encoder.

[0022] The present invention also provides a referential expression understanding system based on pixel-word association modeling, which is applicable to the above-mentioned referential expression understanding method based on pixel-word association modeling, and includes a text feature extraction module, a pixel feature extraction module, a pixel-word association enhancement module and a decoding module;

[0023] The text feature extraction module is used to extract text features;

[0024] The pixel feature extraction module is used to extract pixel features;

[0025] The pixel-word association enhancement module is embedded after several convolutional layers in the pixel feature extraction module to obtain the correlation between pixels and text words, and embed the text features into the pixel features based on the correlation for enhancement;

[0026] The decoding module is used to integrate the enhanced visual features and original text features through a cross-attention mechanism, and output the final detection results using a forward neural network.

[0027] Preferably, the cross-layer regularization loss For each pixel-word association enhancement module generated Apply distribution constraints for training: the true labels Convert to a binary mask of the same size as the input image , where the bounding box , that is, the pixel value in the true label is set to 1, and the rest of the pixel values are set to 0; adjust the mask The size of , so that it corresponds to different pixel-word association enhancement modules Maintain the same spatial resolution; finally, perform spatial normalization to obtain the flattened and , mapped into a probability distribution through the Softmax function and Then, define the loss for and The KL divergence between is calculated as follows:

[0028] ;

[0029] ;

[0030] ;

[0031] Where, and Respectively and The elements, is the temperature parameter that controls the distribution shape in the Softmax function, represents the total number of feature enhancement modules, It is the product of width and height in the spatial resolution of the feature map.

[0032] The beneficial effects of the present invention include at least: the present invention proposes a feature enhancement module based on pixel-word association, which improves the language guidance effect in the visual feature extraction process by explicitly calculating the pixel-word correlation between visual features and language features, and can better focus on image areas related to language, thereby significantly improving the accuracy of target detection.

[0033] The present invention optimizes the text parameter update design of the convolution module by integrating the feature enhancement module into the existing pixel feature extraction network, namely the convolutional encoder. This integration method not only reduces the number of parameters, but also avoids the complexity of designing additional modules for different architectures. It can effectively focus on the text reference area, thereby significantly improving the detection effect of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the method flow of Example 1 of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of the feature enhancement module of Example 1 of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of a multi-stage cascade decoder according to embodiment 1 of the present invention;

[0037] Figure 4 Schematic diagram of the detection effect of the method of Example 1 of the present invention and the existing method;

[0038] Figure 5 This is a schematic diagram of the improved structure of the ResNet network according to Example 2 of the present invention;

[0039] Figure 6 This is a schematic diagram of the improved structure of the Transformer network according to Example 3 of the present invention;

[0040] Figure 7 This is a schematic diagram of the overall network structure of Example 3 of the present invention. DETAILED DESCRIPTION

[0041] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0042] Example 1

[0043] like Figure 1As shown, an embodiment of the present invention provides a method for understanding referential expressions based on pixel-vocabulary association modeling, comprising the following steps:

[0044] Step S1: Get the text description and the corresponding original image.

[0045] Specifically, in the embodiment of the present invention, a text description and its corresponding original image are obtained, and a pre-processed text-image pair is formed through data enhancement or data normalization.

[0046] First, basic data augmentation or normalization is performed to generate preprocessed image and text pairs.

[0047] The image is a 3-channel color image obtained by a common image acquisition device. It refers to the expression text used to describe the object to be detected, including the object's attributes, location, and category. The target box is used to represent the coordinates of the detected object in the image, which is expressed as ,in represents the coordinates of the center of the object, and Represents the width and height of the object respectively.

[0048] For image data augmentation, commonly used techniques include random scaling, cropping, and flipping. Random scaling can fluctuate within a 640×640 pixel range, with sizes ranging from 480, 576, and 640. It is recommended that the probability of random cropping be kept below 0.5. This example primarily involves marking the beginning and end of text with special word vectors, as well as cropping and completing the text to ensure a fixed length for the text sequence.

[0049] Step S2: extract text features of the text description; extract pixel features of the original image through a convolutional encoder, and embed a feature enhancement module based on pixel-word association after several convolutional layers of the convolutional encoder; the feature enhancement module based on pixel-word association calculates the correlation between visual pixels and text vocabulary, and embeds text features into pixel features to enhance the pixel features.

[0050] In this embodiment, the input text is encoded into discrete word vectors through a pre-trained word segmenter and input into a text encoder to extract text features.

[0051] Specifically, the word segmentation tool is used to segment the text and convert it into word vectors for subsequent calculation and analysis; specified marker word vectors are added at the beginning and end of the text to mark the start and end of the text expression; by truncating the text and padding it with specified word vectors, the length of all text inputs is ensured to be consistent, thereby achieving standardization of text expressions of different lengths.

[0052] Text encoders typically use a pretrained text encoding model, such as BERT or CLIP. These models typically consist of several Transformer encoding layers and are pretrained using self-supervision or masked data modeling to extract and integrate global information from the text. To enable subsequent multimodal fusion and input to the decoder for prediction, a linear layer is typically used to reduce the dimensionality of the BERT output to align the dimensionality of the text features with the visual features. This is not discussed in detail here.

[0053] The processed image is input into a convolutional encoder, which extracts visual features layer by layer. This convolutional encoder is typically a convolutional neural network. Common convolutional encoders include ResNet and ConvNext, but are not limited to these in this embodiment. These networks effectively extract visual features from images through a multi-layer structure.

[0054] In the process of extracting visual features, in order to accurately locate the target object referred to by the natural language description in the image, the extracted visual features should focus on the local area related to the text description. To this end, this embodiment provides a feature enhancement module based on pixel-word association, the structure of which is as follows: Figure 2 As shown, by setting the feature enhancement module after any convolution layer of the convolution encoder, the pixel features extracted by the convolution layer are processed, and the correlation map between the visual features and the text features is calculated to enhance the visual features. The specific operations are as follows.

[0055] Given the flattened visual features and the text features of the input sentence ,in Indicates the height of the image. Indicates width, The number of channels representing visual features, Indicates the length of the sentence, The number of channels representing text features. First, the visual features and language features are mapped to a unified dimensional space through linear mapping to align them and calculate the pixel-word level association. The specific form is:

[0056] ;

[0057] in, and are the linear projection matrices of visual and language features respectively, is the unified dimension after projection, U It represents the pixel-word level association between visual features and text features. It is shared in the feature enhancement modules based on pixel-word association in different layers to ensure that text features are projected into the same feature space.

[0058] Use attention-based pooling to generate weights to obtain text for visual features pixel-level correlation information.

[0059] Considering that different words in a text representation have different contributions to the target they refer to, this embodiment uses the following method to generate word-level attention weights for different words in the input expression: , to express the importance of each word:

[0060] ;

[0061] in, Represents a nonlinear projection function that transforms language features The Words are mapped to corresponding attention weights . This function consists of two linear layers and a Non-linear activation layer.

[0062] Through pixel-word level association and attention weight ,The calculation process of calculating the pixel-sentence level association information is as follows:

[0063] ;

[0064] Here, express No. i List, Is the sigmoid function, responsible for normalizing each element value to Range. Calculation results Aggregates the correlation between each pixel and all words. Therefore, this embodiment will is considered as a spatial weight mask and applied to to enhance features.

[0065] The final feature calibration is expressed as:

[0066] ;

[0067] in represents a broadcast element-wise multiplication operation so that the calibrated features More attention is paid to language-related pixels, thereby improving the effectiveness of the feature enhancement module in visual localization tasks.

[0068] Step S3: The enhanced visual features and original text features are input into a multi-stage cascade decoder, the features are integrated through a cross-attention mechanism, and a forward neural network is used to output the final detection results, i.e., the predicted target box coordinates.

[0069] Specifically, this embodiment adopts a multi-stage cascade decoder, whose structure is as follows: Figure 3 As shown in Figure 2, the decoder uses a learnable token and a cross-attention mechanism to iteratively aggregate and extract useful information from visual and textual features. The design of the cascaded decoder eliminates the computational overhead required for further feature enhancement, thereby improving the efficiency of the overall framework.

[0070] The cross-attention mechanism specifically uses the learnable token as the query and cross-attention with the text and visual features to aggregate target-related features. The forward neural network used to output the results refers to a fully connected network with several layers stacked. The number of channels in the hidden layer can be set as needed, usually set to 256. The number of output channels in the output layer is 4, corresponding to the relative position of the final target coordinates in the image. .

[0071] like Figure 4 As shown in FIG, a schematic diagram comparing the detection results of the method of this embodiment and the existing method, wherein the red box is the true label and the green box is the prediction result. It can be seen that compared with the existing method, the method of this embodiment has achieved better prediction results in different scenarios. Even if different methods have located the location of the object, the method of this embodiment still achieves more accurate detection results, such as the prediction of barbecue in the second row in the figure.

[0072] Example 2

[0073] In this embodiment, based on the first embodiment, the convolutional encoder in step S2 is replaced by the ResNet network, and the third, fourth and fifth convolutional layers of the ResNet network are further selected, and a feature enhancement module is added. The structure is as follows: Figure 5 Specifically expressed as:

[0074] ;

[0075] Afterwards, the enhanced features Will be adjusted to The size of , and input to the next layer. In the feature maps of different layers, 、 and The value of may vary.

[0076] This operation aims to highlight language-related visual information and ensure that the enhanced features match the dimensions of subsequent layers.

[0077] Example 3

[0078] This embodiment is based on the second embodiment. After adding the ResNet network, the Transformer encoder is stacked, and a feature enhancement module is added before the linear projection of the query and key of the Transformer encoder. The structure is as follows: Figure 6 As shown, the calculation process is as follows:

[0079] ;

[0080] ;

[0081] in, represents a multi-head self-attention layer, It is the abbreviation of query vector query, key vector key and value vector value in the attention layer input, and the enhanced output It will be passed to the subsequent normalization layer and feedforward network.

[0082] By adjusting the attention score, regions associated with text are enhanced. This design ensures that the enhanced output more effectively reflects the relationship between image and text, and is passed to subsequent layer normalization and feedforward networks. The integration of feature enhancement modules based on pixel-word association not only improves the feature expression capabilities of each layer, but also enhances the model's ability to understand and process image and text information.

[0083] The overall network structure after integrating Examples 1 to 3 is as follows Figure 7 It is shown, but not intended to limit the present invention.

[0084] Example 4

[0085] The present invention also provides a referential expression understanding system based on pixel-word association modeling, which is applicable to the above-mentioned referential expression understanding method based on pixel-word association modeling, and includes a text feature extraction module, a pixel feature extraction module, a pixel-word association enhancement module and a decoding module;

[0086] Text feature extraction module, used to extract text features;

[0087] Pixel feature extraction module, used to extract pixel features;

[0088] The pixel-word association enhancement module is embedded after several convolutional layers in the pixel feature extraction module. It is used to obtain the correlation between pixels and text words and embed text features into pixel features based on the correlation for enhancement.

[0089] The decoding module is used to integrate the enhanced visual features and original text features through the cross-attention mechanism, and output the final detection results using a forward neural network.

[0090] This embodiment trains the system by minimizing the gap between the predicted box and the labeled box. At this stage, the model is trained by minimizing the IoU loss and L1 loss between the predicted box and the labeled box, and an additional cross-layer consistency loss is used. To constrain the pixel-word association distribution of the feature enhancement module.

[0091] First, the coordinate box constraint aims to directly compare the difference between the predicted bounding box and the true bounding box. This process is achieved by calculating the difference between the coordinate position, width and height of the model output and the true annotation, thereby ensuring that the model can accurately locate the target object. Second, cross-layer consistency aims to ensure that the visual feature matrices generated by different decoder layers are consistent. Maintain consistency when processing the same input. Since the visual features extracted by different encoder layers There are significant differences in receptive field and information extraction, and the returned matrix To solve this problem, a loss function is introduced for cross-layer consistency. , by imposing distribution constraints on the matrix returned by each layer to ensure consistency when processing similar spatial regions. Combining these two constraints, the model can not only improve the accuracy of predictions, but also maintain consistent attention to the target area at different levels, thereby achieving better image understanding and object detection capabilities.

[0092] Specifically, the true label Convert to a binary mask of the same size as the input image , where the true label Set the pixel values inside to 1 and the rest to 0; adjust the mask The size of , so that it corresponds to different pixel-word association enhancement modules Maintain the same spatial resolution; finally, perform spatial normalization to obtain the flattened and , mapped into a probability distribution through the Softmax function and Then, define the loss for and The KL divergence between is calculated as follows:

[0093] ;

[0094] ;

[0095] ;

[0096] in, and Represents vectors and The elements, It is the temperature parameter that controls the distribution shape in the Softmax function and is generally set to 0.2. represents the total number of feature enhancement modules based on pixel-word association, It is the product of width and height in the spatial resolution of the feature map. The process will minimize and The distribution difference between The large values are concentrated in designated target area. This process ensures The area cited by the text can be effectively enhanced, and through the above operations, the ground truth value of the annotation The information in can also be obtained through binary masking Efficient transfer to shallow layers of the visual encoder.

[0097] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. Only preferred embodiments of the present invention are presented. While the description is relatively specific and detailed, it should not be construed as limiting the scope of the present invention. As long as there are no conflicts in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] It should be noted that, for those skilled in the art, various modifications and improvements can be made without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for understanding referential expressions based on pixel-lexical association modeling, characterized by: The following steps are involved: Step S1: Obtain text description and corresponding original image; Step S2: extracting text features of the text description; extracting pixel features of the original image through a convolutional encoder, and embedding a feature enhancement module based on pixel-word association after several convolutional layers of the convolutional encoder; the feature enhancement module based on pixel-word association calculates the correlation between visual pixels and text words, and embeds the text features into the pixel features to enhance the pixel features; Step S3: The enhanced visual features and original text features are input into a multi-stage cascade decoder, the features are integrated through a cross-attention mechanism, and the final detection results are output using a forward neural network; In step S2, the method for calculating the correlation between visual pixels and text words by the feature enhancement module based on pixel-word association includes: mapping pixel features and text features to a unified dimensional space for alignment through linear mapping, and calculating the association at the pixel-word level; generating word-level attention weights for different words in the input expression; and performing an attention pooling operation to convert the pixel-word level association into a pixel-sentence level correlation through the word-level attention weights.

2. The method for understanding referential expressions based on pixel-word association modeling according to claim 1, characterized in that: The word-level attention weights The expression is: ； Where, represents the nonlinear projection function; Indicates the first i vocabulary; N l Indicates the number of words.

3. The method for understanding referential expressions based on pixel-word association modeling according to claim 2, characterized in that: The expression of the pixel-sentence level correlation is: ； Where, Indicates the The association of words to pixels; is the sigmoid function; Indicates the number of words in a text expression.

4. The method for understanding referential expressions based on pixel-word association modeling according to claim 1, wherein: In step S2, the ResNet network is used to extract pixel features.

5. The method for understanding referential expressions based on pixel-word association modeling according to claim 4, characterized in that: A feature enhancement module is embedded after the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer of the ResNet network.

6. The method for understanding referential expressions based on pixel-word association modeling according to claim 5, characterized in that: A Transformer encoder is stacked after the ResNet network.

7. The method for understanding referential expressions based on pixel-word association modeling according to claim 6, characterized in that: A feature enhancement module is embedded before the linear projection of the Transformer encoder.

8. A referential expression understanding system based on pixel-word association modeling, applicable to the referential expression understanding method based on pixel-word association modeling according to any one of claims 1 to 7, characterized in that: It includes text feature extraction module, pixel feature extraction module, pixel-word association enhancement module and decoding module; The text feature extraction module is used to extract text features; The pixel feature extraction module is used to extract pixel features; The pixel-word association enhancement module is embedded after several convolutional layers in the pixel feature extraction module to obtain the correlation between pixels and text words, and embed the text features into the pixel features based on the correlation for enhancement; The decoding module is used to integrate the enhanced visual features and original text features through a cross-attention mechanism, and output the final detection results using a forward neural network.

9. The referential expression understanding system based on pixel-word association modeling according to claim 8, characterized in that: Through cross-layer regularization loss For each pixel-word association enhancement module generated Apply distribution constraints for training: the true labels Convert to a binary mask of the same size as the input image , where the true label Set the pixel values inside to 1 and the rest to 0; adjust the mask The size of , so that it corresponds to different pixel-word association enhancement modules Maintain the same spatial resolution; finally, perform spatial normalization to obtain the flattened and , mapped into a probability distribution through the Softmax function and Then, define the loss for and The KL divergence between is calculated as follows: ； ； ； ； Where, and Respectively and The elements, is the temperature parameter that controls the distribution shape in the Softmax function, represents the total number of pixel-word association-based feature enhancement modules in the embodiment, is the product of width and height in the spatial resolution of the feature map, Represents the coordinates of the center of the object, w and h represent the width and height of the object respectively, Indicates the The association of words to pixels; is the sigmoid function; Indicates the number of words in the text expression, Represents the word-level attention weight.

Images