Referential expression understanding method based on multi-scale cross-modal feature fusion

Abstract

This invention relates to a method for understanding referential expressions based on multi-scale cross-modal feature fusion, drawing on the field of language-image multimodal fusion. The cross-modal feature fusion process combines linear feature modulation and a visually guided language attention module for feature fusion. The model also utilizes language to select and filter multi-scale grid-level features, adaptively selecting key cues from low-level and high-level fused features, and establishing relationships between cross-scale information through dynamic routing. Experimental results demonstrate that the new model architecture achieves new state-of-the-art performance across multiple benchmarks and provides new insights and directions for REC research.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal fusion based on language and image, and specifically relates to a method for understanding referential expressions based on multi-scale cross-modal feature fusion. Background Art

[0002] Referring expression comprehension (REC) is a challenging and important task in the field of computer vision. This task requires the computer to reason and analyze the target area corresponding to the description based on a given image and natural language description. It is a fundamental task in many fields such as human-computer interaction, visual question answering, and image retrieval. Unlike traditional object detection that uses predefined category labels to classify fixed objects, REC involves a comprehensive understanding of complex natural language and variable images. The research field of REC mainly focuses on fusion strategies, fusion stages, and detection head design. However, although REC is a cross-modal task, it is still closely related to object detection.

[0003] Modern object detection models can be roughly converted into three levels: network backbone, neck and detection head. For the neck part, experience with detection models has shown that assembling a well-matched feature pyramid network (FPN) is important for bridging the semantic gap and improving performance. However, on the one hand, the current REC models usually simply use an averaging strategy to fuse multi-scale features, or even just use single-scale features, and the research on the fusion of multi-scale features is far from enough. On the other hand, since low-level features contain more attribute information such as color, texture, etc., and high-level features have rich semantic information, multi-scale feature fusion can combine the advantages of low-level features and high-level features to be suitable for different language expressions. The present invention proposes a multi-scale fusion method suitable for REC tasks to help the model reason and accurately locate targets of different sizes. Summary of the Invention

[0004] Technical problems to be solved

[0005] To overcome the shortcomings of existing techniques, the present invention provides a method for understanding referential expressions based on multi-scale cross-modal feature fusion. This method dynamically allocates and selects fine-grained information from multi-scale feature maps using language gates and union gates. This method is a more efficient and reliable method for understanding referential expressions.

[0006] Technical Solution

[0007] A method for understanding referential expressions based on multi-scale cross-modal feature fusion, characterized by the following steps:

[0008] Step 1: The image is first resized to the same size, then Resnet-101 is used to extract feature maps of n scales, and then mapped to the same dimension d through 1×1 convolution to obtain For language information, we first decompose the language information into words, and then get the feature vectors corresponding to each word through word embedding. The longest sentence is set to have T words, and the blanks of sentences with less than T words are filled with PAD markers; the CLS marker is added at the beginning of the sentence and the SEP marker is added at the end of the sentence; the word vector after position encoding is input into the BERT network to get the feature vectors of each word that integrates the sentence information.

[0009] Step 2: Input E and V into the cross-modal interaction attention module of the model, which consists of two parts: the linear feature modulation module FiLM and the visually guided language attention module; in the FiLM module, a feature-based affine transformation is applied to adaptively affect the output of the network. For a given language feature First, obtain the entire expression E through the average strategy F , and then specifically through:

[0010]

[0011]

[0012]

[0013] Where W i γ , W i β , and are the weights and biases of two multilayer perceptrons MLP with activation function Tanh. For formula (3), ⊙ and Represents element-wise multiplication and addition respectively; finally, standard 3×3 convolution and ReLU operations are applied to produce multi-level fusion features

[0014] For the vision-guided language attention module, the visual features Vi are first flattened into where N i =H i ×W i is the number of visual features, and then according to the language feature E and visual feature Z i Calculated by the following formula:

[0015]

[0016]

[0017] Where W i Q 、 and is the embedding matrix, where the query, key, and value in this attention module are denoted by Q, K, and V respectively; m is the number of attention heads, d is the feature dimension, and for simplicity, only one language attention module is used for each level of visual features; then, A i It is further encoded by two feed-forward networks FFN with residual connections to form a fused output

[0018] By connecting F f and F t Get F ft , and then use three 1×1 convolutional layers to transform F ft Mapped to dimension d; finally, the combined features are obtained

[0019] Step 3: Build a language-guided feature pyramid module FPN

[0020] First, we construct a routing space of depth K, in which the scaling factor between adjacent stages is constrained to 2. For each routing node, the input consists of two parts: a multi-level feature map and a language vector based on an attention mechanism. The grid-level features at each scale in each routing node are hard-selected by a language gate.

[0021] First, the input through the language gate consists of two parts: a multi-level feature map and a language vector based on the attention mechanism. The language vector based on the attention mechanism is obtained by the following formula:

[0022] a k =softmax(EW k ) (5)

[0023]

[0024] Where W k ∈R 256×1 is the learning weight, k represents the depth, will be shared to each scale and grid feature; the multi-scale feature map can be expressed as Where i is the i-th scale, k is the k-th layer, N = (H i ×W i ); Language gate through Dynamic selection of language vectors The grid-level features in , the specific operations are as follows:

[0025]

[0026]

[0027] * and · represent convolution operations and Hadamard products, respectively; conv(·) represents a 3×3 convolutional network, and σ(·) is the activation function; tanh, i.e., max(0, tahn(·)), is used as the gate switch; when the input is negative, the output of this function is always 0, which makes it unnecessary to have an additional threshold in the inference stage;

[0028] After that, the output will be Y i，k Upsampling, keeping unchanged and downsampling operations are performed from small scale to large scale respectively; the specific operations are as follows:

[0029]

[0030] use To represent the aggregate output in routing node I, The fine-grained features in are further refined by the union gate. Specifically, It will be calculated as follows:

[0031]

[0032]

[0033] Where conv is a 1×1 convolutional network that maps the input features into a channel, l represents the lth node; the nodes in the last layer will be used for multi-scale fusion, and the fusion method is as follows:

[0034]

[0035] Formula (11) aggregates the information of different nodes to obtain F AVG , which will be used as the input of the detection head;

[0036] Step 4: Use anchor-free detection head to locate the object

[0037] For the output F of step 3 AVG , first use a 1×1 convolution layer to get a feature map of shape w×h×5, representing the five predicted values {t x , t y , t w , t h t}, where the first two values represent the center offset, t w and t h Represent the normalized width and height respectively; the last t is the confidence score, indicating whether there is a center point of the object at that location; finally, the cross entropy loss L is applied to the center point t cls， Apply MSE loss L on center offset, width and heightoff ; At the same time, GIoU loss is used as auxiliary loss; finally, the entire function is defined as:

[0038]

[0039] L off =(Δx-t x ) 2 +(Δ y -t y ) 2 (14)

[0040] C ij =1 or 0 indicates whether the current grid contains the center point of the true value target, Indicates the offset of the center point from the center of the grid, where x and y refer to int(·) indicates that the operation rounds the score to the nearest integer; only the grid where the true value center is located is L off ; The total loss function is as follows:

[0041] Loss = L cls +λ off L off +L giou (15)

[0042] Among them, λ off Set to 5, the network selects the center point with the highest score to generate the bounding box; IoU is the metric used in REC to measure the degree of overlap between the prediction and the true situation.

[0043] A computer system, characterized in that it includes: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0044] A computer-readable storage medium is characterized by storing computer-executable instructions, which are used to implement the above method when executed.

[0045] Beneficial effects

[0046] The present invention provides a referential expression understanding method based on multi-scale cross-modal feature fusion, which uses an innovative and effective multi-scale cross-modal fusion to perform referential expression understanding. Unlike previous models, this model combines linear feature modulation and visually guided language attention modules for feature fusion in the cross-modal feature fusion process. At the same time, the model uses language to select and filter multi-scale grid-level features, adaptively selects key clues from low-level and high-level fused features, and establishes relationships between cross-scale information through dynamic routing. Experimental results show that the new model architecture has reached a new state-of-the-art level in multiple benchmark tests and provides new insights and directions for REC research. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0048] Figure 1 This is a schematic diagram of the network structure for understanding referential expression in the first stage of the method of the present invention.

[0049] Figure 2 Results of referential expression understanding based on multi-scale cross-modal feature fusion mechanism. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0051] The technical solution of the present invention includes the following modules: the first part is a module for extracting and encoding language and image information, the second part is a cross-modal feature fusion module based on linear feature modulation (FiLM) and language-guided visual attention mechanism, the third part is a multi-scale feature fusion module based on language guidance, and the fourth part is the target positioning process. In the first part, the Resnet-101 convolutional neural network and the BERT pre-training model are used to extract features of image information and language information respectively. In the second part, FiLM and language-guided visual attention are used to fuse cross-modal features respectively and then map the two to obtain the final fused features. In the third part, features of different scales are selected for fusion by using language gates and joint gates in the language-guided multi-scale feature fusion module to execute a dynamic routing strategy. Finally, the fused multi-scale features are sent to the prediction head to obtain the target object area.

[0052] Based on the above modules, the embodiment of the present invention provides a one-stage candidate-free referential expression method that combines a cross-modal attention mechanism and language-guided multi-scale fusion. The specific process is as follows:

[0053] Step 1: The image is first resized to the same size, then Resnet-101 is used to extract feature maps of n scales, and then mapped to the same dimension d through 1×1 convolution to obtain For language information, we first decompose the language information into words, and then get the feature vectors corresponding to each word through word embedding. The longest sentence is set to T words, and the blank spaces of sentences with less than T words are filled with PAD markers. The CLS marker is added at the beginning of the sentence, and the SEP marker is added at the end of the sentence. The word vector after position encoding is input into the BERT network to get the feature vectors of each word that integrates the sentence information.

[0054] Step 2: Input E and V into the cross-modal interaction attention module of the model. This module consists of two parts, the linear feature modulation module (FiLM) and the visually guided language attention module. In the FiLM module, a feature-based affine transformation is applied to adaptively influence the output of the network. For a given language feature First, we use a simple averaging strategy to obtain the entire expression E F , and then specifically through:

[0055]

[0056]

[0057]

[0058] Where W i γ , W i β , and are the weights and biases of two multilayer perceptrons (MLPs) with activation function Tanh. For formula (3). ⊙ and Represents element-wise multiplication and addition respectively. Finally, standard 3×3 convolution and ReLU operations are applied to produce multi-level fusion features.

[0059] For the vision-guided language attention module, the visual features Vi are first flattened into where N i =H i ×W i is the number of features of the visual mark. Then according to the language feature E and visual feature Z i Calculated by the following formula:

[0060]

[0061]

[0062] Where W i Q 、 and is the embedding matrix. The query, key, and value in this attention module are denoted by Q, K, and V respectively. m is the number of attention heads, and d is the feature dimension. For simplicity, only one language attention module is used for each level of visual features. i It is further encoded by two feed-forward networks (FFN) with residual connections to form a fused output

[0063] By connecting F f and F t Get F ft , and then use three 1×1 convolutional layers to transform F ft Mapped to dimension d. Finally, the combined features are obtained

[0064] Step 3: Build a language-guided feature pyramid module (FPN).

[0065] First, a routing space of depth K is constructed. In this routing space, the scaling factor between adjacent stages is constrained to 2. For each routing node, the input consists of two parts: a multi-level feature map and a language vector based on an attention mechanism. The grid-level features at each scale in each routing node are hard-selected by a language gate. In addition, since REC is a cross-modal task, a data-dependent joint gate module is established in the node to further refine the grid based on the aggregated information.

[0066] First, the input through the language gate consists of two parts: a multi-level feature map and a language vector based on the attention mechanism. The language vector based on the attention mechanism is obtained by the following formula:

[0067] a k =softmax(EW k ) (5)

[0068]

[0069] Where W k ∈R 256×1 is the learning weight, k represents the depth, will be shared to each scale and grid feature. The multi-scale feature map can be expressed as Where i is the i-th scale, k is the k-th layer, N = (H i ×W i ). Language gate passed Dynamic selection of language vectors The grid-level features in , the specific operations are as follows:

[0070]

[0071]

[0072] * and · denote convolution and Hadamard product operations, respectively. conv(·) represents a 3×3 convolutional network, and σ(·) is the activation function. Tanh (max(0, tahn(·))) is used as the gate. When the input is negative, the output of this function is always 0, eliminating the need for additional thresholds during inference.

[0073] After that, the output will be Y i，k Perform upsampling, keeping unchanged, and downsampling operations from small scale to large scale. The specific operations are as follows:

[0074]

[0075] use To represent the aggregate output in routing node I, in order to improve the effectiveness of the deep network of this module, a bottleneck module with residual connection is used. In this module, The fine-grained features in are further refined by the union gate. In particular, It will be calculated as follows:

[0076]

[0077]

[0078] Where conv is a 1×1 convolutional network that maps the input features into a channel, and l represents the lth node. The nodes in the last layer will be used for multi-scale fusion. The fusion method is as follows:

[0079]

[0080] Formula (11) aggregates the information of different nodes to obtain F AVG , which will be used as the input of the detection head.

[0081] Step 4: Use the anchor-free detection head to locate the target. For the output F of step 3 AVG , first use a 1×1 convolution layer to get a feature map of shape w×h×5, representing the five predicted values {tx , t y , t w , t h t}, where the first two values represent the center offset, t w and t h Represent the normalized width and height respectively. The last t is the confidence score, indicating whether the center point of the object exists at that location. Finally, the cross entropy loss L is applied to the center point t cls , apply MSE loss L on center offset, width and height off At the same time, GIoU loss is used as auxiliary loss. Finally, the entire function is defined as:

[0082]

[0083] L off =(Δx-t x ) 2 +(Δ y -t y ) 2 (14)

[0084] C ij =1 or 0 indicates whether the current grid contains the center point of the true value target. Indicates the offset of the center point from the center of the grid, where x and y refer to int(·) indicates that the operation rounds the fraction to the nearest integer. Only the grid where the true value center is located is L off The total loss function is as follows:

[0085] Loss = L cls +λ off L off +L giou (15)

[0086] Among them, λ off When set to 5, the network selects the center point with the highest score to generate the bounding box. Intersection-over-Union (IoU) is a metric used in REC to measure the overlap between the prediction and the ground truth. Following previous work, we use IoU@0.5 to measure prediction accuracy.

[0087] Example 1:

[0088] 1. Image feature extraction

[0089] Given an image of a natural scene, the entire image is resized to 640×640 and fed into the feature extraction network for forward propagation. This example uses ResNet-101 to extract image features. Three feature maps at scales of 20×20×2048, 40×40×1024, and 80×80×512 are obtained. Then, using 1×1 convolution, these three feature maps are mapped to the same dimension d=256.

[0090] 2. Extraction of language features

[0091] The sentence information is decomposed into words, and the feature vectors corresponding to each word are obtained through word embedding. This embodiment stipulates that the longest sentence has 20 words. Then the word vector after position encoding is input into the BERT network to obtain the feature vector E of each word in the fused sentence information, E∈R 20×256 .

[0092] 3. Feature Fusion Using Cross-Modal Attention

[0093] The image features are expanded into vectors of (400×256), (1600×256), and (6400×256) dimensions, and fed into the cross-modal fusion module along with the language features (20×256). In the language-guided visual attention module, the language features serve as the input to Q, and the image features serve as the input to K and V. Each module consists of two identical self-attention layers, with a total of eight attention heads. In FiLM, an affine transformation is performed on the given language features to obtain the fused features. Finally, the final fused features are obtained by concatenating the two fused features and remapping them to dimension d using a 1×1 convolution.

[0094] 4. Multi-scale fusion based on language guidance

[0095] The three-scale feature maps and language features fused in the previous stage are input into the routing nodes. The feature map grid is first filtered using the language through a language gate, followed by upsampling and downsampling. The original resolution, upsampled, and downsampled feature maps are then fed into a joint gate for information aggregation. A bottleneck module with residual connections is used between each layer of routing nodes to ensure the effectiveness of the deep network. This network consists of six layers. Finally, in the final layer of routing nodes, the feature maps at the three scales are averaged to obtain the final fused features.

[0096] 5. Target Positioning

[0097] For the output of the previous stage, a 1×1 convolution layer is first used to obtain a w×h×5 feature map, which represents the five predicted values {t x , t y , t w , t h t}, where the first two values represent the center offset, tw and t h denotes the normalized weight and height, respectively. The last t is a confidence score, indicating whether the center point of the object exists at that location. During inference, the network selects the center point with the highest score to generate the bounding box.

[0098] 6. Model training

[0099] The entire training process is end-to-end. This example uses four training sets: RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame, as metrics for model training and evaluation. The Adam method is used as the optimizer, with a batch size of 8 and an initial learning rate of 1e-4. This example trains the model for 20 rounds on a 1080Ti GPU, with the learning rate halved after every 10 rounds.

[0100] 8. Model Application

[0101] After the above training process, multiple models can be obtained. The best model (the one with the best test effect on the test set) is selected for application. For the input images and sentences, it is only necessary to adjust the image to 640×640 size and normalize it, and perform word segmentation on the sentence before it can be used as the input of the model. The parameters of the entire network model are fixed. It only needs to input the image data and language data and propagate forward. The image and language feature vectors V and E are obtained in turn, and then automatically passed into the cross-modal feature module, the language-guided multi-scale fusion module and the positioning module to directly obtain the prediction results. The actual effect diagram is as follows Figure 2 As shown, this method can efficiently give the accurate location of the sentence description information in the image.

[0102] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present invention, and these modifications or replacements should all be included in the scope of protection of the present invention.

Claims

1. A method for understanding referential expressions based on multi-scale cross-modal feature fusion, characterized by Here are the steps: Step 1: The image is first resized to the same size, then Resnet-101 is used to extract feature maps of n scales, and then mapped to the same dimension d through 1×1 convolution to obtain For language information, we first decompose the language information into words, and then get the feature vectors corresponding to each word through word embedding. The longest sentence is set to have T words, and the blanks of sentences with less than T words are filled with PAD markers; the CLS marker is added at the beginning of the sentence and the SEP marker is added at the end of the sentence; the word vector after position encoding is input into the BERT network to get the feature vectors of each word that integrates the sentence information. Step 2: Input E and V into the model’s cross-modal interaction attention module, which consists of two parts: the linear feature modulation module FiLM and the visually guided language attention module; In the FiLM module, a feature-based affine transformation is applied to adaptively affect the output of the network. For a given language feature First, obtain the entire expression E through the average strategy F , and then specifically through: in and are the weights and biases of two multilayer perceptrons (MLPs) with activation function Tanh. For formula (3), ⊙ and ⊕ represent element-wise multiplication and addition, respectively. Finally, standard 3×3 convolution and ReLU operations are applied to produce multi-level fusion features. For the vision-guided language attention module, the visual features Vi are first flattened into where N i =H i ×W i is the number of visual features, and then according to the language feature E and visual feature Z i Calculated by the following formula: in and is the embedding matrix, where the query, key, and value in this attention module are denoted by Q, K, and V respectively; m is the number of attention heads, d is the feature dimension, and for simplicity, only one language attention module is used for each level of visual features; then, A i It is further encoded by two feed-forward networks FFN with residual connections to form a fused output By connecting F f and F t Get F ft , and then use three 1×1 convolutional layers to transform F ft Mapped to dimension d; finally, the combined features are obtained Step 3: Build a language-guided feature pyramid module FPN First, we construct a routing space of depth K, in which the scaling factor between adjacent stages is constrained to 2. For each routing node, the input consists of two parts: a multi-level feature map and a language vector based on an attention mechanism. The grid-level features at each scale in each routing node are hard-selected by a language gate. First, the input through the language gate consists of two parts: a multi-level feature map and a language vector based on the attention mechanism. The language vector based on the attention mechanism is obtained by the following formula: a k =softmax(EW k ) (5) Where W k ∈R 256×1 is the learning weight, k represents the depth, will be shared to each scale and grid feature; the multi-scale feature map can be expressed as Where i is the i-th scale, k is the k-th layer, N = (H i ×W i ); Language gate through Dynamic selection of language vectors The grid-level features in , the specific operations are as follows: * and · denote convolution and Hadamard product, respectively; conv(·) denotes a 3×3 convolutional network, and σ(·) is the activation function; tanh, i.e., max(0, tahn(·)), is used as the gate switch; when the input is negative, the output of this function is always 0, which eliminates the need for an additional threshold in the inference phase; After that, the output will be Y i,k Upsampling, keeping unchanged and downsampling operations are performed from small scale to large scale respectively; the specific operations are as follows: use To represent the aggregate output in routing node I, The fine-grained features in are further refined by the union gate. Specifically, It will be calculated as follows: Where conv is a 1×1 convolutional network that maps the input features into a channel, l represents the lth node; the nodes in the last layer will be used for multi-scale fusion, and the fusion method is as follows: Formula (11) aggregates the information of different nodes to obtain F AVG , which will be used as the input of the detection head; Step 4: Use anchor-free detection head to locate the object For the output F of step 3 AVG , first use a 1×1 convolution layer to get a feature map of shape w×h×5, representing the five predicted values {t x ,t y ,t w ,t h t}, where the first two values represent the center offset, t w and t h Represent the normalized width and height respectively; the last t is the confidence score, indicating whether there is a center point of the object at that location; finally, the cross entropy loss L is applied to the center point t cls , apply MSE loss L on center offset, width and height off ; At the same time, GIoU loss is used as auxiliary loss; finally, the entire function is defined as: L off =(Δx-t x ) 2 +(Δy-t y ) 2 (14) C ij =1 or 0 indicates whether the current grid contains the center point of the true value target, Indicates the offset of the center point from the center of the grid, where x and y refer to int(·) indicates that the operation rounds the score to the nearest integer; only the grid where the true value center is located is L off ; The total loss function is as follows: Loss=L cls +λ oFF L off +L giou (15) Among them, λ off Set to 5, the network selects the center point with the highest score to generate the bounding box; IoU is the metric used in REC to measure the degree of overlap between the prediction and the true situation.

2. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

3. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

Images