Self-adaptive indicator understanding method and system and storage medium

Abstract

The invention provides a self-adaptive index understanding method and system and a storage medium, and the method comprises the steps: carrying out the extraction of a text feature through a text encoder, and integrating the text feature with visual features obtained through the extraction of different network layers in a visual encoder through a gating mechanism in the visual encoder, so as to obtain multi-modal feature embedding; embedding and updating parameters of a network layer in a visual encoder based on the multi-modal features so as to update the visual features and obtain final visual features; and finally, based on the text features and the final visual features, performing aggregation and prediction through a multi-mode decoder and a forward neural network, and positioning a target position described by the text in the image. According to the method, the visual encoder can pay attention to the region referred to by the text based on the text information in a mode of guiding parameter updating, so that the performance of the visual encoder in the aspect of referring to the understanding task is improved, and the result accuracy of the referring to the understanding task is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a method, system and storage medium for understanding reference expression. Background Art

[0002] With the continuous development of the field of human-computer interaction and the exploration of general artificial intelligence, the referential expression understanding task has become a hot topic and focus of current research. The goal of the referential expression understanding task is to locate the position of the relevant target in the image based on the descriptive information of the referential expression text. The referential expression understanding task involves two modalities: visual images and language. In order to avoid the ambiguity of reasoning, the key issue is to align the language and the visual target area related to it, so as to provide assistance for downstream tasks, such as visual question answering, image description, and visual language navigation. In addition, the referential expression understanding task technology can also be used in the field of intelligent robots to help intelligent robots better understand complex instructions, thereby serving intention understanding and autonomous decision-making.

[0003] At present, methods based on the general object detection framework are usually used to solve the referential expression understanding task. For the two-stage method based on the general object detection framework, the completion of the referential expression understanding task is too dependent on the pre-generated region candidate boxes or pre-designed anchor boxes, which is difficult to adapt to more complex texts or more flexible targets. For the one-stage method based on the general object detection framework, they use independent visual and text encoders to extract visual and text features respectively, and then fuse these features in the multi-modal decoder and make an end-to-end final judgment. Most of these traditional methods based on the general object detection framework use independent modality-specific branches to extract visual or text features. However, the referential expression understanding task often requires locating objects with different text descriptions in the same image. Since its independent visual encoder produces the same visual features for the same image, there is a lack of interaction between the feature branches extracted by the text encoder and the visual encoder. The focus area of ​​the visual encoder is often overfitted to the foreground object in the training data. The performance of the visual encoder in the referential expression understanding task is not fully utilized, resulting in low accuracy of the result of the referential expression understanding task. Summary of the invention

[0004] In order to solve the problem of low accuracy of the results of referring expression understanding tasks performed by the traditional method based on the general target detection framework described in the background technology, the present invention provides an adaptive referring expression understanding method, system and storage medium.

[0005] The method of the present invention comprises:

[0006] Obtain an image and a text description corresponding to the image, perform data enhancement on the image, and standardize the text description corresponding to the image to form a preprocessed image and preprocessed text;

[0007] Input the preprocessed text into the text encoder to extract text features;

[0008] The preprocessed image is input into the first network layer of the visual encoder to extract visual features, and the text features are input into other network layers of the visual encoder except the first network layer. Starting from the second layer of the visual encoder, the visual features extracted by the previous network layer and the text features are integrated through a gating mechanism to obtain a multimodal feature embedding. The parameters in the network layer are updated based on the integrated multimodal feature embedding. The network layer with updated parameters takes the visual features of the previous layer as input and extracts the visual features output by the layer. Each network layer is iteratively updated and extracted in this way to obtain the final visual features.

[0009] The text features and the final visual features are input into the multimodal decoder, where they are aggregated through cross-attention and input into the feed-forward neural network to predict the target box coordinates. The target position of the text description is located in the image based on the target box coordinates.

[0010] Furthermore, the data enhancement method of the image is: randomly scaling the image, cropping it while retaining the target frame, filling it with black pixels to ensure the consistency of the input size, and performing data enhancement by changing the color or contrast of the image to obtain a preprocessed image; the standardization processing method of the text description corresponding to the image is: using a word segmentation tool to convert the text into a word vector, adding a specified tag word vector at the beginning and end of the text to mark the text position, and fixing the text input length by truncation and filling with specified word vectors.

[0011] Furthermore, the method for extracting text features is: inputting the serialized text into a language model and extracting text features; the language model is a pre-trained Bert model, which is composed of 12 layers of Transformer encoding layers, and is pre-trained in masked language modeling and next sentence prediction tasks, and a linear layer is used to reduce the dimension of the Bert output so that the output text features are aligned with the dimensions of the visual features.

[0012] Furthermore, in the process of extracting the final visual features, the different modal features are first mapped to a fixed dimension through a linear layer for alignment, and then the different modal parameters are mapped to a coefficient space using a linear layer and a Sigmoid activation function, and the Hadamard product is performed with the original features again for constraint and fusion to output the final multimodal feature embedding. The specific operations include:

[0013] Given text features And the flattened visual features Where Nt Refers to the length of the text sequence, C t Refers to the text feature dimension, H, W refers to the width and height of the visual feature, C v Refers to the visual feature dimension;

[0014] First, a pooling operation along the sequence length dimension is performed to process text features of different lengths and visual features of different spatial sizes; then, a linear mapping is used to generate an embedding E of fixed dimension. t and E v , and then a gating mechanism with a Sigmoid activation function is used to fuse visual and text embeddings. The calculation process is as follows:

[0015] E t =W t F t , E v =W v F v ,

[0016]

[0017] Where δ represents the ReLU activation function, σ represents the Sigmoid activation function, and is the trainable parameter matrix, C mm Refers to the dimension of multimodal feature embedding, Used to control how much text information is retained;

[0018] After a parameterized gating mechanism, the final multimodal feature embedding E is obtained by adding the visual features to the text features rescaled by the coefficient α mm :

[0019] E mm =αE t +E v .

[0020] Furthermore, in the process of extracting the final visual features, based on the multimodal feature embedding E mm Generate a set of weighted coefficients, use the coefficients to perform weighted sum on a set of learnable low-rank matrices, and use the weighted sum result as the parameter update matrix to guide the weight update of the visual encoder, including:

[0021] The weight update matrix is ​​decomposed and expressed as the sum of vector outer products:

[0022]

[0023] Among them, B i , A i ∈Rd×1 Represents the matrix B, the i-th column and i-th row of A; based on B i , A i In the form of outer product sum, a set of weighted coefficients are generated by multimodal feature embedding to control the parameter update matrix:

[0024]

[0025] Among them, W0∈R d×k represents the fixed, independent weight matrix of the visual encoder, ΔW represents the conditional weight update matrix, and B i ∈R d×1 , A i ∈R 1×d Here is a set of learnable vectors, and the parameter update is approximated by the outer product form, coefficients w1, w2, ..., w r is a scalar generated from multimodal feature embedding, where r refers to the rank of the conditional weight update matrix, which is a hyperparameter that can be set by yourself. ;

[0026] Based on the multimodal feature embedding obtained by the gating mechanism, this set of coefficients is generated through a learnable linear mapping:

[0027] [w1, w2, ..., w r ] T =W g E mm +[b1,b2,...,b r ] T ,

[0028] Among them, W g ∈R r×d , [b1, b2, ..., b r ] T is the parameter matrix, E mm ∈R d is a layer-specific multimodal embedding generated by the text features and visual features output by the previous layer; all parameter matrices W0, B, A, W g , [b1, b2, ..., b r ] T They are all learnable during the training phase.

[0029] Furthermore, in the process of extracting the final visual features, the parameters of the visual encoder are updated by multimodal feature embedding to achieve dynamic extraction of text-guided visual features. For the visual encoder commonly used in referential expression understanding, it is composed of a convolutional neural network or a neural network stacked with Transformer layers. The convolutional neural network usually includes ResNet50, ResNet101, and the Transformer stacked neural network usually includes VisionTransformer, Swin-Transformer visual feature extractor;

[0030] For the common convolution module Conv k×k , and its weight update is expressed as Where k represents the convolution kernel size, c in and c out Represents the number of input channels and output channels, and expands this weight update into a two-dimensional matrix, expressed as And use two matrices of rank r To approximate this parameter update matrix;

[0031] The given weight update consists of two consecutive convolutional layers Conv with convolution sizes 1 and k. B and Conv A To approximate, the two consecutive convolutional layers use r as output and input channels, and the multimodal conditional parameter update of the convolutional block can be expressed as:

[0032] X output =Conv k×k (X)+Conv A (W mm ⊙Conv B (X)

[0033] Among them, W mm is the weight coefficient generated by multimodal feature embedding, and its dimension is r;

[0034] For the Transformer layer, its self-attention module also contains a linear mapping consisting of matrix multiplication;

[0035] Consider the standard self-attention mechanism:

[0036]

[0037] Among them, the input visual feature X is calculated after three different linear mappings, W q , W k and W v Refers to the weights of three different linear mappings, corresponding to Query, Key and Value in self-attention calculation; for the parameter matrix Wq , W k The update can be written as follows:

[0038]

[0039] Where W q =W q +ΔW q , W k =W k +ΔW k Refers to the updated parameter matrix, W q and W k Both ΔWq and ΔW are obtained by end-to-end training of the network. k It is obtained by multimodal feature embedding control, and its calculation process is as follows:

[0040]

[0041]

[0042] Among them, B i ∈R d×1 , A i ∈R 1×d Here is a set of learnable vectors, and the parameter update is approximated by the outer product form, coefficients w1, w2, ..., w r is a scalar generated from the multimodal feature embedding, where r refers to the rank of the conditional weight update matrix.

[0043] Furthermore, the method for aggregating the final visual features and text features is: the final visual features, text features and a learnable embedding vector are input into a multimodal decoder composed of a transformer self-attention layer, and the learnable embedding integrates the visual and text features in the forward process through a cross-attention mechanism, wherein the cross-attention mechanism specifically refers to using a learnable token as a query to perform cross-attention with text and visual features respectively to aggregate target-related features.

[0044] Furthermore, the method for predicting the target frame coordinates by the forward neural network is as follows: the forward neural network used for prediction includes a multi-layer stacked fully connected network, the hidden layer activation function is ReLU, the output layer activation function is Sigmoid, the number of channels of the hidden layer can be set as needed, and the number of output channels of the output layer is 4, corresponding to the normalized relative position (x, y, w, h) of the final target coordinates in the image, the center coordinates x, y, and the object width and height w, h.

[0045] The present invention also proposes an adaptive referential expression understanding system, which includes a data acquisition and preprocessing module, a text feature extraction module, a final visual feature extraction module, and a target position positioning module.

[0046] The data acquisition and preprocessing module is used to acquire an image and a text description corresponding to the image, perform data enhancement on the image, and standardize the text description corresponding to the image to form a preprocessed image and preprocessed text.

[0047] The text feature extraction module is used to input the preprocessed text into the text encoder to extract text features.

[0048] The final visual feature extraction module is used to input the preprocessed image into the first network layer of the visual encoder to extract visual features, and input the text features into other network layers of the visual encoder except the first network layer. Starting from the second layer of the visual encoder, the visual features extracted by the previous network layer and the text features will be integrated through a gating mechanism to obtain multimodal feature embedding, and the parameters in the network layer are updated based on the integrated multimodal feature embedding. Then, the network layer with updated parameters uses the visual features of the previous layer as input and extracts the visual features output by the layer. By analogy, each network layer is iteratively updated and extracted to obtain the final visual features.

[0049] The target position positioning module is used to input the text features and the final visual features into the multimodal decoder, aggregate the final visual features and the text features in the multimodal decoder through cross attention, and input them into the forward neural network to predict the target frame coordinates, and locate the target position of the text description in the image according to the target frame coordinates.

[0050] The present invention also proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for understanding adaptive referential expression as described above is implemented.

[0051] Compared with the prior art, after the text features are extracted by the text encoder, the text features are integrated with the visual features extracted by different network layers in the visual encoder through a gating mechanism in the visual encoder to obtain a multimodal feature embedding, and the parameters of the network layer in the visual encoder are adaptively updated based on the multimodal feature embedding, thereby updating the visual features to obtain the final visual features; finally, based on the text features and the final visual features, the multimodal decoder and the forward neural network are aggregated and predicted to locate the target position of the text description in the image. The present invention addresses the problem of lack of interaction between the feature branches extracted by the text feature encoder and the visual encoder in the traditional method, and guides the parameter update so that the visual encoder can focus on the area referred to by the text based on the text information, thereby improving the performance of the visual encoder in the referential expression understanding task and greatly improving the accuracy of the results of the referential expression understanding task. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a flow chart of the method.

[0053] Figure 2 Flowchart of the gating mechanism.

[0054] Figure 3 Schematic diagram of the parameter update process.

[0055] Figure 4 It is a system architecture diagram of the present invention.

[0056] Figure 5 1 is a flow chart comparing a conventional method and the method of the present invention, wherein (a) is the conventional method and (b) is the method of the present invention. DETAILED DESCRIPTION

[0057] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] An adaptive method for understanding referential expressions, the flow chart is as follows Figure 1 As shown, the specific steps are described as follows.

[0059] First, an image and a text description corresponding to the image are obtained, and data enhancement is performed on the image and the text description corresponding to the image is standardized to form a preprocessed image and preprocessed text.

[0060] Specifically, the data enhancement method for the image is as follows: randomly scale the image, crop it while retaining the target frame, fill it with black pixels to ensure the consistency of the input size, and perform data enhancement by changing the color or contrast of the image to obtain the preprocessed image.

[0061] Specifically, the standardized processing method for the text description corresponding to the image is: use the word segmentation tool to convert the text into a word vector, add a specified tag word vector at the beginning and end of the text to mark the text position, and use truncation and specified word vector padding to fix the text input length.

[0062] Then, the preprocessed text is input into the text encoder to extract text features.

[0063] Specifically, the text feature extraction method is:

[0064] The serialized text is input into the language model and the text features are extracted; the language model is a pre-trained Bert model, which consists of 12 layers of Transformer encoding layers and is pre-trained in masked language modeling and next sentence prediction tasks. A linear layer is used to reduce the dimension of the Bert output so that the output text features are aligned with the dimensions of the visual features.

[0065] Next, the preprocessed image is input into the first network layer of the visual encoder to extract visual features, and the text features are input into other network layers of the visual encoder except the first network layer. Starting from the second layer of the visual encoder, the visual features and text features extracted by the previous network layer will be integrated through a gating mechanism to obtain multimodal feature embedding. The parameters in the network layer are updated based on the integrated multimodal feature embedding, and then the network layer with updated parameters uses the visual features of the previous layer as input and extracts the visual features of the output of this layer. Each network layer is iteratively updated and extracted in this way to obtain the final visual features.

[0066] Specifically, in the second network layer, the input text features and the layer of visual features extracted by the first network layer are integrated through the gating mechanism to obtain the integrated two-layer multimodal feature embedding, and the parameters in the second network layer are updated based on the integrated multimodal feature embedding. Then, the second network layer with updated parameters and the preprocessed image input to the first network layer are used to extract the second layer of visual features. Similarly, each network layer is iteratively updated and extracted to obtain the final visual features.

[0067] Specifically, in the process of extracting the final visual features, the different modal features are first mapped to a fixed dimension through a linear layer for alignment, and then the different modal parameters are mapped to the coefficient space using a linear layer and a Sigmoid activation function, and the Hadamard product is re-performed with the original features for constraint and fusion to output the final multimodal feature embedding. The flowchart of the gating mechanism is shown in the figure. Figure 2 As shown, the specific operations include:

[0068] Given text features And the flattened visual features Where N t Refers to the length of the text sequence, C t Refers to the text feature dimension, H, W refers to the width and height of the visual feature, C v Refers to the visual feature dimension;

[0069] First, we process text features of different lengths and visual features of different spatial sizes by pooling along the sequence length dimension. Then, we use linear mapping to generate an embedding E of fixed dimension. t and E v , and then a gating mechanism with a Sigmoid activation function is used to fuse visual and text embeddings. The calculation process is as follows:

[0070] E t =W t F t , E v =W v F v ,

[0071]

[0072] Where δ represents the ReLU activation function, σ represents the Sigmoid activation function, and is the trainable parameter matrix, C mm Refers to the dimension of multimodal feature embedding, Used to control how much text information is retained;

[0073] After a parameterized gating mechanism, the final multimodal feature embedding E is obtained by adding the visual features to the text features rescaled by the coefficient α mm :

[0074] E mm =αE t +E v .

[0075] Then, based on the multimodal feature embedding E mmGenerate a set of weighted coefficients, use the coefficients to perform weighted sum on a set of learnable low-rank matrices, and use the weighted sum result as the parameter update matrix to guide the weight update of the visual encoder, including:

[0076] The weight update matrix is ​​decomposed and expressed as the sum of vector outer products:

[0077]

[0078] Among them, B i , A i ∈R d×1 Represents the matrix B, the i-th column and i-th row of A; based on B i , A i In the form of outer product sum, a set of weighted coefficients are generated by multimodal feature embedding to control the parameter update matrix:

[0079]

[0080] Among them, W0∈R d×k represents the fixed, independent weight matrix of the visual encoder, ΔW represents the conditional weight update matrix, and B i ∈R d×1 , A i ∈R 1×d Here is a set of learnable vectors, and the parameter update is approximated by the outer product form, coefficients w1, w2, ..., w r is a scalar generated from multimodal feature embedding, where r refers to the rank of the conditional weight update matrix, which is a hyperparameter that can be set by yourself;

[0081] Based on the multimodal feature embedding obtained by the gating mechanism, this set of coefficients is generated through a learnable linear mapping:

[0082] [w1, w2, ..., w r ] T =W g E mm +[b1,b2,...,b r ] T ,

[0083] Among them, W g ∈R r×d , [b1, b2, ..., b r ] T is the parameter matrix, E mm ∈R d is a layer-specific multimodal embedding generated by the text features and visual features output by the previous layer; all parameter matrices W0, B, A, W g , [b1, b2, ..., b r ]T They are all learnable during the training phase.

[0084] Finally, the parameters of the visual encoder are updated through multimodal feature embedding to dynamically extract text-guided visual features. For the common visual encoder in referential expression understanding, it is composed of a convolutional neural network or a neural network stacked with Transformer layers. The convolutional neural network usually includes ResNet50, ResNet101, and the Transformer stacked neural network usually includes Vision Transformer and win-Transformer visual feature extractors. The flowchart of parameter update is as follows Figure 3 shown.

[0085] For the common convolution module Conv k×k , and its weight update is expressed as Where k represents the convolution kernel size, c in and c out Represents the number of input channels and output channels, and expands this weight update into a two-dimensional matrix, expressed as And use two matrices of rank r To approximate this parameter update matrix;

[0086] The given weight update consists of two consecutive convolutional layers Conv with convolution sizes 1 and k. B and Conv A To approximate, the two consecutive convolutional layers use r as output and input channels, and the multimodal conditional parameter update of the convolutional block can be expressed as:

[0087] X output =Conv k×k (X)+Conv A (W mm ⊙Conv B (X)

[0088] Among them, W mm is the weight coefficient generated by multimodal feature embedding, and its dimension is r;

[0089] For the Transformer layer, its self-attention module also contains a linear mapping consisting of matrix multiplication;

[0090] Consider the standard self-attention mechanism:

[0091]

[0092] Among them, the input visual feature X is calculated after three different linear mappings, W q , W k and Wv Refers to the weights of three different linear mappings, corresponding to Query, Key and Value in self-attention calculation; for the parameter matrix W q , W k The update can be written as follows:

[0093]

[0094] Where W q =W q +ΔW q , W k =W k +ΔW k Refers to the updated parameter matrix, W q and W k All are obtained by end-to-end training of the network, ΔW q and ΔW k It is obtained by multimodal feature embedding control, and its calculation process is as follows:

[0095]

[0096]

[0097] Among them, B i ∈R d×1 , A i ∈R 1×d Here is a set of learnable vectors, and the parameter update is approximated by the outer product form, coefficients w1, w2, ..., w r is a scalar generated from the multimodal feature embedding, where r refers to the rank of the conditional weight update matrix.

[0098] Finally, the text features and the final visual features are input into the multimodal decoder, where they are aggregated through cross-attention and input into the feed-forward neural network to predict the target box coordinates. The target position of the text description is located in the image based on the target box coordinates.

[0099] Specifically, the method for aggregating the final visual features and text features is: the final visual features, text features and a learnable embedding vector are input into a multimodal decoder composed of a transformer self-attention layer, and the learnable embedding is integrated with the visual and text features in the forward process through the cross-attention mechanism. The cross-attention mechanism specifically refers to using a learnable token as a query to perform cross-attention with text and visual features respectively to aggregate target-related features.

[0100] The method of predicting the target frame coordinates by the forward neural network is as follows: the forward neural network used for prediction includes a multi-layer stacked fully connected network, the hidden layer activation function is ReLU, the output layer activation function is Sigmoid, the number of channels of the hidden layer can be set as needed, and the number of output channels of the output layer is 4, corresponding to the normalized relative position (x, y, w, h) of the final target coordinates in the image, the center coordinates x, y, and the object width and height w, h.

[0101] An adaptive reference expression understanding system, the architecture diagram is as follows Figure 4 As shown, it consists of a data acquisition and preprocessing module, a text feature extraction module, a final visual feature extraction module, and a target position positioning module.

[0102] The data acquisition and preprocessing module is used to acquire images and text descriptions corresponding to the images, perform data enhancement on the images, and standardize the text descriptions corresponding to the images to form preprocessed images and preprocessed texts.

[0103] The text feature extraction module is used to input the preprocessed text into the text encoder to extract text features.

[0104] The final visual feature extraction module inputs the preprocessed image into the first network layer of the visual encoder to extract visual features, and inputs the text features into other network layers of the visual encoder except the first network layer. Starting from the second layer of the visual encoder, the visual features and text features extracted from the previous network layer will be integrated through a gating mechanism to obtain a multimodal feature embedding. The parameters in the network layer are updated based on the integrated multimodal feature embedding, and then the network layer with updated parameters uses the visual features of the previous layer as input and extracts the visual features of the output of this layer. Each network layer is iteratively updated and extracted in this way to obtain the final visual features.

[0105] The target position localization module is used to input the text features and the final visual features into the multimodal decoder, aggregate the final visual features and text features through cross-attention in the multimodal decoder, and input them into the forward neural network to predict the target box coordinates. The target position of the text description in the image is located based on the target box coordinates.

[0106] The specific implementation method of each module in the system is consistent with that described in the above method and will not be repeated here.

[0107] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements an adaptive referential expression understanding method and an adaptive referential expression understanding system as described above.

[0108] In the referential expression understanding task, the present invention guides the adaptive update of the parameters of the visual encoder based on the text features of the language, thereby improving the problem that the independent visual feature extraction process in the referential expression understanding task affects the final prediction result. Most of the existing traditional methods use an attention mechanism based on text participation or a dynamic visual feature encoder guided by language, which requires a sophisticated design or a relatively complex parameter generation network, and has poor prediction results for more flexible language expressions in practical applications. The present invention uses text feature information to generate a set of coefficients to control a fixed parameter update matrix, so that the parameters of the visual encoder can be dynamically updated, and its parameter update range is limited to a fixed parameter subspace, so that it can still maintain stable performance when performing referential expression understanding tasks.

[0109] When guiding the parameters of the visual encoder based on the text features of the language, the present invention first fuses the text features with the visual features based on the gating mechanism, and obtains multimodal feature embedding to guide the subsequent parameter update. Compared with traditional methods, most traditional methods only rely on text feature information, which limits the flexibility of traditional methods in practical applications, and the prediction performance of the referential expression understanding task is limited by the text feature information; while the present invention uses a gating mechanism to control the fusion of text information, which greatly improves the text perception ability when performing the referential expression understanding task.

[0110] The present invention integrates the design of language-guided parameter update into common network modules, and also designs text parameter update for convolutional modules and Transformer modules, so that they can realize the fusion of text information with a lower number of parameters and there is no need to design additional modules for networks with different architectures.

[0111] Figure 5 The figure is a flow chart comparing the traditional method and the method of the present invention. (a) is the traditional method, which uses an independent visual feature extraction process and an independent visual branch. Its visual encoder can only extract fixed visual features for a fixed image, which makes it focus on the foreground area identified during the training process, and cannot focus on different specific areas according to different text expressions. Its attention is distributed in the foreground area and is not concentrated. (b) is the method of the present invention, which uses a multimodal conditional visual feature extraction process, and can dynamically update the parameters of the visual encoder according to different text information, so that it focuses on the relevant area referred to by the text. , thereby further improving the prediction effect for the specified object.

[0112] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of complete hardware embodiment, complete software embodiment, or the embodiment in combination with software and hardware. Moreover, the application can adopt the form of the computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java, C++, Python and literal scripting language JavaScript, etc.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0116] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0117] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An adaptive denotative expression understanding method, characterized in that: include: Obtain an image and a text description corresponding to the image, perform data enhancement on the image, and standardize the text description corresponding to the image to form a preprocessed image and preprocessed text; Input the preprocessed text into the text encoder to extract text features; The preprocessed image is input into the first network layer of the visual encoder to extract visual features, and the text features are input into other network layers of the visual encoder except the first network layer. Starting from the second layer of the visual encoder, the visual features extracted by the previous network layer and the text features are integrated through a gating mechanism to obtain a multimodal feature embedding. The parameters in the network layer are updated based on the integrated multimodal feature embedding. The network layer with updated parameters takes the visual features of the previous layer as input and extracts the visual features output by the layer. Each network layer is iteratively updated and extracted in this way to obtain the final visual features. The text features and the final visual features are input into the multimodal decoder, where they are aggregated through cross-attention and input into the feed-forward neural network to predict the target box coordinates. The target position of the text description is located in the image based on the target box coordinates.

2. The method for understanding an adaptive referential expression according to claim 1, characterized in that: The data enhancement method of the image is: randomly scaling the image, cropping it while retaining the target frame, filling it with black pixels to ensure the consistency of the input size, and performing data enhancement by changing the color or contrast of the image to obtain a preprocessed image; The standardized processing method of the text description corresponding to the image is: using a word segmentation tool to convert the text into a word vector, adding a specified tag word vector at the beginning and end of the text to mark the text position, and fixing the text input length by truncation and padding with specified word vectors.

3. The method for understanding adaptive referential expression according to claim 2, characterized in that: The method for extracting the text features is: The serialized text is input into the language model and the text features are extracted; the language model is a pre-trained Bert model, which consists of 12 layers of Transformer encoding layers and is pre-trained in masked language modeling and next sentence prediction tasks. A linear layer is used to reduce the dimension of the Bert output so that the output text features are aligned with the dimensions of the visual features.

4. The method for understanding an adaptive referential expression according to claim 1, characterized in that: In the process of extracting the final visual features, the different modal features are first mapped to a fixed dimension through a linear layer for alignment, and then the different modal parameters are mapped to the coefficient space using a linear layer and a Sigmoid activation function, and the Hadamard product is performed with the original features again for constraint and fusion to output the final multimodal feature embedding. The specific operations include: Given text features And the flattened visual features Where N t Refers to the length of the text sequence, C t Refers to the text feature dimension, H, W refers to the width and height of the visual feature, C v Refers to the visual feature dimension; First, we process text features of different lengths and visual features of different spatial sizes by pooling along the sequence length dimension. Then, we use linear mapping to generate an embedding E of fixed dimension. t and E v , and then a gating mechanism with a Sigmoid activation function is used to fuse visual and text embeddings. The calculation process is as follows: HAVE BEEN t =W t F t ,HAVE BEEN v =W v F v , Where δ represents the ReLU activation function, σ represents the Sigmoid activation function, and is the trainable parameter matrix, C mm Refers to the dimension of multimodal feature embedding, Used to control how much text information is retained; After a parameterized gating mechanism, the final multimodal feature embedding E is obtained by adding the visual features to the text features rescaled by the coefficient α mm : AND mm =αE t +E v 。 5. The method for understanding adaptive referential expression according to claim 4, characterized in that: In the process of extracting the final visual features, based on the multimodal feature embedding E mm Generate a set of weighted coefficients, use the coefficients to perform weighted sum on a set of learnable low-rank matrices, and use the weighted sum result as the parameter update matrix to guide the weight update of the visual encoder, including: The weight update matrix ΔW is decomposed and expressed as the sum of vector outer products: Among them, B i , A i ∈R d×1 Represents the matrix B, the i-th column and i-th row of A; based on B i , A i In the form of outer product sum, a set of weighted coefficients are generated by multimodal feature embedding to control the parameter update matrix: Among them, W0∈R d×k represents the fixed, independent weight matrix of the visual encoder, ΔW represents the conditional weight update matrix, and B i ∈R d×1 , A i ∈R 1×d Here is a set of learnable vectors, and the parameter update is approximated by the outer product form, coefficients w1, w2, ..., w r is a scalar generated from the multimodal feature embedding, where r refers to the rank of the conditional weight update matrix; Based on the multimodal feature embedding obtained by the gating mechanism, this set of coefficients is generated through a learnable linear mapping: [w1,w2,…,w r ] T =W g E mm +[b1,b2,…,b r ] T , Among them, W g ∈R r×d ,[b1,b2,…,b r ] T is the parameter matrix, E mm ∈R d is a layer-specific multimodal embedding generated by the text features and visual features output by the previous layer; all parameter matrices W0, B, A, W g , [b1, b2, ..., b r ] T They are all learnable during the training phase.

6. The method for understanding adaptive referential expression according to claim 5, characterized in that: In the process of extracting the final visual features, the parameters of the visual encoder are updated by multimodal feature embedding to achieve dynamic extraction of text-guided visual features. For the visual encoder commonly used in referential expression understanding, it is composed of a convolutional neural network or a neural network stacked with Transformer layers. The convolutional neural network usually includes ResNet50, ResNet101, and the neural network stacked with Transformer layers usually includes Vision Transformer, Swin-Transformer visual feature extractor; For the common convolution module Conv k×k , and its weight update is expressed as Where k represents the convolution kernel size, c in and c out Represents the number of input channels and output channels, and expands this weight update into a two-dimensional matrix, expressed as And use two matrices of rank r To approximate this parameter update matrix; Since the given weight update consists of two consecutive convolutional layers Conv with convolution sizes 1 and k B and Conv A To approximate, the two consecutive convolutional layers use r as output and input channels, and the multimodal conditional parameter update of the convolutional block can be expressed as: X output =Conv k×k (X)+Conv A (W mm ⊙Conv B (X)) Among them, W mm is the weight coefficient generated by multimodal feature embedding, and its dimension is r; For the Transformer layer, its self-attention module also contains a linear mapping consisting of matrix multiplication; Consider the standard self-attention mechanism: Among them, the input visual feature X is calculated after three different linear mappings, W q , W k and W v Refers to the weights of three different linear mappings, corresponding to Query, Key and Value in self-attention calculation; for the parameter matrix W q , W k The update can be written as follows: Where W′ q =W q +ΔW q , W′ k =W k +ΔW k refers to the updated parameter matrix, W q and W k All are obtained by end-to-end training of the network, ΔW q and ΔW k It is obtained by multimodal feature embedding control, and its calculation process is as follows: Among them, B i ∈R d×1 , A i ∈R 1×d Here is a set of learnable vectors, and the parameter update is approximated by the outer product form, coefficients w1, w2, ..., w r It is a scalar generated from multimodal feature embedding, where r refers to the rank of the conditional weight update matrix, which is a hyperparameter that can be set by yourself.

7. The method for understanding adaptive referential expression according to claim 6, characterized in that: The method for aggregating the final visual features and text features is: the final visual features, text features and a learnable embedding vector are input into a multimodal decoder composed of a transformer self-attention layer, and the learnable embedding integrates the visual and text features in the forward process through a cross-attention mechanism, wherein the cross-attention mechanism specifically refers to using a learnable token as a query to perform cross-attention with text and visual features respectively to aggregate target-related features.

8. The method for understanding adaptive referential expression according to claim 7, characterized in that: The method for predicting the target frame coordinates by the forward neural network is as follows: the forward neural network used for prediction includes a multi-layer stacked fully connected network, the hidden layer activation function is ReLU, the output layer activation function is Sigmoid, the number of channels of the hidden layer can be set as needed, and the number of output channels of the output layer is 4, corresponding to the normalized relative position (x, y, w, h) of the final target coordinates in the image, the center coordinates x, y, and the object width and height w, h.

9. An adaptive referential expression understanding system implementing the method of any one of claims 1 to 8, characterized in that: It includes data acquisition and preprocessing module, text feature extraction module, final visual feature extraction module, and target position positioning module; The data acquisition and preprocessing module is used to acquire images and text descriptions corresponding to the images, perform data enhancement on the images, and standardize the text descriptions corresponding to the images to form preprocessed images and preprocessed texts; The text feature extraction module is used to input the preprocessed text into the text encoder to extract text features; The final visual feature extraction module is used to input the preprocessed image into the first network layer of the visual encoder to extract visual features, and input the text features into other network layers of the visual encoder except the first network layer. Starting from the second layer of the visual encoder, the visual features extracted by the previous network layer and the text features are integrated through a gating mechanism to obtain a multimodal feature embedding, and the parameters in the network layer are updated based on the integrated multimodal feature embedding. Then, the network layer after the parameter update uses the visual features of the previous layer as input and extracts the visual features output by the layer, and so on, iteratively updates and extracts each network layer to obtain the final visual features. The target position positioning module is used to input the text features and the final visual features into the multimodal decoder, aggregate the final visual features and the text features in the multimodal decoder through cross attention, and input them into the forward neural network to predict the target frame coordinates, and locate the target position of the text description in the image according to the target frame coordinates.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements an adaptive referential expression understanding method as described in any one of claims 1-8.

Images