Representation understanding method based on data generation tuning and weight autonomous evolution

Through the method of data generation tuning and autonomous evolution of weights, visual features are dynamically extracted and matched finger representations to meet the needs, solving the problem of mismatch between the visual backbone network extraction features and finger representations, and achieving high-performance finger representation understanding.

CN118918318AActive Publication Date: 2024-11-08NORTHWESTERN POLYTECHNICAL UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411053464.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-08
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

In the existing method of understanding of reference representation, there is a mismatch between the fixed architecture and weighted visual features extracted by the visual backbone network and the features required for reference representation, resulting in missing or redundant features, and inherent preferences have nothing to do with the text. , affects performance.

Method used

Using the method of tuning and weighting based on data generation, the initial reference data generation module, the context construction module with negative examples, the contextual reference data generation module, the language backbone network, the language adaptive weight generator and The visual backbone network dynamically extracts visual features and uses language adaptive weights to generate queries, keys, and values, and performs matrix decomposition to generate dynamic weights to achieve active perception and matching of visual features.

Benefits of technology

Actively extracting visual features related to expression without manually modifying the visual backbone architecture, and a large number of experiments have proved that advanced performance has been achieved on a wide range of data sets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918318A_ABST
    Figure CN118918318A_ABST
Patent Text Reader

Abstract

The invention discloses an indicator understanding method based on data generation tuning and weight autonomous evolution. The first part is an initial indicator data generation module, the second part is a context construction module with a negative example, and the third part is a context indicator data generation module. The fourth part is a language backbone network, the fifth part is a language adaptive weight generator, the sixth part is a visual backbone network, and the seventh part is a representative understanding prediction module.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of language image multimodal fusion, and specifically relates to a method for understanding referential expressions based on data generation optimization and autonomous weight evolution. Background Art

[0002] Referential expression understanding aims to detect specific objects given natural language descriptions, which is more flexible and purposeful than general object detection, which can only locate objects within a fixed set of categories. Free-form language descriptions can specify specific visual properties of the target object, such as category, attributes, relationship with other objects, relative / absolute position, etc. Due to the similarity with detection tasks, previous referential expression understanding methods usually follow the general object detection framework and focus on the design of cross-modal interaction modules. Despite the impressive performance, visual backbone networks have not been well explored. Specifically, visual backbone networks extract visual features with fixed architectures and weights without considering referential expressions. This passive feature extraction may lead to mismatches between the extracted visual features and the features required for various referential expressions, such as missing or redundant features.

[0003] Meanwhile, fixed visual backbone networks have an inherent preference for images, which may be irrelevant to the text of referential expressions. Ideally, visual backbone networks should make full use of referential expressions, as expressions can provide information and inclinations about the desired visual features. Some methods have noticed this phenomenon and proposed corresponding solutions, which achieve expression-aware visual feature extraction by inserting carefully designed interaction modules into visual backbone networks. Although the performance can be improved by adjusting visual features, the extraction-adjustment paradigm inevitably contains a large number of feature extraction components with fixed weights. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a method for referring expression understanding based on data generation tuning and weight autonomous evolution, the first part is the initial referring expression data generation module, the second part is the context construction module with negative examples, the third part is the context referring expression data generation module. The fourth part is the language backbone network, the fifth part is the language adaptive weight generator, the sixth part is the visual backbone network, and the seventh part is the referring expression understanding prediction module.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] Step 1: Given an image dataset D, take one of the images Generate template T using designation expression k Generate an initial expression x ik; Traverse the data set D and obtain the initial expression data set X1;

[0007] Step 2: For a given image I i , give the model a prompt: "Please describe this picture", and let the model generate the corresponding description C i ; Change the image I in X1 i The expression x ik Explicitly add it to the context as the context information of the negative sample, and at the same time send a prompt to the model: "Please avoid generating this type of question", and the context information known by the negative example is: i ,C i ,x ik >

[0008] Step 3: Using the contextual information from the negative example obtained in step 2, repeat step 1 to generate the optimized expression data x i1 , the process is expressed as:

[0009] I i2 =F θ (X i ,C i ,T k ,x i1 ) (1)

[0010] Step 4: Given a referential expression x i1 , use the N-layer language backbone network to segment the expression and add a [CLS] tag at the beginning to extract the language features Where L and d l Represent the number of tags and the dimension of language features respectively; then, the language feature F l Input into the language adaptive weight generator to generate weights for the visual backbone network; next, given an image Where W×H×3 represents the size of the image, expressing the perceived visual features is extracted through the visual backbone network, where C and s represent the number of channels and the stride of the visual features respectively; finally, the language features represented by the [CLS] tag are The visual features are passed to the referential expression understanding prediction module to predict the bounding box of the object referred to by the referential expression understanding;

[0011] Step 5: Introducing a learnable layer-specific embedding Act on each layer of the visual backbone network to dynamically extract the language features related to the layer; for each group g, the attention Assigned to and The normalized dot product of is expressed as:

[0012] ​

[0013] Then, the language features of active perception By aggregation we get: Finally, a fully connected layer is used to reduce the dimension of the aggregated language features of the i-th layer of the visual backbone network, expressed as:

[0014]

[0015] in Used to reduce the dimension to d h =d l / r, where r is the dimensionality reduction ratio; δ represents the eLU activation function;

[0016] Step 6: Generate language adaptive weights based on the referential expression, which are used to generate query X in the visual backbone network q , key X k Sum value X v , expressed as:

[0017] X q =θ(X;W q ),X k =θ(X;W k ),X v =θ(X;W v ) (3)

[0018] Among them, θ(·; W) represents a linear mapping operation with parameter W, and X represents the input visual feature; is the dynamic mapping weight used to generate queries, keys, and values. in and d out They are feature X and feature dimensions of query / key / value respectively;

[0019] The dynamic weights are generated according to the matrix decomposition paradigm. For the i-th ViT block, the process is expressed as:

[0020]

[0021] in, are layer-specific static learnable weights; and are static learnable weights; is a fully connected layer that aggregates language features As input, generate a shape d w ×d w Dynamic matrix of

[0022] Step 7: Apply direct coordinate regression to predict the bounding box of the pointed object;

[0023] First, the visual features and language features Projection to a lower dimensional space Then the attention weight is calculated by dot product similarity Then, Softmax normalization is performed; then, the visual features are aggregated by weighted summation using the attention weight A; finally, the aggregated visual features are input into a fully connected layer and the Sigmoid function is used to predict the bounding box.

[0024] Step 8: Given the predicted bounding box and the true bounding box b = (x, y, w, h), the detection loss function is defined as follows:

[0025]

[0026] in, and denote L1 loss and general IoU loss respectively, λ L1 and λ giou represents the correlation coefficient used to balance the losses.

[0027] Preferably, N=6, L=40, d l =512.

[0028] The beneficial effects of the present invention are as follows:

[0029] 1. The method of the present invention can actively extract visual features related to expression without manually modifying the visual backbone architecture.

[0030] 2. A large number of experiments have demonstrated the effectiveness of the present invention and achieved advanced performance on widely used datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the method for expressing the present invention

[0032] Figure 2 To understand the results of the method of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0034] This paper proposes a new method for understanding referential expressions based on data generation tuning and autonomous weight evolution. This method takes into account that referential expressions and model weights jointly determine the function of the visual backbone network, and adopts a simpler and fine-grained solution, that is, using data generation tuning and autonomous weight evolution to better adapt to referential expression understanding.

[0035] The main modules of the technical solution of the present invention are as follows: the first part is the initial referential expression data generation module, the second part is the context construction module with negative examples, the third part is the context referential expression data generation module. The fourth part is the language backbone network, the fifth part is the language adaptive weight generator, the sixth part is the visual backbone network, and the seventh part is the referential expression understanding prediction module.

[0036] In the first part, initial referential expression data related to the image is generated so that context with negative examples can be constructed in the subsequent second part. In the second part, the context with negative examples is constructed by using the preliminary visual-language alignment ability of the pre-trained model. In the third part, high-quality referential expression data related to the image is constructed by using the constructed context information and the multimodal reasoning ability of the large model itself. In the fourth part, language features are extracted from the high-quality referential expressions; in the fifth part, dynamic weights of the visual backbone network are generated based on the language features. In the sixth part, visual features are extracted from the original image, and their behavior can be modified by language adaptive weights. In the seventh part, the bounding box and mask of the object being referred to are jointly predicted.

[0037] The method of referring to the expression of understanding includes the following main steps:

[0038] Step 1: Given an image dataset D, take one of the images Generate template T using designation expression k Generate an initial expression x ik Traverse the dataset D to obtain the initial expression dataset X1. In the multimodal large language model, when the language model receives the expression generation template T k Since the model focuses on language generation tasks in the pre-training stage and has not yet fully integrated visual understanding capabilities, the generated expression X1 lacks relevance and visual detail information with the image content, and X1 needs to be further tuned.

[0039] Step 2: For a given image I i , give the model a prompt: "Please describe this picture.", and let the model generate the corresponding description C i Since the expression X1 generated in step 1 lacks relevance to the image content, inspired by self-critical learning, the expression X1 about the image I i The expression x ik Explicitly add it to the context as the context information of the negative sample, and at the same time send a prompt to the model: "Please avoid generating this type of question." The context information known by the negative example is: i ,C i ,x ik >. ​

[0040] Step 3: Using the contextual information from the negative example obtained in step 2, repeat step 1 to generate the optimized expression data x i1 , the process can be expressed as:

[0041] I i2 =Fx(X i ,C i ,T k ,x i1 ) (1)

[0042] Step 4: Given a designation expression x obtained in the above step i1 , use the N-layer language backbone network to segment the expression, add a [CLS] tag at the beginning, and extract the language features Where L and d l Represent the number of tags and the dimension of language features respectively. Then, the language feature F l is fed into the language adaptive weight generator to generate weights for the visual backbone network. Next, given an image Where W×H×3 represents the size of the image, expressing the perceived visual features It can be extracted through the visual backbone network, where C and s represent the number of channels and the stride of the visual features respectively. Finally, the language features represented by the [CLS] tag are The visual features are passed to the referential expression understanding prediction module to predict the bounding box of the object referred to by the referential expression understanding.

[0043] Step 5: Considering that referential expressions correspond to different numbers of language tokens, and each layer of the visual backbone network may pay more attention to different language tokens, try to independently aggregate language features with fixed size to each layer. Inspired by the multi-head attention mechanism, a learnable layer-specific embedding is introduced Acting on each layer of the visual backbone network to dynamically extract layer-related language features, which can improve the flexibility of the model at almost no cost. The calculation is performed in G groups. For each group g, the attention Assigned to and The normalized dot product of is expressed as:

[0044]

[0045] Then, the language features of active perception It can be obtained by aggregation: Finally, a fully connected layer is used to reduce the dimension of the aggregated language features of the i-th layer of the visual backbone network, expressed as:

[0046]

[0047] in Used to reduce the dimension to d h =d l / r, where r is the dimensionality reduction ratio. δ represents the eLU activation function.

[0048] Step 6: To guide the active perception of the visual backbone network, language adaptive weights are generated based on the referential expression, which are used to generate query X in the visual backbone network. q , key X k Sum value X v , expressed as

[0049] X q =θ(X;W q ),X k =θ(X;W k ),X v =θ(X;W v ), (3)

[0050] Among them, θ(·; W) represents a linear mapping operation with parameter W, and X represents the input visual features. is the dynamic mapping weight used to generate queries, keys, and values. in and d out They are feature X and the feature dimensions of query / key / value respectively.

[0051] Considering the channel dimension of the dynamic weight (d out ×d in ) is very large, and the computational pressure of directly generating weights using a fully connected layer is unbearable. One solution is to alleviate this problem by weighted summing K static convolution kernels, but this will increase the number of parameters by K times and face challenging joint optimization problems. Inspired by dynamic channel fusion, we try to generate dynamic weights according to the matrix decomposition paradigm. Taking the i-th ViT block as an example, the process can be expressed as:

[0052]

[0053] in, are layer-specific static learnable weights. and Also static learnable weights, but shared across all ViT blocks to reduce the number of parameters and prevent model overfitting is a fully connected layer that aggregates language features As input, generate a shape d w ×d w The dynamic matrix.

[0054] Step 7: Different from the methods that require elaborate cross-modal interaction modules, our method can obtain expression-related visual features through a language-aware visual backbone network without the need for an additional cross-modal interaction module. Through the proposed concise and efficient referential expression understanding prediction module, visual and linguistic features can be used to predict the bounding box of referential expression understanding. Specifically, direct coordinate regression is applied to predict the bounding box of the referred object. In order to pool visual features along the spatial dimension, a language-adaptive pooling module is proposed, which utilizes language-adaptive attention to aggregate visual features. First, the visual features are aggregated. and language features Projection to a lower dimensional space Then the attention weight is calculated by dot product similarity Then, Softmax normalization is performed. Then, the visual features are aggregated by weighted summation using the attention weight A. Finally, the aggregated visual features are input into a fully connected layer and the Sigmoid function is used to predict the bounding box.

[0055] Step 8: The framework of the present invention can be optimized end-to-end for referring to expression understanding. Given the predicted bounding box and the true bounding box b = (x, y, w, h), the detection loss function is defined as follows:

[0056]

[0057] in, and denote L1 loss and general IoU loss respectively, λ L1 and λ giou represents the correlation coefficient used to balance the losses.

[0058] Example:

[0059] The present invention provides a new method for understanding referential expressions, which better adapts to the understanding of referential expressions through data generation optimization and weight autonomous evolution. The specific process is as follows:

[0060] 1. Generation and optimization of referential expressions;

[0061] Given an image dataset of 10,000 images, take one of them Generate template T using designation expression k:<Please describe this picture. > Generate an initial expression. Traverse 10,000 pictures to obtain the initial expression data set. In the multimodal large language model, when the language model part receives the expression generation template, because the model focuses on the language generation task in the pre-training stage and has not yet fully integrated the visual understanding ability, the generated expression lacks relevance and visual detail information with the image content, and needs to be further tuned. Since the expression at this time lacks relevance with the image content, inspired by self-critical learning, the expression is explicitly added to the context as the negative sample context information, and the model is prompted: "Please avoid generating this type of problem.", to obtain the context information known by the negative example. Using the obtained context information known by the negative example, repeat step 1 to generate the tuned expression data x i1 .

[0062] 2. Extraction of referential expression features;

[0063] Given a designation expression x obtained in the above steps i1 , a 6-layer language backbone network based on a bidirectional encoder structure (BERT) is used to segment the expression, add a [CLS] tag at the beginning, and extract language features Where L = 40 and d l =512 represent the number of tags and the dimension of language features respectively. Then, the language feature F l Input into the language adaptive weight generator to generate weights for the vision backbone network.

[0064] 3. Extraction of image features;

[0065] Given a picture in a natural scene Where W×H×3 represents the size of the image. The entire image is resized to 448×448×3 and input into the feature extraction network for forward propagation. The visual transformer structure (ViT-B) is used as the backbone visual network to obtain image features. The dimension size C=512 and the step size s=14.

[0066] 4. Refers to the prediction of the results of expression;

[0067] Language features that will be indicated by the [CLS] tag The visual features are passed to the referential expression understanding prediction module to predict the bounding box of the object referred to by the referential expression understanding.

[0068] 5. Allocation of language features;

[0069] Considering that referential expressions correspond to different numbers of language tokens, and each layer of the visual backbone network may pay more attention to different language tokens, we try to independently aggregate language features with fixed size to each layer. Inspired by the multi-head attention mechanism, we introduce a learnable layer-specific embedding Acting on each layer of the visual backbone network to dynamically extract the language features related to the layer. For each group g, the attention Assigned to and The normalized dot product of is expressed as:

[0070] 6. Aggregation of language features;

[0071] Actively perceived language features It can be obtained by aggregation: Finally, a fully connected layer is used to reduce the dimension of the aggregated language features of the i-th layer of the visual backbone network, expressed as: in Used to reduce the dimension to d h =32, where r=16 is the dimensionality reduction ratio. δ represents the eLU activation function.

[0072] 7. Generation of language adaptive weights;

[0073] In order to guide the active perception of the visual backbone network, language adaptive weights are generated according to the referential expression, which is used to generate query X in the visual backbone network. q , key X k Sum value X v , denoted by X q =θ(X;W q ),X k =θ(X;W k ),X v =θ(X;W v ), where θ(·; W) represents a linear mapping operation with parameter W, and X represents the input visual features. is the dynamic mapping weight used to generate queries, keys, and values. in =512 and d out =512 are the feature dimensions of feature X and query / key / value respectively.

[0074] 8. Generation of dynamic weights;

[0075] Try to generate dynamic weights according to the matrix decomposition paradigm. Taking the i-th ViT block as an example, the process can be expressed as: in, are layer-specific static learnable weights. and Also static learnable weights, but shared across all ViT blocks to reduce the number of parameters and prevent model overfitting is a fully connected layer that aggregates language features As input, a dynamic matrix of shape 32×32 is generated.

[0076] 9. Refers to the prediction of the expression bounding box;

[0077] Different from the methods that require elaborate cross-modal interaction modules, our method can obtain the expression-related visual features through the language-aware visual backbone network without the need for an additional cross-modal interaction module. Through the proposed concise and efficient referential expression understanding prediction module, visual and linguistic features can be used to predict the bounding box of referential expression understanding, and direct coordinate regression can be applied to predict the bounding box of the referred object.

[0078] 10. Language Adaptation Collection;

[0079] In order to aggregate visual features along the spatial dimension, a language-adaptive aggregation module is proposed, which utilizes language-adaptive attention to aggregate visual features. First, the visual features are aggregated and language features Projection to a lower dimensional space Then the attention weight is calculated by dot product similarity Then, Softmax normalization is performed. Then, the visual features are aggregated by weighted summation using the attention weight A. Finally, the aggregated visual features are input into a fully connected layer and the Sigmoid function is used to predict the bounding box.

[0080] 11. Model loss;

[0081] The proposed framework can be optimized end-to-end for referential expression understanding. Given a predicted bounding box and the true bounding box b = (x, y, w, h), the detection loss function is defined as follows: in, and denote L1 loss and general IoU loss respectively, λ L1 and λ giou represents the correlation coefficient used to balance the losses.

[0082] 12. Model training;

[0083] The resolution of the input image is resized to 448×448. ViT-Base (visual encoder) is used as the visual backbone and the visual backbone is adapted to higher resolution images. The visual backbone is pre-trained on a dataset (MS-COCO) where overlapping images in the validation / test sets are excluded. The maximum length of the referential expression is set to 40 and the case-free base model of the six-layer BERT is used as the language backbone to generate language features. L1 and λ giou Set to 1. focal and λ dice is set to 4. The dimensionality reduction ratio r is set to 16. The initial learning rate is 4e-5 for the vision and language backbone and 4e-4 for the remaining components. The model is optimized end-to-end using an optimizer (AdamW) for 90 epochs with a batch size of 256, where the weight decay is set to 1e-4 and the learning rate is reduced by a factor of 10 after 60 epochs. Data augmentation operations include random horizontal flipping.

[0084] 13. Application of the model;

[0085] After the above training process, multiple models can be obtained. The best model (the one with the best test results on the test set) is selected for application. For the input image and question, you only need to adjust the image to 448×448 size and normalize it, and perform word segmentation on the sentence, which can be used as the input of the model. The parameters of the entire network model are fixed. As long as the image data and language data are input and propagated forward, the prediction result can be directly obtained. The actual effect diagram is as follows Figure 2 As shown, referential expression understanding can be performed efficiently based on this method.

Claims

1. A method for understanding referential expressions based on data generation optimization and weight autonomous evolution, characterized in that: The steps include: Step 1: Given an image dataset D, take one of the images Generate template T using designation expression k Generate an initial expression x ik ; Traverse the data set D and obtain the initial expression data set X1; Step 2: For a given image I i , give the model a prompt: "Please describe this picture", and let the model generate the corresponding description C i ; Change the image I in X1 i The expression x ik Explicitly add it to the context as the context information of the negative sample, and at the same time send a prompt to the model: "Please avoid generating this type of question", and the context information known by the negative example is: i ,C i ,x ik >​ Step 3: Using the contextual information from the negative example obtained in step 2, repeat step 1 to generate the optimized expression data x i1 , the process is expressed as: I i2 =F θ (X i ,C i ,T k ,x i1 ) (1) Step 4: Given a referential expression x i1 , use the N-layer language backbone network to segment the expression and add a [CLS] tag at the beginning to extract the language features Where L and d l Dimensions representing the number of tokens and linguistic features, respectively; Then, the language feature F l Input into the language adaptive weight generator to generate weights for the visual backbone network; Next, given an image Where W×H×3 represents the size of the image, expressing the perceived visual features is extracted through the visual backbone network, where C and s represent the number of channels and the stride of the visual features respectively; finally, the language features represented by the [CLS] tag are The visual features are passed to the referential expression understanding prediction module to predict the bounding box of the object referred to by the referential expression understanding; Step 5: Introducing a learnable layer-specific embedding Act on each layer of the visual backbone network to dynamically extract the language features related to the layer; for each group g, the attention Assigned to and F i g The normalized dot product of is expressed as: Then, the language features of active perception By aggregation we get: Finally, a fully connected layer is used to reduce the dimension of the aggregated language features of the i-th layer of the visual backbone network, expressed as: in Used to reduce the dimension to d h =d l / r, where r is the dimensionality reduction ratio; δ represents the eLU activation function; Step 6: Generate language adaptive weights based on the referential expression, which are used to generate query X in the visual backbone network q , key X k Sum value X v , expressed as: X q =θ(X;W q ),X k =θ(X;W k ),X v =θ(X;W v ) (3) Among them, θ(·; W) represents a linear mapping operation with parameter W, and X represents the input visual feature; is the dynamic mapping weight used to generate queries, keys, and values. in and d out They are feature X and feature dimensions of query / key / value respectively; The dynamic weights are generated according to the matrix decomposition paradigm. For the i-th ViT block, the process is expressed as: in, are layer-specific static learnable weights; and are static learnable weights; is a fully connected layer that aggregates language features As input, generate a shape d w ×d w Dynamic matrix of Step 7: Apply direct coordinate regression to predict the bounding box of the pointed object; First, the visual features and language features Projection to a lower dimensional space Then the attention weight is calculated by the dot product similarity Then, Softmax normalization is performed; then, the visual features are aggregated by weighted summation using the attention weight A; finally, the aggregated visual features are input into a fully connected layer and the Sigmoid function is used to predict the bounding box. Step 8: Given the predicted bounding box and the true bounding box b = (x, y, w, h), the detection loss function is defined as follows: in, and denote L1 loss and general IoU loss respectively, λ L1 and λ giou represents the correlation coefficient used to balance the losses.

2. According to claim 1, a method for understanding referential expressions based on data generation optimization and weight autonomous evolution is characterized in that: N=6, L=40, d l =512.

Citation Information

Patent Citations

  • Method, device, equipment and medium for processing visual task by using universal model

    CN115830330A

  • ViT and sliding window attention fusion-based visual pointer understanding method and system

    CN116258931A

  • A method for training a convolutional neural network for image recognition using image-conditioned masked language modeling

    US20210312628A1