Method for representing understanding of a pointer based on semi-structured text information

Abstract

The method for representing and understanding of the patent based on semi-structured text information is to extract features of images and texts respectively to obtain image features and text features; the text features are processed by a soft segmentation module to divide the text features into subject-dominant text encoding and context-dominant text encoding; in a multi-modal Transformer fusion module, a similarity matrix between the image features and the text features is calculated to generate multi-modal representation; based on the features, two frame predictions are respectively performed by a progressive inference module, each prediction generates a prediction frame, an absolute error loss and a generalized intersection over union loss between the prediction frame and a label frame are calculated, and the absolute error loss and the generalized intersection over union loss are linearly combined to form an optimization function, and model parameters are adjusted through the optimization function. The method significantly improves the accuracy and robustness of visual positioning on multiple benchmark datasets, and provides an efficient solution for complex language query understanding.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of pointer representation understanding, and particularly relates to a pointer representation understanding method based on semi-structured text information guidance. BACKGROUND

[0002] As a long-standing challenge in vision and language tasks, visual grounding aims to localize targets according to natural language queries. The region annotation forms a superficial correlation between the topic of expression and some common visual entities, which hinders the understanding of the language content and structure by the model. However, the current single-stage method is difficult to uniformly model the visual and language structure due to the structural gap between the continuous image blocks and the discrete text tokens. SUMMARY

[0003] In view of the problems existing in the prior art, the present application provides a pointer representation understanding method based on semi-structured text information guidance, which aims to effectively align the unannotated context information to the stable semantic space corrected by the word-level prior knowledge obtained by using CLIP by designing a cross-modal content alignment module. A multi-branch modulation positioning module is established to obtain modulation positioning through language structure. Through a soft segmentation mechanism, the expression is deconstructed into a fixed semi-structure while ensuring the integrity of the language content, thereby solving the technical problem that the visual and language structure is difficult to be uniformly modeled in the prior art.

[0004] In order to achieve the above purpose, the specific scheme of the present application is as follows:

[0005] The pointer representation understanding method based on semi-structured text information comprises the following steps:

[0006] Step 1, feature extraction is performed on the image and the text to obtain image features and text features, respectively;

[0007] Step 2, the text features in step 1 are processed by a soft segmentation module to segment the text features into subject-dominant text encoding and context-dominant text encoding;

[0008] Step 3, in the multi-modal Transformer fusion module, a similarity matrix between the image features and the text features in step 1 is calculated, and multi-modal information fusion is performed based on the similarity matrix to generate multi-modal representation;

[0009] Step 4, based on the multi-modal representation and the context-dominant text encoding in step 3, a context-guided feature value is calculated through an attention mechanism;

[0010] Step 5, through the progressive inference module, the subject-oriented text encoding in step 2 and the influence of the context-guided feature values in step 4 are used to make two frame predictions respectively, each of which generates a prediction frame, and the prediction frame includes the left upper corner horizontal coordinate, the left upper corner vertical coordinate, the width and the height;

[0011] Step 6, define the optimization function of the model, including calculating the absolute error loss and the generalized intersection over union loss between the prediction frame in step 4 and the label frame, and linearly combining the absolute error loss and the generalized intersection over union loss to form an optimization function, and adjusting the model parameters through the optimization function.

[0012] Further, the step of processing the text features by the soft segmentation module in step 2 is as follows:

[0013] Step 21, use the Stanza word parser to mark the referring text expression, extract the subject and context information;

[0014] Step 22, extract the word with nsubj dependency through logical judgment;

[0015] Step 23, mark the dependency position as 1 and the rest as 0 to form an identification vector;

[0016] Step 24, according to the identification vector, calculate the average of the subject word and the context word, and concatenate them to the beginning of the feature vector;

[0017] Step 25, under the influence of different averages, get the attention weighted representation of subject orientation and context orientation through the improved attention pooling module.

[0018] Further, the formula of the attention weighted representation of subject orientation and context orientation in step 25 is as follows:

[0019] ,

[0020] In the formula, m s represents the weighted text features; α i represents the text representation of the i-th position; represents a 0-1 identifier.

[0021] Further, the formula for calculating the similarity matrix in step 3 is as follows:

[0022] ,

[0023] In the formula, and both represent hyperparameters set artificially; represents an L2 regularization processor; and respectively represent the visual representation obtained by the CLIP model and the text representation obtained by the BERT model; T represents matrix transposition;

[0024] The calculation formula of the multi-modal information fusion is as follows:

[0025]

[0026] In the formula, W q and W k both represent a learnable fully connected layer; E v and E l are corresponding position encodings; F i v and F l respectively represent the visual text features obtained by the Swin-transformer model and the BERT model; S represents a similarity matrix; L1 is an L1 regularization processor; d k represents a hyperparameter set artificially; Q represents a query vector; K represents a key vector; K T represents the transpose of K; a c represents a modified attention map.

[0027] Further, the calculation formula of the context-guided feature value in step 4 is as follows:

[0028]

[0029]

[0030] In the formula, W Q , W K , and W V respectively represent a learnable fully connected layer; G represents a multi-modal representation; F l represents a text representation obtained by the BERT model; q sl represents a context-guided feature value; Q c represents a query vector obtained by text encoding guided by the context; K C represents a key vector obtained by the multi-modal representation; V c represents a value vector obtained by the multi-modal representation; K T c represents the transpose of K c ; d k represents the feature dimension of K c .

[0031] Further, the steps of the two frame predictions in step 5 are as follows:

[0032] ​​​First box prediction: using subject-oriented text encoding and multi-modal representation, the left upper corner horizontal coordinate, the left upper corner vertical coordinate, the width and the height of the target box are preliminarily predicted by a multi-layer perception, and the calculation formula is:

[0033] ,

[0034] In the formula, ReLU is an activation function; b3, b4 and W3, W4 are parameters in a full connection layer; q cl represents a predicted value;

[0035] Second box prediction: using context-guided feature values and multi-modal representation, the target box in the first prediction is optimized and adjusted to further refine the position and size of the target box.

[0036] Further, the calculation formula of the absolute error loss in step 6 is:

[0037] ,

[0038] In the formula, and are the model predicted value and the label value respectively; L1 represents a loss function;

[0039] The calculation formula of the generalized intersection over union loss is:

[0040] ,

[0041] In the formula, A is a predicted box; B is a label box; IoU is the intersection over union between the predicted box and the real box; C is the smallest closed region containing A and B; Union(A, B) is the union area of A and B;

[0042] The calculation formula of the optimization function is:

[0043] ,

[0044] In the formula, and are the model predicted box and the label box respectively; is L1 regularization; is a GIoU loss function; γ is a manually set hyperparameter.

[0045] Advantages of the present application

[0046] The present application is a neural network algorithm based on a Transformer model as a skeleton, which can effectively utilize the word-level prior knowledge obtained by the CLIP model for semantic space correction through a cross-modal content alignment module and a multi-branch modulation positioning module, and can also deconstruct natural language queries through a soft segmentation mechanism to ensure the integrity of the language content while gradually understanding its structure. This method significantly improves the model's understanding ability of complex language queries, especially in cases where there is a surface correlation between the theme of the expression and the common visual entity. Experimental results show that this method significantly improves the performance on multiple benchmark datasets, proving its superiority and effectiveness in the visual positioning task. This method can more finely understand and process visual and language information, thereby improving the accuracy and robustness of visual positioning. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A flowchart of the method for representing expression understanding based on semi-structured text information of the present application. DETAILED DESCRIPTION

[0048] The present application will be further explained and described in conjunction with the accompanying drawings and specific embodiments, it should be noted that the specific embodiments are not used to limit the scope of the present application.

[0049] As shown in Figure 1 The method for representing expression understanding based on semi-structured text information provided by the present specific embodiment includes the following steps:

[0050] Step 1, feature extraction of visual features, text features and prior features in images and texts, respectively obtaining image features and text features;

[0051] The step of feature extraction of visual features, text features and prior features in images and texts is as follows:

[0052] Step 11, the text will be masked and completed to a specified length, and then the BERT model will be used for text feature extraction;

[0053] Step 12, the image will be masked and completed to 640*640, and then the Swin-transformer model will be used for visual feature extraction;

[0054] Step 13, the image will be masked and completed to 640*640, and then the CLIP model will be used for prior visual feature extraction.

[0055] The visual features refer to pre-processing the image and then encoding it using the Swin-transformer model.

[0056] The text feature refers to a representative expression, which is encoded by a BERT model.

[0057] The prior feature refers to an image, which is encoded by a frozen CLIP model visual end.

[0058] Step 2: The text feature in step 1 is processed by a soft segmentation module to divide the text feature into subject-oriented text encoding and context-oriented text encoding.

[0059] The specific steps of processing the text feature by the soft segmentation module are as follows:

[0060] Step 21: Use the Stanza word parser to mark the representative text expression and extract the subject and context information.

[0061] Step 22: Extract the word with dependency "nsubj" (adjectival possessive pronoun) through logical judgment.

[0062] Step 23: Mark the dependency position as 1 and the rest as 0 to form an identification vector for distinguishing the subject and context.

[0063] Step 24: Improve the attention pooling module to calculate the average of the subject word and the context word according to the identification vector and concatenate the average of the subject word and the context word to the beginning of the feature vector.

[0064] The calculation formula of the average of the subject word and the context word is as follows:

[0065] ,

[0066] where n is the size of the calculation batch; m represents the (m-1)th word in the sentence (the first position has no actual semantic); S represents the subject word; and C represents the context word.

[0067] Step 25: Under the influence of different averages, the subject-oriented and context-oriented attention weighted representations are obtained by the improved attention pooling module, and the calculation formula is as follows:

[0068] ,

[0069] where m s represents the weighted text feature; α i represents the text representation at the i-th position; and 0-1 identifier.

[0070] Step 3, in the multi-modal Transformer fusion module, the similarity matrix between the image features and the text features in step 1 is calculated, and multi-modal information fusion is performed based on the similarity matrix to generate multi-modal representation;

[0071] The multi-modal transformer fusion module is as follows:

[0072] Step 31, a pre-trained Swin-transformer model is used as a visual backbone to extract image features and a BERT model is used as a text backbone to extract text features to encode the image-text data set to obtain corresponding representation;

[0073] The visual backbone refers to the backbone network architecture responsible for extracting image features.

[0074] The text backbone refers to the backbone network architecture responsible for extracting text features.

[0075] Step 32, the frozen CLIP model visual end is used to encode the image to obtain a visual representation with priori;

[0076] Step 33, the visual representation obtained by the CLIP model is calculated with the text for similarity score to obtain a similarity distribution matrix S, which is used to represent the similarity distribution of the image block and the text;

[0077] The calculation formula of the similarity matrix is as follows:

[0078] ,

[0079] In the formula, and both represent artificial set hyperparameters; represents an L2 regularization processor; and represent the visual representation obtained by the CLIP model and the text representation obtained by the BERT model respectively; T represents matrix transposition;

[0080] Step 34, under the influence of the similarity matrix S, multi-modal transformer fusion is performed, taking the visual representation as the query vector and the text representation as the key vector, and after Q-K matrix calculation, point multiplication is performed with the similarity matrix to amplify the attention value of the similar part, and the calculation formula of the multi-modal information fusion is as follows:

[0081] ,

[0082] In the formula, W q and W k both represent learnable fully connected layers; E v and E lcorresponding position encoding; F i v and F l respectively represent visual text features obtained by Swin-transformer model and BERT model; S represents similarity matrix; L1 is L1 regularization processor; d k represents hyperparameters set artificially; Q represents query vector; K represents key vector; K T represents transpose of K; a c represents modified attention map.

[0083] Step 4, based on the multi-modal representation and context dominant text encoding described in step 3, the context guided feature value is calculated through the attention mechanism; specifically, the multi-modal output under the influence of the similarity matrix S is the key vector and the value vector, the subject or context dominant text feature obtained by the soft segmentation module is linearly combined with the corresponding position encoding as the query vector, and the context guided feature value is obtained after the first three-layer attention mechanism calculation, and the calculation formula is as follows:

[0084]

[0085]

[0086] In the formula, W Q , W K , W V respectively represent learnable full connection layers; G represents multi-modal representation; F l represents text representation obtained by BERT model; q sl represents context guided feature value; Q c represents query vector obtained by context dominant text encoding; K C represents key vector obtained by multi-modal representation; V c represents value vector obtained by multi-modal representation; K T c represents transpose of K c ; d k represents feature dimension of K c .

[0087] Step 5, through the progressive inference module, under the influence of the subject dominant text encoding in step 2 and the context guided feature value in step 4, the context guided feature value is predicted through three-layer perception, the feature value is reduced through the full connection layer, and the value is changed through the nonlinear change ReLU activation function to avoid overfitting, and finally the frame prediction is obtained, and the frame prediction is performed twice, each time generating a prediction frame, and the prediction frame includes left upper corner horizontal coordinate, left upper corner vertical coordinate, width and height; ​​

[0088] The two steps of frame prediction are as follows:

[0089] First frame prediction: using subject-oriented text encoding and multi-modal representation, the left upper corner horizontal coordinate, the left upper corner vertical coordinate, the width and the height of the target frame are preliminarily predicted by a multi-layer perception, and the calculation formula is:

[0090] ,

[0091] In the formula, ReLU is an activation function; b3, b4 and W3, W4 are parameters in the full connection layer; q cl represents the predicted value;

[0092] Second frame prediction: using context-guided feature values and multi-modal representation, the target frame of the first prediction is optimized and adjusted to further refine the position and size of the target frame.

[0093] Step 6, define the optimization function of the model, including calculating the absolute error loss and the generalized intersection over union loss between the predicted frame and the real label frame, and linearly combining the absolute error loss and the generalized intersection over union loss to form the optimization function, and adjusting the model parameters through the optimization function. The predicted frame refers to the bounding box generated by the model according to the input image, which represents the prediction of the model on the target position and size. The label frame refers to the annotated real bounding box, which is used to represent the position and size of the target object in the image.

[0094] The optimization function of the model is the predicted frame value guided by the context and subject features, which will be calculated with the label frame by the absolute error loss (LAD) and the generalized intersection over union loss (GIoU).

[0095] The forming steps of the optimization function are as follows:

[0096] Step 61, since the progressive prediction module has two output heads, assuming that (x1, y1, w1, h1) and (x2, y2, w2, h2) are the results of the model prediction, and in addition, the label is (x3, y3, w3, h3);

[0097] Step 62, calculate the L1 loss of the model output, by calculating the sum of the absolute value difference between the predicted frame value and the label frame, taking x1 as an example:

[0098] ,

[0099] Step 63, the predicted box value obtained by the context and subject feature guidance will be calculated with the label box Generalized Intersection over Union (GIoU), the calculation formula of Generalized Intersection over Union loss is:

[0100] ,

[0101] In the formula, A is the predicted box; B is the label box; IoU is the intersection over union between the predicted box and the label box; C is the minimum closed region containing A and B; Union(A, B) is the union area of A and B;

[0102] The GIoU loss output by the calculation model is as follows:

[0103] ,

[0104] In the formula, A inter is the intersection area of the predicted box and the label box; A union is the union area of the predicted box and the label box; A C is the area of the minimum closed rectangle containing the two boxes; A enclose is the area in the closed rectangle but not in the union.

[0105] The calculation formula of the absolute error loss is:

[0106] ,

[0107] In the formula, and are the model prediction value and the label value respectively; L1 represents the loss function;

[0108] The calculation formula of the optimization function is:

[0109] ,

[0110] In the formula, and are the model predicted box and the label box respectively; is L1 regularization; is the GIoU loss function; γ is a manually set hyperparameter.

[0111] The following evaluates the generation method on RefCOCO / RefCOCO+ / RefCOCOg / ReferItGame corpus.

[0112] RefCOCO / RefCOCO+ / RefCOCOg: These three datasets belong to the field of Referential Expression Comprehension (REC), and the images come from MSCOCO. The expressions of RefCOCO and RefCOCO+ are collected through a two-player game in ReferItGame, while the expressions of RefCOCOg are collected non-interactively on Amazon Mechanical Turk, so they are longer and more complex. RefCOCOg has two partitions, "google" and "umd". The "google" partition does not have a public test set and there is overlap between the training set and the validation set, while the "umd" partition does not have such overlap.

[0113] ReferItGame: Contains images from SAIAPR12 and collects expressions through a two-player game. In the game, the first player sees an image with object annotations and uses natural language to express the object, and the second player clicks the corresponding area according to the expression, and both sides score and exchange roles.

[0114] Flickr30k Entities: Contains images in the Flickr30k dataset, and the query sentence is a short noun phrase in the image title. Compared with RefCOCO / + / g, the query is simpler and more understandable, but the ambiguity of the expression increases, resulting in relatively high noise.

[0115] Table 1 Dataset distribution

[0116]

[0117] In the evaluation section, the method of the embodiment and the method of the prior art are compared and evaluated in the following three parts:

[0118] (1) Intersection over Union (IoU) is a commonly used evaluation index in target detection and image segmentation tasks, which is used to measure the overlap between the predicted bounding box or region and the ground truth box. Simply put, it calculates the proportion of the intersection area of two bounding boxes (predicted box and true box) to the union area.

[0119] (2) Grounding Accuracy (ACCU) is usually defined as the proportion when the IoU (Intersection over Union) between the predicted bounding box and the ground truth box exceeds a certain preset threshold. This threshold can be adjusted according to the specific task or dataset, and the threshold values are 0.5, 0.7, etc. In this embodiment, the setting is consistent with the mainstream, which is 0.5.

Claims

1. A method for representing understanding of a pointer based on semi-structured text information, characterized by, The method comprises the following steps: Step 1: feature extraction is performed on the image and the text to obtain image features and text features respectively; Step 2: the text features in step 1 are processed by a soft segmentation module to divide the text features into subject-dominant text encoding and context-dominant text encoding; The processing of the text features by the soft segmentation module is as follows: Step 21: a Stanza word parser is used to mark the referring text expression to extract subject and context information; Step 22: a word with a dependency item of nsubj adjective noun pronoun is extracted through logical judgment; Step 23: the dependency item position is marked as 1 and the rest is marked as 0 to form a mark vector; Step 24: according to the mark vector, the average value of the subject word and the context word is calculated, and the average value of the subject word and the context word is spliced to the first position of the feature vector; Step 25: under the influence of different average values, the subject-dominant and context-dominant attention weighted representations are obtained through an improved attention pooling module; Step 3: in a multi-modal Transformer fusion module, a similarity matrix between the image features and the text features in step 1 is calculated, and multi-modal information fusion is performed based on the similarity matrix to generate a multi-modal representation; Step 4: based on the multi-modal representation in step 3 and the context-dominant text encoding, a context-guided feature value is calculated through an attention mechanism; Step 5: through a progressive reasoning module, under the influence of the subject-dominant text encoding in step 2 and the context-guided feature value in step 4, two frame predictions are respectively performed, each time generating a prediction frame, and the prediction frame includes a left upper corner horizontal coordinate, a left upper corner vertical coordinate, a width and a height; Step 6: an optimization function of the model is defined, including calculating the absolute error loss and the generalized intersection over union loss between the prediction frame in step 4 and the label frame, and linearly combining the absolute error loss and the generalized intersection over union loss to form the optimization function, and adjusting the model parameters through the optimization function.

2. The method of claim 1, wherein, The formula of the subject-dominant and context-dominant attention weighted representations in step 25 is as follows: ， wherein m s denotes the weighted text features; a i represents the text representation of the i-th position; denotes a 0-1 identifier.

3. The method of claim 1, wherein, The formula for calculating the similarity matrix in step 3 is as follows: ， In the formula, and both represent the hyperparameters set artificially; represents an L2 regularization processor; and respectively represent the visual representation obtained by the CLIP model and the text representation obtained by the BERT model; T represents matrix transposition; The formula for multi-modal information fusion is as follows: ， wherein W q and W k both represent a learnable fully connected layer; E v and E l are corresponding position encodings; F i v and F l represent visual-textual features obtained by a Swin-transformer model and a BERT model, respectively; S represents a similarity matrix; L1 is an L1 regularization processor; d k represents a hyperparameter set artificially; Q represents a query vector; K represents a key vector; K T represents a transpose of K; a c represents a modified attention map.

4. The method of claim 1, wherein, The formula for calculating the context-guided feature value in step 4 is as follows: ， ， wherein W Q , W K , W V respectively represent learnable fully connected layers; G represents a multi-modal representation; F l represents a text representation obtained by a BERT model; q sl represents a context-guided feature value; Q c represents a query vector obtained by context-guided text encoding; K C represents a key vector obtained by the multi-modal representation; V c represents a value vector obtained by the multi-modal representation; K T c represents a transpose of K c ; d k represents a feature dimension of K c .

5. The method of claim 1, wherein, The steps of the two frame predictions in step 5 are as follows: First frame prediction: using the subject-dominant text encoding and the multi-modal representation, a multi-layer perception is used to preliminarily predict the left upper corner horizontal coordinate, the left upper corner vertical coordinate, the width and the height of the target frame, and the formula is as follows: ， where ReLU is an activation function; b3, b4, and W3, W4 are parameters in the fully connected layers; q cl denotes the predicted value; Second frame prediction: using the context-guided feature value and the multi-modal representation, the target frame of the first prediction is optimized and adjusted to further refine the position and size of the target frame.

6. The method of claim 1, wherein, The formula for calculating the absolute error loss in step 6 is as follows: ， wherein and are the model predicted values and the label values, respectively; L1represents the loss function; The formula for calculating the generalized intersection over union loss is as follows: ， In the formula, A is the prediction frame; B is the label frame; IoU is the intersection over union between the prediction frame and the true frame; C is the smallest closed region containing A and B; Union(A,B) is the union area of A and B; The formula for calculating the optimization function is as follows: ， wherein, and are the model predicted and label boxes, respectively; is the L1 regularization; is the GIoU loss function; γ is a hyperparameter set artificially.

Images