Language guide-based referential expression understanding reasoning network system and reasoning method
By utilizing the LGR-NET system and its text feature extension and cross-modal alignment fusion modules, the problem of insufficient utilization of text features in existing technologies is solved, enabling more efficient cross-modal reasoning and object localization, and improving the accuracy of REC tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2023-12-01
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies fail to fully utilize the differences between image and text features in cross-modal reasoning, resulting in suboptimal results in REC tasks, especially in the referential expression comprehension task, where existing methods fail to effectively utilize spatial and semantic information in text features.
The Language-Guided Referential Understanding and Reasoning Network System (LGR-NET) expands and utilizes text features from multiple perspectives through a text feature extractor, an image feature extractor, a text feature expander (TFE), a text-guided cross-modal alignment module (TCA), and a text-guided cross-modal fusion module (TCF) to perform cross-modal alignment and fusion, ultimately generating accurate bounding boxes.
It improves the accuracy of the REC task by fully utilizing the spatial and semantic information in text features, enhancing cross-modal alignment and fusion, and improving the model's accuracy in locating objects.
Smart Images
Figure CN117609536B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer intelligent processing technology, and in particular to a language-guided referential expression comprehension reasoning network system and reasoning method. Background Technology
[0002] Representational understanding (REC) is a fundamental task in the fields of vision and language, aiming to locate image regions based on natural language representations. REC requires models to capture key cues in text and perform accurate cross-modal reasoning.
[0003] To solve the REC task, a key challenge is how to perform accurate cross-modal reasoning: In the existing technology, the following three methods are usually used for cross-modal reasoning: two-stage method, one-stage method and Transformer-based REC method;
[0004] ① Two-stage approach: First, a set of region proposals is generated from the image. Then, a cross-modal similarity metric is used to measure the matching score between the candidate regions and the referential representation. Finally, the region with the highest matching score is selected as the final prediction result.
[0005] ② One-stage method: Typically, multimodal fusion is performed while extracting image features, and bounding boxes are predicted directly on predefined anchor points.
[0006] In summary, both types of methods heavily rely on the performance of off-the-shelf object detectors. Specifically, the first two types of methods are based on two-stage or one-stage object detectors, respectively. The two-stage process typically first generates a set of region proposals for the image and then retrieves the region with the highest matching score to a given expression as the final result. In contrast, the one-stage method directly predicts the predefined anchor with the highest confidence score as the result in extracting image features. Both methods are based on general object detectors and predict the result on pre-generated candidate objects. Therefore, their performance is usually limited by the generated proposals or predefined anchors.
[0007] ③ The REC method based on Transformer:
[0008] Originally proposed for machine translation, the Transformer has been widely used in various natural language processing tasks. Recently, it has been extended to computer vision tasks such as image classification and object detection. Transformer-based REC methods use the Transformer architecture for feature extraction and multimodal interaction, directly generating bounding boxes. The pioneering work TransVG encodes image features using a CNN backbone and a Transformer encoder. BERT was used to extract language features and constructed a Transformer encoder (called the Vision-Language Transformer) to fuse concatenated image-text features. These frameworks employ stacked Transformer layers for cross-modal inference and directly regress bounding boxes without the need for off-the-shelf detectors. Compared to the two-stage and one-stage methods described earlier, these Transformer-based methods are more elegant and perform better.
[0009] However, most converter-based methods typically treat images and text equally, often performing cross-modal reasoning in a coarse manner and utilizing textual features holistically without detailed consideration (e.g., spatial information); this underutilization of textual features leads to suboptimal results.
[0010] In summary, existing methods for performing cross-modal reasoning typically treat image and text features equally, using them in a simplistic way and concatenating them for homogeneous reasoning. Furthermore, text features are used as a whole without specific distinction. However, we argue that image and text features play distinct roles in the REC task. Referential representation is a crucial guide for cross-modal reasoning, while images serve as the carriers for target localization. Therefore, REC models need to capture important cues in the text and use these cues in conjunction with images to gradually reason, identify the target object, and ultimately locate it within the image. Summary of the Invention
[0011] To address the aforementioned technical problems, this invention provides a language-guided referential expression comprehension reasoning network system and reasoning method, proposing a language-guided reasoning network model (LGR-NET) to fully utilize text features for effective cross-modal reasoning and accurately locate the referenced object.
[0012] A language-guided referential expression comprehension reasoning network system and reasoning method, wherein:
[0013] The language-guided referential expression understanding reasoning network system includes: a text feature extractor, an image feature extractor, a text feature expander (TFE), a text-guided cross-modal alignment module (TCA), and a text-guided cross-modal fusion module (TCF).
[0014] The text feature extractor is used to extract text features.
[0015] The image feature extractor is used to extract image features.
[0016] A predictive lexical unit is used to capture key visual and textual features for bounding box prediction and to locate the referent. To fully capture cues in the referential expression, the text feature expander is used to expand the text features in three ways: generating coordinate embeddings (coordinate encoding), word embeddings (word vectors), and sentence embeddings (sentence vectors). These three text features are then repeatedly fed into the cross-modal alignment module and the cross-modal fusion module to participate in the calculation of the cross-modal alignment loss for cross-modal inference. The three expanded text features are fully utilized when performing cross-modal inference.
[0017] At the same time, the predictive terms are also fully learned; the prediction head uses the predictive terms to generate the bounding box of the object it refers to.
[0018] The cross-modal alignment module is used for calculating the input and alignment loss of coordinate embedding, word embedding, and sentence embedding.
[0019] The cross-modal fusion module is used for the fusion of three textual features: coordinate embedding, word embedding, and sentence embedding.
[0020] The reasoning method based on a language-guided referential expression comprehension reasoning network system includes the following steps:
[0021] Step 1: Multimodal feature extraction;
[0022] Using the Swing Transformer as the image feature extractor:
[0023] Input an RGB image ; where 3 represents the number of RGB channels, and H and W are the height and width of the image, respectively;
[0024] First, the initial image feature map is obtained through the image block segmentation module. Where C is the number of channels for the initial image features;
[0025] Secondly, four feature maps are generated through four downsampling stages (4x, 8x, 16x, and 32x). ;
[0026] Then, use a convolution kernel size of The convolutional neural network unifies the channels of the four feature maps as follows: By progressively applying mean pooling and averaging operations to adjacent feature maps, a unified feature map aggregating all feature maps is ultimately obtained. ;
[0027] Meanwhile, in order to improve the feature extraction effect of large target objects, in the feature map Based on The max-pooling operation generates feature maps. .
[0028] Finally, the two feature maps are... and One-dimensional flattening and stitching are performed to obtain image features. , where the feature length .
[0029] Using BERT as the text feature extractor:
[0030] Adding to the text vector before and after the word list mapping and Lexical units; set the maximum sentence length to After feature extraction by BERT, text features are obtained. ;
[0031] After extracting the image and text features, two fully connected neural networks are used to project the image and text features into the same feature space. Finally, the mapped image features are obtained. and mapping text features This is used for subsequent module input;
[0032] Step 2: Text Feature Expansion;
[0033] To fully utilize text features for cross-modal reasoning, a TFE module is used to extend text features;
[0034] TFE generates coordinate encodings of spatial features, word vectors containing all text features, and sentence vectors containing condensed features; the first two are used in the TCA and TCF modules, respectively, and the last one is used for cross-modal loss calculation.
[0035] First, a novel coordinate encoding is generated to enhance the spatial representation of predicted lexical units;
[0036] As an example, referential expressions often include spatial relationships between location words or objects, such as "in front" or "5 o'clock position"; using this spatial information can effectively enhance the spatial representation of predictive lemmas used for location.
[0037] A two-dimensional coordinate is generated using the [CLS] lexical unit via a multilayer perceptron (MLP); the formula for this process is as follows:
[0038]
[0039] FFN consists of two linear layers and a ReLU activation function; It is a feature representation of the [CLS] lexical unit. They are normalized two-dimensional coordinates. The function transforms two-dimensional coordinates into a single coordinate system. 3D sinusoidal positional encoding;
[0040] Then, TFE directly outputs text features. These word vectors are used in the subsequent TCF module;
[0041] Finally, TFE generates a sentence vector for cross-modal loss calculation. The sentence vector is generated from the [CLS] vector, as shown in the following formula:
[0042] ;
[0043] Step 3: Text-guided cross-modal alignment;
[0044] Image features extracted previously Insert a predicted word before As the initial cross-modal representation ,Right now:
[0045] ;
[0046] To align text and image features, an attention mechanism is employed.
[0047] Furthermore, before employing the attention mechanism, spatial representation needs to be introduced from the referential expression to enhance the predictive lexical units; that is, coordinate vectors generated in TFE are added to the predictive lexical units of each layer. ;
[0048]
[0049] Then, a multi-head self-attention mechanism, residual links, and layer normalization are used to update and align visual features.
[0050]
[0051] in: These are the mapping weight matrices for query, key, and value; the output of TCA. That is, aligned image features. It is the number of channels. For layer normalization;
[0052] Step 4: Text-guided cross-modal fusion;
[0053] Employ a cross-modal attention mechanism to fuse text features;
[0054] Aligned image features As a query, text features As the key and value, the text-guided cross-modal attention formula is as follows:
[0055]
[0056] in: These are the mapping weight matrices for query, key, and value, respectively.
[0057] Here, the cross-modal representation of the output Key text features related to the referent object are captured; at the same time, the predicted lexical units aggregate visually relevant text features for subsequent TCA; by stacking N layers of TCA and TCF modules, they are made to work alternately.
[0058] Step 5: Predict the head;
[0059] Based on the fully learned predicted terms from the output of the last layer. The final predicted bounding boxes are generated using a three-layer FFN and sigmoid activation function.
[0060] ;
[0061] in, These represent the coordinates of the center point of the prediction box. Indicates the width and height of the prediction box;
[0062] Step Six: Loss and Training;
[0063] To train LGR-NET, a loss function with two terms is used: the former is the bounding box regression loss, and the latter is the cross-modal alignment loss.
[0064]
[0065] The former loss helps predict the marginal features of the referent captured by the lexical unit; the latter loss promotes the capture of semantic consistency features with the referential object; hyperparameters Used to balance the two;
[0066] As an example, our model consists of N stacked TCA and TCF modules, representing the predicted bounding box of the i-th layer as... The real label is Therefore, the first term of the loss is calculated as follows:
[0067] ;
[0068] in, and These are GIoU loss and L1 loss, respectively.
[0069] In addition, to enhance the alignment effect between the referential target and the referential expression, a contrastive alignment loss is adopted; matching image-text pairs in a batch are treated as positive samples, and non-matching ones are treated as negative samples.
[0070] The alignment loss is as follows:
[0071]
[0072] Where: B is the batch size. This represents the inner product. Where... It is a trainable temperature parameter used to control the smoothness of the distribution. Among these, sentence features... The output from TFE, object features are derived from predicted terms. ;
[0073]
[0074] To simplify the formula expression, the layer subscript is ignored;
[0075] As an example, prediction accuracy is used to evaluate the results of the inference network model; that is, a prediction is considered correct when the intersection-over-union (IoU) between the predicted box and the ground truth box is greater than 0.5, and the accuracy of the model on the test set and validation set is calculated accordingly.
[0076] The beneficial effects of this invention are:
[0077] This invention proposes a Language Guided Inference Network (LGR-NET) model to fully utilize the guidance of referential expressions; in order to locate the referenced object, a predictive lexical is set up to capture cross-modal features; in addition, to fully utilize text features, it is extended in three aspects through a Text Feature Extension (TFE) module.
[0078] LGR-NET is proposed for REC tasks; LGR-NET emphasizes using text features to guide cross-modal reasoning from three aspects: coordinate embedding of text generation helps predictive lemmas capture key visual features; text features are used for alternating cross-modal reasoning; a novel cross-modal loss enhances cross-modal alignment; thus, text features fully guide the overall cross-modal reasoning process of the model from multiple perspectives, making full use of clues in the text and greatly improving model performance. Attached Figure Description
[0079] Figure 1 This diagram compares the differences between existing homogenized reasoning methods and the reasoning method of this invention in solving REC tasks, based on the language-guided referential expression comprehension reasoning network system and reasoning method of this invention.
[0080] Figure 2 This is a schematic diagram of the LGR-NET framework of the language-guided referential expression comprehension reasoning network system of the present invention.
[0081] Figure 3 This is a flowchart of the reasoning method of the language-guided referential expression comprehension reasoning network system of the present invention. Detailed Implementation
[0082] Below, for reference Figures 1 to 3 The figure shows a language-guided referential expression comprehension reasoning network system and reasoning method, in which:
[0083] The language-guided referential expression understanding reasoning network system includes: a text feature extractor, an image feature extractor, a text feature expander (TFE), a text-guided cross-modal alignment module (TCA), and a text-guided cross-modal fusion module (TCF).
[0084] The text feature extractor is used to extract text features.
[0085] The image feature extractor is used to extract image features.
[0086] A predictive lexical unit is used to capture key visual and textual features for bounding box prediction and to locate the referent. To fully capture cues in the referential expression, the text feature expander is used to expand the text features in three ways: generating coordinate embeddings (coordinate encoding), word embeddings (word vectors), and sentence embeddings (sentence vectors). These three text features are then repeatedly fed into the cross-modal alignment module and the cross-modal fusion module to participate in the calculation of the cross-modal alignment loss for cross-modal inference. The three expanded text features are fully utilized when performing cross-modal inference.
[0087] At the same time, the predictive terms are also fully learned; the prediction head uses the predictive terms to generate the bounding box of the object it refers to.
[0088] The cross-modal alignment module is used for calculating the input and alignment loss of coordinate embedding, word embedding, and sentence embedding.
[0089] The cross-modal fusion module is used for the fusion of three text features: coordinate embedding, word embedding, and sentence embedding.
[0090] The reasoning method based on a language-guided referential expression comprehension reasoning network system includes the following steps:
[0091] Step 1: Multimodal feature extraction;
[0092] Using the Swing Transformer as the image feature extractor:
[0093] Input an RGB image ; where 3 represents the number of RGB channels, and H and W are the height and width of the image, respectively;
[0094] First, the initial image feature map is obtained through the image block segmentation module. Where C is the number of channels for the initial image features;
[0095] Secondly, four feature maps are generated through four downsampling stages (4x, 8x, 16x, and 32x). ;
[0096] Then, use a convolution kernel size of The convolutional neural network unifies the channels of the four feature maps as follows: By progressively applying mean pooling and averaging operations to adjacent feature maps, a unified feature map aggregating all feature maps is ultimately obtained. ;
[0097] Meanwhile, in order to improve the feature extraction effect of large target objects, in the feature map Based on The max-pooling operation generates feature maps. .
[0098] Finally, the two feature maps are... and One-dimensional flattening and stitching are performed to obtain image features. , where the feature length .
[0099] Using BERT as the text feature extractor:
[0100] Adding to the text vector before and after the word list mapping and Lexical units; set the maximum sentence length to After feature extraction by BERT, text features are obtained. ;
[0101] After extracting the image and text features, two fully connected neural networks (FFNs) are used to project the image and text features into the same feature space. Finally, the mapped image features are obtained. and mapping text features This is used for subsequent module input;
[0102] Step 2: Text Feature Expansion;
[0103] To fully utilize text features for cross-modal reasoning, a TFE module is used to extend text features;
[0104] TFE generates coordinate encodings of spatial features, word vectors containing all text features, and sentence vectors containing condensed features; the first two are used in the TCA and TCF modules, respectively, and the last one is used for cross-modal loss calculation.
[0105] First, a novel coordinate encoding is generated to enhance the spatial representation of predicted lexical units;
[0106] As an example, referential expressions often include spatial relationships between location words or objects, such as "in front" or "5 o'clock position"; using this spatial information can effectively enhance the spatial representation of predictive lemmas used for location.
[0107] A two-dimensional coordinate is generated using the [CLS] lexical unit via a multilayer perceptron (MLP); the formula for this process is as follows:
[0108]
[0109] FFN consists of two linear layers and a ReLU activation function; It is a feature representation of the [CLS] lexical unit. They are normalized two-dimensional coordinates. The function transforms two-dimensional coordinates into a single coordinate system. 3D sinusoidal positional encoding;
[0110] Then, TFE directly outputs text features. These word vectors are used in the subsequent TCF module;
[0111] Finally, TFE generates a sentence vector for cross-modal loss calculation. The sentence vector is generated from the [CLS] vector, as shown in the following formula:
[0112] ;
[0113] Step 3: Text-guided cross-modal alignment;
[0114] Image features extracted previously Insert a predicted word before As the initial cross-modal representation ,Right now:
[0115] ;
[0116] To align text and image features, an attention mechanism is employed.
[0117] Furthermore, before employing the attention mechanism, spatial representation needs to be introduced from the referential expression to enhance the predictive lexical units; that is, coordinate vectors generated in TFE are added to the predictive lexical units of each layer. ;
[0118]
[0119] Then, a multi-head self-attention mechanism, residual links, and layer normalization are used to update and align visual features.
[0120]
[0121] in: These are the mapping weight matrices for query, key, and value; the output of TCA. That is, aligned image features. It is the number of channels. For layer normalization;
[0122] Step 4: Text-guided cross-modal fusion;
[0123] Employ a cross-modal attention mechanism to fuse text features;
[0124] Aligned image features As a query, text features As the key and value, the text-guided cross-modal attention formula is as follows:
[0125]
[0126] in: These are the mapping weight matrices for query, key, and value, respectively.
[0127] Here, the cross-modal representation of the output Key text features related to the referent object are captured; at the same time, the predicted lexical units aggregate visually relevant text features for subsequent TCA; by stacking N layers of TCA and TCF modules, they are made to work alternately.
[0128] Step 5: Predict the head;
[0129] Based on the fully learned predicted terms from the output of the last layer. The final predicted bounding boxes are generated using a three-layer FFN and sigmoid activation function.
[0130] ;
[0131] in, These represent the coordinates of the center point of the prediction box. Indicates the width and height of the prediction box;
[0132] Step Six: Loss and Training;
[0133] To train LGR-NET, a loss function with two terms is used: the former is the bounding box regression loss, and the latter is the cross-modal alignment loss.
[0134]
[0135] The former loss helps predict the marginal features of the referent captured by the lexical unit; the latter loss promotes the capture of semantic consistency features with the referential object; hyperparameters Used to balance the two;
[0136] As an example, our model consists of N stacked TCA and TCF modules, representing the predicted bounding box of the i-th layer as... The real label is Therefore, the first term of the loss is calculated as follows:
[0137] ;
[0138] in, and These are GIoU loss and L1 loss, respectively.
[0139] In addition, to enhance the alignment effect between the referential target and the referential expression, a contrastive alignment loss is adopted; matching image-text pairs in a batch are treated as positive samples, and non-matching ones are treated as negative samples.
[0140] The alignment loss is as follows:
[0141]
[0142] Where: B is the batch size. This represents the inner product. Where... It is a trainable temperature parameter used to control the smoothness of the distribution. Among these, sentence features... The output from TFE, object features are derived from predicted terms. ;
[0143]
[0144] To simplify the formula expression, the layer subscript is ignored;
[0145] As an example, prediction accuracy is used to evaluate the results of the inference network model; that is, a prediction is considered correct when the intersection-over-union (IoU) between the predicted box and the ground truth box is greater than 0.5, and the accuracy of the model on the test set and validation set is calculated accordingly.
[0146] This invention proposes a Language Guided Inference Network (LGR-NET) model to fully leverage the guidance of referential expressions; a predictive tag is set up to capture cross-modal features in order to locate the referenced object; furthermore, to fully leverage text features, it is extended in three aspects by our Text Feature Extension Module (TFE).
[0147] LGR-NET is proposed for REC tasks; LGR-NET emphasizes using text features to guide cross-modal reasoning from three aspects: coordinate embedding of text generation helps predictive lemmas capture key visual features; text features are used for alternating cross-modal reasoning; a novel cross-modal loss enhances cross-modal alignment; thus, text features fully guide the overall cross-modal reasoning process of the model from multiple perspectives, making full use of clues in the text and greatly improving model performance.
[0148] To better understand the principles of this invention, specific embodiments are illustrated below, with reference to... Figure 1 As shown:
[0149] For example, this invention concisely demonstrates two REC frameworks for generating bounding boxes from an image filled with oranges and corresponding denotative expressions; as a fundamental visual-language task, REC can drive a variety of applications, including image description, visual question answering (VQA), and visual navigation.
[0150] by Figure 1 For example, the model needs to capture key clues, including "orange" and "5 o'clock position"; the former refers to the object, and the latter indicates spatial information;
[0151] Guided by key clues, the model can "understand" what needs to be located and which one; therefore, using homogeneous reasoning schemes from existing technologies is insufficient and prone to inference bias, such as... Figure 1 The existing technology image positioning is incorrect, positioning the orange in the lower left corner, that is, inferring the language and text information from the five o'clock position as the seven o'clock position. This is especially prone to errors when the reference expression is very complex.
[0152] Therefore, this invention employs full utilization of text features for accurate cross-modal reasoning, resulting in accurate and error-free reasoning results.
[0153] The above description is only a preferred embodiment of the present invention. It should be understood that the above description of the embodiments is only for the purpose of helping to understand the method and core idea of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the idea and principle of the present invention should be included within the scope of protection of the present invention.
Claims
1. A language-guided referential expression comprehension reasoning network system, characterized in that, include: Text feature extractor, image feature extractor, text feature expander, text-guided cross-modal alignment module, and text-guided cross-modal fusion module; The text feature extractor is used to extract text features. The image feature extractor is used to extract image features. The text feature expander expands text features in three ways: generating coordinate encoding, word vectors, and sentence vectors. The first two are used in the text-guided cross-modal alignment module and the text-guided cross-modal fusion module, respectively, and the last one is used for cross-modal loss calculation. A two-dimensional coordinate is generated by a multilayer perceptron using [CLS] tokens, the text features are used as word vectors, and the sentence vectors are generated from the [CLS] vectors. The text-guided cross-modal alignment module inserts a predicted word before the extracted image features as an initial cross-modal representation and uses an attention mechanism to align the image and text features. Before adopting the attention mechanism, spatial representation needs to be introduced from the referential expression to enhance the prediction lexical units; that is, coordinate vectors generated in the text feature extension module are added to the prediction lexical units of each layer; then, multi-head self-attention mechanism, residual linking, and layer normalization are used to update and align image features. The text-guided cross-modal fusion module uses a cross-modal attention mechanism to fuse text features; aligned image features are used as queries, and text features are used as keys and values; the output cross-modal representation captures key text features related to the referent object. Meanwhile, the predicted terms aggregate image-related text features for subsequent cross-modal alignment; by stacking N layers of cross-modal alignment and cross-modal fusion modules, they are made to work alternately; at the same time, the predicted terms are also fully learned. The prediction head uses prediction terms to generate bounding boxes for the objects it points to.
2. A reasoning method based on a language-guided referential expression comprehension reasoning network system, characterized in that, Includes the following steps: Step 1: Multimodal feature extraction; extracting image and text features; Step 2, Text Feature Expansion; First, a two-dimensional coordinate is generated using the [CLS] lexical unit through a multilayer perceptron; this is used for the cross-modal alignment module for text guidance; Then, output the text features. The word vectors are used as a cross-modal fusion module for text guidance; finally, a sentence vector is generated from the [CLS] vectors for cross-modal loss calculation. Step 3: Text-guided cross-modal alignment; A predicted term is inserted before the extracted image features as an initial cross-modal representation. Before employing an attention mechanism to align text and image features, spatial representation needs to be introduced from the denotative expression to enhance the predicted lexical units; that is, coordinate vectors generated in the text feature extension module are added to the predicted lexical units at each layer. ; Then, a multi-head self-attention mechanism, residual links, and layer normalization are used to update and align image features. Step 4: Text-guided cross-modal fusion; Employ a cross-modal attention mechanism to fuse text features; Aligned image features as query, text features As key and value, the output cross-modal representation captures key text features related to the referent object; at the same time, predicted lexical units aggregate image-related text features for subsequent cross-modal alignment modules; by stacking N layers of cross-modal alignment fusion and cross-modal fusion modules, they are made to work alternately; Step 5: Predict the head; Based on the fully learned predicted terms from the output of the last layer. The final predicted bounding boxes are generated using a three-layer FFN and sigmoid activation function. ； in, These represent the coordinates of the center point of the prediction box. Indicates the width and height of the prediction box; Step Six: Loss and Training; Training uses a loss function with two terms: the first is the bounding box regression loss, and the second is the cross-modal alignment loss. The former loss helps predict the marginal features of the referent captured by the lexical unit; the latter loss promotes the capture of semantic consistency features with the referential object; hyperparameters Used to balance the two.
3. The reasoning method of the language-guided referential expression comprehension reasoning network system according to claim 2, characterized in that, Referential expressions include spatial relationships between location words or objects, using spatial information to effectively enhance the spatial representation of predictive lexical units used for location.
4. The reasoning method of the language-guided referential expression comprehension reasoning network system according to claim 3, characterized in that, The predicted bounding box of the i-th layer is represented as The real label is Therefore, the first item of the loss is calculated as follows: ； in, and These are GIoU loss and L1 loss, respectively.
5. The reasoning method of the language-guided referential expression comprehension reasoning network system according to claim 3, characterized in that, The prediction accuracy is used to evaluate the results of the inference network model; that is, a prediction is considered correct when the intersection-union ratio of the predicted box and the ground truth box is greater than 0.5, and the accuracy of the model on the test set and the validation set is calculated accordingly.