A method and system for understanding referential expressions for computer vision language alignment

Abstract

The present invention discloses a method and system for understanding referential expressions for computer vision language alignment, including: analyzing a language expression input by a computer through a language parser to generate text features; using a selective filtering module guided by an expression to adaptively select visual features related to the language expression from a feature map of a visual object; constructing a multimodal attention map based on the text features and the visual features; and executing a noun-oriented multi-step reasoning strategy to achieve graph reasoning and target object matching. The present invention selects relevant visual features of individual objects through an expression-guided selective filtering module, and uses them together with expression representation for graph construction. The negative impact of functional words in the expression is effectively reduced through a noun-oriented reasoning strategy, thereby improving the accuracy of text-to-object matching.

Description

Technical Field

[0001] The present invention belongs to the technical field of image-text matching and pattern recognition, and in particular relates to a method and system for understanding referential expressions for computer vision language alignment. Background Art

[0002] Visual-language alignment is a hot topic in the field of computer vision and has been widely used in many applications, such as referential expression understanding (REC), referential expression segmentation (RES), visual question answering, image description, scene understanding, video localization retrieval, and robot navigation. Among the above applications, REC is a language-to-vision matching task that requires the establishment of a relationship model between visual objects and language expressions. Therefore, the computer not only needs to extract the necessary information from the given expression, but also needs to locate the target object instance in the given image or video that matches the expression. Analyzing and extracting relevant features of nouns and relational phrases from the given expression is the key to successfully linking referential expressions to the real world. Taking the expression "a woman is standing to the right of a computer" as an example, the goal is to locate the woman in the image or video that matches the text description. This involves multiple reasoning steps, first identifying "computer" and then locating "woman" based on the hint of the relational phrase "to the right".

[0003] In the existing technology, the best performing REC algorithms are mainly graph-based methods and Transformer-based methods. Compared with Transformer-based methods, graph-based methods provide more compact solutions, while transformer-based methods require higher memory and computing resources.

[0004] However, in the graph construction stage, existing graph-based REC (GREC) methods usually only calculate node weights based on feature vectors pooled from the object's feature graph. This does not effectively establish the relationship between objects and expressions. In particular, since these GREC methods usually adopt object detectors to generate object candidate boxes for matching, these candidate box regions usually contain a large amount of background areas that are irrelevant to the expression.

[0005] At the same time, in the process of graph construction, existing GREC methods only consider the information of given expressions to construct edges between nodes, while ignoring the information of the nodes themselves. In fact, the features of nodes and expressions come from different feature domains. Without the help of node features, expressions may not fully reflect the relationship between nodes. In addition, most GREC methods perform fixed-step reasoning when matching target objects, ignoring the complexity of expressions, which may lead to unsatisfactory reasoning results. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a method for understanding referential expressions for computer vision language alignment to solve the problems existing in the above prior art.

[0007] To achieve the above object, the present invention provides a method for understanding referential expressions for computer vision language alignment, comprising:

[0008] The language expression input by the computer is analyzed by the language parser to generate text features;

[0009] Adaptively selecting visual features related to the language expression from a feature map of a visual object using an expression-guided selection filter module;

[0010] Based on the text features and the visual features, construct a multimodal attention map;

[0011] A noun-guided multi-step reasoning strategy is implemented to achieve graph reasoning and target object matching.

[0012] Preferably, the process of analyzing the language expression input by the computer by the language parser includes:

[0013] Embed each word in the language expression into a feature vector;

[0014] Processing the feature vector using a bidirectional long short-term memory network to generate output features for each word;

[0015] The final hidden state of the bidirectional long short-term memory network is used as the feature representation of the entire language expression.

[0016] Preferably, the process of adaptively selecting visual features related to the language expression from the feature map of the visual object comprises:

[0017] The word features output by the bidirectional long short-term memory network are linearly converted into filters;

[0018] Applying the filter to a feature map of the visual object and calculating a relevance score between each pixel and the word;

[0019] Based on the correlation scores, the retained visual features are determined through threshold screening and Sigmoid function.

[0020] Preferably, the process of constructing a multimodal attention map includes:

[0021] constructing a node of a graph based on each target object of the visual object, wherein the node contains visual features and linguistic features of the object;

[0022] constructing graph edges according to the relationship between the nodes and the language expressions;

[0023] The first weight of the node and the second weight of the edge are calculated to obtain the correlation between the expression feature and the visual feature, and a multimodal attention graph is constructed based on the correlation.

[0024] Preferably, the noun-oriented reasoning strategy includes:

[0025] determining a number of inference steps based on the number of nouns or noun phrases in the language expression;

[0026] At each reasoning step, the weights of the nodes in the graph are updated to reflect the relevance of the nodes to the current noun of interest.

[0027] Preferably, the process of achieving target object matching includes:

[0028] Calculate the relevance score between each object candidate and the language expression by inner product;

[0029] The object candidate with the highest correlation score is selected as the target object matching the language expression.

[0030] The present invention discloses a system for understanding referential expressions for computer vision language alignment, which is used to implement the method for understanding referential expressions for computer vision language alignment, and comprises:

[0031] The language parsing module is used to analyze the language expression input by the computer and generate text features;

[0032] An expression-guided selection and filtering module for adaptively selecting visual features related to the language expression from a feature map of a visual object;

[0033] An image construction module, configured to construct a multimodal attention map based on the text features and the visual features;

[0034] The reasoning and matching module is used to execute a noun-oriented multi-step reasoning strategy to achieve graph reasoning and target object matching.

[0035] Preferably, the system also includes training the referential expression understanding system through a triple loss function to optimize the match between the target object and the language expression.

[0036] Preferably, the formula for training the reference expression understanding system is:

[0037] L triplet =max(δ neg +Δ-δ gt ,0)

[0038] Among them, δ neg and δgt are the scores of other objectives and expressions describing the objectives respectively, and Δ is the boundary value.

[0039] Compared with the prior art, the present invention has the following advantages and technical effects:

[0040] The present invention provides a method for understanding referential expressions for computer vision language alignment, comprising: firstly, analyzing a language expression input by a computer through a language parser to generate text features; secondly, using an expression-guided selection and filtering module to adaptively select visual features related to the language expression from a feature graph of a visual object; then, based on the text features and the visual features, constructing a multimodal attention graph; and finally, executing a noun-oriented multi-step reasoning strategy to achieve graph reasoning and target object matching.

[0041] The present invention proposes an expression-guided selective filtering module, which is well compatible with the graph-based REC method. The module is used to select relevant visual features of individual objects and is used together with the expression representation for graph construction. This module has been proven to be very effective in more accurately matching the target object with the given expression. In addition, the present invention proposes a noun-guided reasoning strategy that determines the number of reasoning steps based on the number of nouns or noun phrases. This method effectively reduces the negative impact of functional words in expressions, thereby improving the accuracy of text-to-object matching. In addition, compared with other state-of-the-art Transformer-based methods, the method of the present invention consumes less computing resources during training and performs excellently. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Embodiment 1

[0047] This embodiment provides a method for understanding referential expressions for computer vision language alignment, including:

[0048] The language expression input by the computer is analyzed by the language parser to generate text features;

[0049] Adaptively selecting visual features related to the language expression from a feature map of a visual object using an expression-guided selection filter module;

[0050] Based on the text features and the visual features, construct a multimodal attention map;

[0051] A noun-guided multi-step reasoning strategy is implemented to achieve graph reasoning and target object matching.

[0052] Specifically, Figure 1 As shown, this embodiment consists of 4 parts, including: a language parser, a multimodal attention graph construction module, a noun-oriented reasoning module and a matching module. First, the expression is analyzed by a Bi-LSTM-based language parser and the corresponding text features are generated. Next, a multimodal attention graph is constructed by introducing an EGSFM module and a graph construction module to cover the objects in the picture. Once the graph is constructed, a noun-oriented multi-step reasoning strategy is executed for information aggregation and matching of graph nodes, where the size of the node in the graph indicates that the weight assigned to the node is higher.

[0053] S1. Analyze the language expression input by the computer through the language parser to generate text features;

[0054] Specifically, we first calculate the soft distribution of L words in the expression, where the expression is expressed as More specifically, the L words in the expression are first embedded into a set of feature vectors. The noun phrase of the expression is represented by {e1,e2,...,e T}. T represents the number of noun phrases in the expression, where e i ∈d e .

[0055] Then, these embedded word vectors are input into a bidirectional LSTM (Bi-LSTM). The output features of each word after bidirectional LSTM processing are represented as H = {h1,h2,...,h L}, where h i ∈d h is the concatenation of the forward and backward LSTM outputs of the lth word. At the same time, the output concatenation of the last hidden state of the forward and backward LSTM is q∈d q is used as the feature representation of the entire expression.

[0056] The historical information represented by the expression can be obtained by computing a series of soft distributions of words in the text expression through the introduced progressive noun-guided decomposition strategy.

[0057] Typically, after t decompositions, this embodiment hopes that the expression will focus on the first t nouns or noun phrases. In other words, the weights of the first t nouns or noun phrases in the t-th soft distribution should be greater than other words. Assume that the expression has T nouns or noun groups. In the t-th step, first, feature q is converted to q by a learned linear transformation function. (t) , then q (t) The output y of the previous step (t-1) ∈d y (the final output of step t-1) is connected, and the result is represented as u (t) ∈d u ,Right now

[0058]

[0059] Where W (t) is a trainable parameter matrix, b (t) is the trainable bias at step t. When t = 1, y (t-1) =y (0) is a randomized vector.

[0060] In addition, by calculating u (t) and h l The similarity between them can predict the correlation between the lth word and the entire expression, as follows:

[0061]

[0062] Where W u , W α , W qy , W h , is a trainable parameter matrix, b u is a trainable bias. Indicates u (t) The result after activation by the activation function ReLU. and express and h l The similarity score between represents the result of normalizing the addition of two similarity scores. Regarding the last formula, if t-1 = 0, then e (0) = 0 and e (1) = e1, otherwise e (t)represents the fusion of the embedding vectors of the first t nouns or noun groups. In the last equation, the e on the right side of the equation (t) Indicates the characteristics of the t-th noun in the expression, e (t-1) represents the features of the t-1th noun before the expression. Through the last formula, the fusion of the features of the t-1th noun before the expression is (t-1) and the feature e of the t-th noun (t) The fused features are represented by e (t) , which represents the fusion of the features of the first t nouns. Finally, y (t) Can be expressed as

[0063] According to the above description, after t inference steps, this embodiment hopes that the expression will focus on the first t nouns or noun groups. In other words, in the soft distribution after the tth inference (in formula (2) ) The weight of the first t nouns or noun groups in the expression should be greater, so the loss function L e Defined as:

[0064]

[0065] S2, using an expression-guided selection filtering module to adaptively select visual features related to the language expression from a feature map of the visual object;

[0066] Based on the text features and the visual features, a multimodal attention map is constructed.

[0067] First, a directed graph is constructed based on the given image. In this embodiment, the graph is represented as G = (V, E, S, X). The nodes of the graph encode the information of the objects in the image, and the edges represent the relationship between the nodes and the expressions. For a given image I, which contains K target objects, denoted as Represents a set of graph nodes, Represents the ROI feature map of K target objects, and the feature map size is H s ×W s ×C s . is a set of edges, where e ij Represents the relationship between object i and object j. X is a set of feature vectors of K target objects, where the i-th vector x i is node v i The concatenation of the feature vector of and its spatial features (i.e., the coordinates of the upper left and lower right corners of the object’s bounding box).

[0068] Once G is constructed, this embodiment can construct a multimodal graph G for the method framework of this embodiment based on G M . GM The nodes of G contain both object information from the image and information about the expression. M The definition of edges in G is the same as that of edges in G. M The details of the construction of nodes and edges are further described below.

[0069] S21, G M Construction of graph nodes

[0070] Graph node features. In order to capture the relationship between the lth word and the ith node in G, this embodiment introduces an expression-guided selection filtering module (EGSFM), which can effectively extract visual features related to text features, thereby integrating effective text features into graph node features, making the features of graph nodes richer and making the matching of text and graph nodes more accurate in subsequent multi-step reasoning.

[0071] Specifically, each word feature h from the Bi-LSTM output is first l Linear transformation to the visual feature map s i 1×1×c-dimensional filters with the same channel dimensions Here’s how:

[0072]

[0073] Where W trans is a trainable parameter matrix.

[0074] Then, Applied to feature map s i , to calculate the lth word and s i In this case, if s i The feature at position (r1, r2) has a high correlation with the lth word, so the corresponding score should be larger. After the calculation is completed, this embodiment obtains s i The score moment

[0075]

[0076] in Represents the convolution kernel In the feature map i The convolution result, M i,l is a 2D score matrix with size H s ×W s .

[0077] M i,l The score of reflects the low correlation between the corresponding pixel and the lth word.i,l The fraction of M that is greater than a certain threshold. i,l Apply the Sigmoid function to calculate. We get σ(M i,l )={σ(M i,l (1,1)),...,σ(M i,l (H s ,W s ))}。σ(M i,l ).σ(M i,l ) in the range of [0,1]. If σ(M i,l (h,w))≥ξ th ,ξ th The value in this article is 0.5. This embodiment will retain this score. Finally, the correlation score between the i-th node and the l-th word is:

[0078]

[0079] Among them, 1 is the indicator function. Similar to the lth word, the correlation score c between other words in the expression and the i-th node i,1 ,c i,2 ,...,c i,L Can also be obtained.

[0080] Then, the expression can be reformulated as:

[0081]

[0082] Finally, this embodiment will H i fused to the G node to construct a multimodal graph. Specifically, G M The feature of the i-th node is calculated as

[0083]

[0084] in, is a trainable parameter matrix, is a trainable bias.

[0085] S21, G M Calculation of node weights and edge weights

[0086] S211, node weight

[0087] Similar to the above calculation, this embodiment first converts the expression feature q into a filter To extract G M The K feature maps of the node are obtained by applying the Sigmoid function σ(·) to obtain the corresponding size H s ×W sThe K score matrices σ(M q,1 ),σ(M q,2 ),...,σ(M q,K ). In σ(M q,i ), greater than the threshold ξ th The score of G will be retained. M The weight of the i-th node is calculated as

[0088]

[0089] Among them, 1 is the indicator function.

[0090] S211, edge weight

[0091] The edge weight reflects the M The relationship between different nodes, the higher the weight, the stronger the correlation between the i-th node and the j-th node. M The weight between the i-th node and the j-th node is calculated as follows:

[0092]

[0093] in W q is a trainable parameter matrix.

[0094] S3. Execute a noun-guided multi-step reasoning strategy to achieve graph reasoning and target object matching.

[0095] S31. In order to better perform text-to-object matching, this embodiment introduces a noun-based reasoning strategy for the multimodal graph G M . Unlike the fixed reasoning steps in other GREC methods, the number of reasoning steps in the noun-guided reasoning module should be flexibly adjusted according to the number of nouns and noun groups in the expression. This adaptive method can not only save computing resources for short expressions, but also better understand the underlying semantics in long and complex expressions. Therefore, the number of reasoning steps in the noun-guided reasoning module is consistent with the number of outputs generated by the progressive noun-guided decomposition strategy (i.e., T). Specifically, for an expression involving T nouns or noun groups, the method of this embodiment uses T reasoning steps to decompose GREC into G nouns. M Perform multi-step reasoning.

[0096] It should be noted that in the language interpreter part, after the parser processes the expression through T decomposition steps, the expression can be represented by a series of feature vectors {y (1) ,y (2) ,...,y (T)Each vector contains all the information of the expression, but different vectors have different focuses on the nouns or noun groups of the expression. In the tth graph reasoning step, G M Each node of can be represented as

[0097] Once the weights of nodes and edges in the first inference step are calculated, a graph convolutional network (GCN) can be applied to G M , here, about The calculation details are as follows

[0098]

[0099] in and α ij are the results of formula (2) and formula (6), is a trainable parameter matrix, is a trainable bias. It represents the result of information aggregation of all nodes connected to the i-th node in the first step of reasoning. Represents the result of the transformation of the i-th node itself. express and The fusion result is then fused with the information of the i-th node in the previous step. The previous step here is step 0. In step 0, the feature of the i-th node is

[0100] In the tth (t>1) reasoning step, the nodes in the graph may represent a single object or a composite object. In other words, each node in the graph may contain features of multiple objects (i.e., t-1 objects). If a composite node contains more features of the first t-1 nouns or noun groups in the expression, this embodiment gives this node a higher weight. Then, the node in the tth reasoning step The weight of can be obtained as follows:

[0101]

[0102] Where W m1 ,W m2 is a trainable parameter matrix.

[0103] for and The edge weights between , this embodiment hopes that after the tth reasoning step, more features of the first t nouns or noun groups in the expression can be included in the real node (the target described by the expression). and The t-th noun or noun group in the expression contains more information between them, and the weight between them should be larger. At the t-th (i.e. t>1) reasoning step, y (t) Focus on the text features of the first t nouns or noun groups in the expression. and The weights between are obtained as follows:

[0104]

[0105] Represents the i-th node and the jth node The similarity score between the fused text features and the first t nouns or noun groups. is the result of normalizing the similarity score. Finally, in the tth reasoning step, The calculation process is as follows:

[0106]

[0107] in, It represents the result of information aggregation of all nodes connected to the i-th node in the t-th step of reasoning. Represents the result of the transformation of the i-th node itself. express and After fusion, add the i-th node in the previous step The result after feature fusion.

[0108] S32. Matching process and loss function

[0109] Once the multi-step reasoning is completed, the present embodiment continues to use the matching module to match the composite object with the specified expression. After T reasoning steps, the proposed method calculates a score for each object candidate, reflecting the correlation between the object and the expression. i , in this embodiment, the score δ can be calculated by the following inner product i :

[0110]

[0111] Where i=1,2,...,K. W q is a trainable parameter matrix. Then, the following triplet loss function is used as supervision to train the network:

[0112] L triplet =max(δ neg +Δ-δ gt ,0), (15)

[0113] Among them, δ neg and δ gt are the scores of other objectives and the objectives described by the expression, respectively, and Δ is the boundary value.

[0114] The model training process of this embodiment:

[0115] S1. Input training data, including: language features of expressions and visual features of image targets.

[0116] S2. Obtain two sets of features about expressions through the language interpreter, H = {h1,h2,...,h L} and {y (1) ,y (2) ,...,y (T)}.

[0117] S3, with the expression feature H = {h1,h2,...,h L} is used as a guide to construct the multimodal graph G through equations (4)-(8) M , G M The nodes of the graph contain both the visual features of the target and the linguistic features of the expression.

[0118] S4, through equations (9)-(10) is the multimodal graph G M Calculating node weights and the edge weights

[0119] S5. Guided by the noun features of the expression, multi-step reasoning is performed on the constructed multimodal graph through equations (11)-(13). At each step of reasoning, a node in the graph receives information from the nodes connected to it.

[0120]

[0121] S6. Calculate the total loss and update the model parameters through equations (3), (14)-(15).

[0122] S7. Determine whether the maximum number of training rounds R is reached. If so, proceed to S8; otherwise, return to S2.

[0123] S8. Save the model parameters and end the training.

[0124] The model testing phase of this embodiment includes:

[0125] S1. Input test data.

[0126] S2. Constructing a multimodal graph G through the model M .

[0127] S3. Through the multi-step reasoning strategy, after T-step reasoning strategy, each node of the multimodal graph will get a score

[0128] S4. Find the position of the image corresponding to the node with the highest score.

[0129] S5. Output prediction results.

[0130] Beneficial effects of this embodiment:

[0131] This embodiment proposes an expression-guided selective filtering module. This module is well compatible with the graph-based REC method. This module is used to select relevant visual features of individual objects and use them together with expression representations for graph construction. This module has been proven to be very effective in more accurately matching target objects with given expressions.

[0132] This embodiment proposes a noun-oriented reasoning strategy that determines the number of reasoning steps based on the number of nouns or noun phrases. This method effectively reduces the negative impact of function words in expressions, thereby improving the accuracy of text-to-object matching.

[0133] This embodiment proposes a step-by-step noun-oriented decomposition strategy to decompose expressions. By adopting this strategy, the language parser can more accurately locate effective words and phrases.

[0134] This embodiment conducts extensive experiments on three challenging public datasets including RefCOCO, RefCOCO+, and RefCOCOg. The results show that the method of this embodiment outperforms many advanced graph-based methods and is robust in processing complex language expressions.

[0135] Embodiment 2

[0136] This embodiment discloses a system for understanding referential expressions for computer vision language alignment, which is used to implement the method for understanding referential expressions for computer vision language alignment described in Embodiment 1, including:

[0137] The language parsing module is used to analyze the language expression input by the computer and generate text features;

[0138] An expression-guided selection and filtering module for adaptively selecting visual features related to the language expression from a feature map of a visual object;

[0139] An image construction module, configured to construct a multimodal attention map based on the text features and the visual features;

[0140] The reasoning and matching module is used to execute a noun-oriented multi-step reasoning strategy to achieve graph reasoning and target object matching.

[0141] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for understanding referential expressions for computer vision language alignment, characterized in that: The following steps are involved: The language expression input by the computer is analyzed by the language parser to generate text features; Adaptively selecting visual features related to the language expression from a feature map of a visual object using an expression-guided selection filter module; The process of adaptively selecting visual features related to the language expression from the feature map of the visual object includes: The word features output by the bidirectional long short-term memory network are linearly converted into filters; Applying the filter to a feature map of the visual object and calculating a relevance score between each pixel and the word; Based on the correlation score, determining the retained visual features through threshold screening and Sigmoid function; Based on the text features and the visual features, construct a multimodal attention map; Implement a noun-guided multi-step reasoning strategy to achieve graph reasoning and target object matching; The noun-oriented reasoning strategies include: determining a number of inference steps based on the number of nouns or noun phrases in the language expression; At each reasoning step, the weights of the nodes in the graph are updated to reflect the relevance of the nodes to the current noun of interest.

2. The method for understanding referential expressions for computer vision language alignment according to claim 1, characterized in that: The process of analyzing the language expression input by the computer through the language parser includes: Embed each word in the language expression into a feature vector; Processing the feature vector using a bidirectional long short-term memory network to generate output features for each word; The final hidden state of the bidirectional long short-term memory network is used as the feature representation of the entire language expression.

3. The method for understanding referential expressions for computer vision language alignment according to claim 1, characterized in that: The process of constructing a multimodal attention map includes: constructing a node of a graph based on each target object of the visual object, wherein the node contains visual features and linguistic features of the object; constructing graph edges according to the relationship between the nodes and the language expressions; The first weight of the node and the second weight of the edge are calculated to obtain the correlation between the expression feature and the visual feature, and a multimodal attention graph is constructed based on the correlation.

4. The method for understanding referential expressions for computer vision language alignment according to claim 1, characterized in that: The process of achieving target object matching includes: Calculate the relevance score between each object candidate and the language expression by inner product; The object candidate with the highest correlation score is selected as the target object matching the language expression.

5. A system for understanding referential expressions for computer vision language alignment, for implementing the method for understanding referential expressions for computer vision language alignment according to any one of claims 1 to 4, comprising: The language parsing module is used to analyze the language expression input by the computer and generate text features; An expression-guided selection and filtering module for adaptively selecting visual features related to the language expression from a feature map of a visual object; An image construction module, configured to construct a multimodal attention map based on the text features and the visual features; The reasoning and matching module is used to execute a noun-oriented multi-step reasoning strategy to achieve graph reasoning and target object matching.

6. The system for understanding referential expressions for computer vision language alignment according to claim 5, characterized in that: It also includes training the referential expression understanding system through a triple loss function to optimize the match between the target object and the language expression.

7. The system for understanding referential expressions for computer vision language alignment according to claim 6, characterized in that: The formula for training the denotative expression understanding system is: L triplet =max(δ neg +D-d gt ,0) Among them, δ neg and δ gt are the scores of other objectives and expressions describing the objectives respectively, and Δ is the boundary value.

Images