Scene graph generation method based on unified decoder

Abstract

The invention discloses a scene graph generation method based on a unified decoder. The method comprises the following steps: 1) extracting image features; (2) after image features are extracted, a decoder is used for predicting a relation triple; and 3) estimating the obtained visual triad according to the relation triad, and obtaining a finally generated scene graph. The invention provides a novel single-stage scene graph generation architecture which is constructed by a unified decoder with task-specific queries, an available solution is provided for the problem of weak coupling in relation triple prediction, entities are respectively positioned by using the task-specific queries, and the task-specific queries are used for solving the problem of weak coupling in relation triple prediction. And semantic features are fused in the triple to share coupling features, so that better performance is realized with fewer parameters.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a scene graph generation method based on a unified decoder. Background Art

[0002] The scene graph generation task is an extremely important part of visual understanding. It can help computers better understand the content of images. By extracting objects and relationships in images, it can obtain a deeper understanding of scene structure and semantic information, providing richer input for other visual tasks.

[0003] Existing scene graph generation (SGG) methods can be roughly divided into two categories: two-stage based methods and single-stage based methods.

[0004] Two-stage methods usually use a pre-trained object detector (e.g., Faster R-CNN) to generate entity proposals. Then, each entity is paired together as a complete graph structure for relational reasoning. To build visual context, some studies use RNN, GNN, or attention mechanisms to propagate information between entities and their relations. Others exploit different types of prior knowledge, namely linguistic priors, statistical priors, and knowledge graphs. Motifs found that there are many fixed patterns in SGG, indicating that the relationship category is largely determined by the labels of the subject-object pairs. Loss reweighting, data augmentation, and unbiased representation learning are used to alleviate this problem. However, the two-stage method is built on an object detector with a large number of parameters, which incurs a lot of computational overhead and hinders end-to-end training. In addition, the two-stage method pairs all objects to generate triple proposals, resulting in quadratic growth in the solution search space.

[0005] Single-stage methods predict all relations at once by using an encoder-decoder architecture. These single-stage methods bypass the pairing between all object proposals and directly decode the <subject, predicate, object> triples, which will cost fewer parameters and computation. But using three decoders to predict subject, predicate, and object separately makes it difficult to extract coupled features shared in triples, while a single decoder has difficulty in obtaining decoupled features of different components of a triple. Therefore, it is necessary to design a unified decoder that takes task-specific queries as input, which not only extracts decoupled visual features separately, but also promotes complex interactions within triples. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a scene graph generation method based on a unified decoder in view of the defects in the prior art.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a scene graph generation method based on a unified decoder, comprising the following steps:

[0008] 1) Image feature extraction;

[0009] For a given original image Map it to high-level image features

[0010]

[0011] Among them, 3 represents 3 color channels, H o is the pixel height of the image, W0 is the pixel width of the image; C represents the number of channels of the feature map X, H, W correspond to the spatial dimensions of the feature map X;

[0012] Then, the Transformer encoder is used to expand the spatial size and extract more compact features for feature X And add position encoding at each layer

[0013] Where d is the dimension of the feature map Y;

[0014] 2) After extracting the image features, use the decoder to predict the relation triples;

[0015] The decoder consists of three fixed-size task-specific queries, namely the subject query Q s ∈R N×d 、Object query Q o ∈R N×d and the predicate query Q p ∈R N×d As input, a decoder with parameter sharing is used to generate task-specific representations at once;

[0016] The details are as follows:

[0017] 2.1) Generate three sets of task-specific queries: subject queries, object queries, and predicate queries; three sets of learned query embeddings Q s , Q o , Q p , each set of queries has N d-dimensional representations, s,i ,q o,i ,q p,i > represents the i-th triple query;

[0018] 2.2) For decoder layer l, a multi-layer perceptron MLP is used to connect the query output of the previous decoder layer (layer l-1);

[0019]

[0020] ​2.3) Run the self-attention mechanism to simulate the interaction between the subject, object, and predicate in the triple;

[0021] At a particular decoder layer l, The size of the task-specific three-tuple query is (bs, N, d), and they are reshaped into the dimension of (1, bs × N, d), and then concatenated into a triple query with dimension (3, bs × N, d). to organize each triple into a sequence;

[0022] Where bs is the number of batch sizes;

[0023] Run a self-attention mechanism on each triplet and explicitly model the interaction of spatial and semantic information between different subtasks;

[0024] Since the transformer architecture is position-invariant, the positional encoding shares the same shape as the query Goes through a similar deformation process as the query to form Then added to the input of each attention layer;

[0025] in, They represent the decoder layer l, Positional encoding;

[0026]

[0027] 2.4) capturing the context of the subject, object, and predicate separately through self-attention operations, and extracting visual features from the image representation in parallel through cross-attention operations to model disentangled features specific to each subtask;

[0028] Among them, the subject self-attention operation and cross-attention operation are expressed as follows:

[0029]

[0030] The output of the last layer of the decoder is to obtain a task-specific representation: Z s ,Z o ,Z p ;

[0031] 2.5) Share the parameters of the decoder to generate a task-specific representation Z at once x , x∈{s,o,p} is input into the feed-forward network FNN to generate a set of relation triple estimates

[0032]

[0033] in, represents the relationship triplet estimation obtained based on the task-specific query and the input image feature Y;

[0034] 3) The final generated scene graph is obtained based on the visual triples estimated from the relation triples.

[0035] According to the above scheme, in step 1), for a given original image I, ResNet is used to map it to a high-level image feature X.

[0036] According to the above scheme, the loss function used in the training of the encoder and decoder is the triple loss L t , expressed as:

[0037] L t =L s +L o +L p ;

[0038] Among them, L s , L o and L p Calculate according to the following formula:

[0039]

[0040] where x∈{s,o,p}, c x is the category of subject / object / predicate, i is the i-th triple, is the corresponding serial number of the predicted triple matched by the i-th real triple; The probability of correctly predicting the subject / object / predicate category for a predicted triple that matches a true triple, b x,i is the true bounding box of the subject / object / predicate, for The estimated value of L box is the L1 loss and the generalized IOU loss L GIOU A linear combination of .

[0041] According to the above scheme, a one-to-many paradigm is used for triple matching in model training, as follows:

[0042] Use K sets of N learnable task-specific queries, denoted as {Q x,1 ,Q x,2 ,...,Q x,K}, for x∈{s,o,p}, this correspondingly produces K groups of triplet predictions, with a one-to-one matching performed within each group, resulting in K predictions for each triplet label.

[0043] The beneficial effects produced by the present invention are:

[0044] 1. The present invention designs a unified decoder that takes three specific queries, namely subject, object and predicate, as input, which can effectively improve the performance compared with one decoder and one type of query.

[0045] 2. This paper proposes a novel single-stage scene graph generation architecture, built by a unified decoder with task-specific queries, which provides a usable solution to the weak coupling problem in relation triple prediction. Task-specific queries are used to locate entities separately, and semantic features are fused in triples to share coupled features, achieving better performance with fewer parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0047] Figure 1 is a method flow chart of an embodiment of the present invention;

[0048] Figure 2 It is a method principle diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] like Figure 1 As shown, a scene graph generation method based on a unified decoder comprises the following steps:

[0051] 1) Image feature extraction;

[0052] For a given original image Use a CNN backbone (such as ResNet) to map it to high-level image features

[0053] Among them, 3 represents 3 color channels, H o is the pixel height of the image, W0 is the pixel width of the image; C represents the number of channels of the feature map X, H, W correspond to the spatial dimensions of the low-resolution feature map X;

[0054] Then, the Transformer encoder is used to expand the spatial size and extract more compact features for feature X And add position encoding at each layer

[0055] Where d is the dimension of feature Y;

[0056] The image feature extractor can be formalized as:

[0057] Backbone(I)→X,Encoder(X)→Y;

[0058] 2) After extracting the image features, use the decoder to predict the relation triples;

[0059] Decoder(Q s ,Q o ,Q p ; Y)→Z s ,Z o ,Z p ;

[0060]

[0061] The decoder consists of three fixed-size task-specific queries, namely the subject query Q s ∈R N×d 、Object query Q o ∈R N×d and the predicate query Q p ∈R N×d As input, it generates a task-specific representation at once using a decoder with shared parameters;

[0062] The details are as follows:

[0063] 2.1) Generate three sets of task-specific queries: subject queries, object queries, and predicate queries; three sets of learned query embeddings Q s , Q o , Q p , each set of queries has N d-dimensional representations, s,i ,q o,i ,q p,i > represents the i-th triple query;

[0064] 2.2) For decoder layer l, a multi-layer perceptron MLP is used to connect the query output of the previous decoder layer (layer l-1);

[0065]

[0066] 2.3) Run the self-attention mechanism to simulate the interaction of subject / object / predicate within the triple;

[0067] For example, when provided with a relation triple such as <person, ride, horse>, the spatial location of “person” can help locate the object “horse”.

[0068] At a particular decoder layer l, The size of the task-specific three-tuple query is (bs, N, d), and they are reshaped into the dimension of (1, bs × N, d), and then concatenated into a triple query with dimension (3, bs × N, d).​ Organize each triple into a sequence;

[0069] Where bs is the number of batch sizes;

[0070] Run a self-attention mechanism on each triplet and explicitly model the interaction of spatial and semantic information between different subtasks;

[0071] Since the transformer architecture is position-invariant, the positional encoding shares the same shape as the query Goes through a similar deformation process as the query to form Then added to the input of each attention layer;

[0072] in, They represent the decoder layer l, Positional encoding;

[0073]

[0074] 2.4) capturing the context of the subject, object, and predicate separately through self-attention operations, and extracting visual features from the image representation in parallel through cross-attention operations to model disentangled features specific to each subtask;

[0075] Among them, the subject self-attention operation and cross-attention operation are expressed as follows, and the object and predicate forms are the same:

[0076]

[0077] The output of the last layer of the decoder is to obtain a task-specific representation: Z s ,Z o ,Z p ;

[0078] 2.5) Share the parameters of the decoder to generate a task-specific representation Z at once x , x∈{s,o,p} is input into the feed-forward network FNN to generate a set of relation triple estimates

[0079]

[0080] in, represents the relationship triplet estimation obtained based on the task-specific query and the input image feature Y;

[0081] The loss function used in encoder and decoder training is the triple loss L t , expressed as:

[0082] L t =L s +Lo +L p ;

[0083] Among them, L s , L o and L p Calculate according to the following formula:

[0084]

[0085] where x∈{s,o,p}, L box is the L1 loss and the generalized IOU loss L GIOU A linear combination of .

[0086] The one-to-many paradigm is used for triple matching in model training, as follows:

[0087] Use K sets of N learnable task-specific queries, denoted as {Q x,1 ,Q x,2 ,...,Q x,K}, for x∈{s,o,p}, this correspondingly produces K groups of triplet predictions, with a one-to-one matching performed within each group, resulting in K predictions for each triplet label.

[0088] 3) The final generated scene graph is obtained based on the visual triples estimated from the relation triples.

[0089] To demonstrate the effectiveness of our method, we compare it with three types of baselines:

[0090] 1. STA baseline: There is only one decoder and one type of query to predict the entire triplet.

[0091] 2. STS baseline: The approach taken by the present invention is to input task-specific queries into a unified decoder.

[0092] 3. TTS baseline: Use three decoders to predict subject, predicate and object respectively.

[0093] The experimental results are shown in Table 1. Compared with the STA baseline, the STS and TTS baselines achieve better performance, which proves the effectiveness of task-specific queries. In addition, the STS baseline can compete with the TTS baseline with nearly 25% fewer parameters, which means that multiple decoders are almost redundant.

[0094] Table 1 Baseline comparison experimental results

[0095]

[0096] The above experiments demonstrate the effectiveness of task-specific queries and unified decoders. In addition, we compare the proposed method with a two-stage method based on Faster R-CNN and a single-stage method based on DETR, as shown in Table 2.

[0097] As for the single-stage method comparison, the proposed method outperforms the state-of-the-art method IterSG by 0.5 / 0.8 and 0.8 / 1.1 in average recall@50 / 100 and recall@50 / 100, respectively. SGTR and IterSG adopt a top-k strategy to select more possible candidate triples. We reimplement this strategy by selecting the 3 most likely predicates for each entity pair to evaluate the model (* indicates that the top k predicted triples are retained for each true relation triple). The results show that the proposed method outperforms IterSG and SGTR (1.9 and 7.8) in coordinated recall@100. The proposed method also achieves the best performance on @20 / 50 / 100. In summary, compared with existing single-stage methods using ResNet-101 as the backbone, the performance of the proposed method is significantly improved with at least 28% fewer parameters. Compared with two-stage models with object detectors that have a large number of parameters and always use additional features, our method still outperforms most two-stage methods. The proposed method adopts a reweighted loss strategy and can compete with all unbiased two-stage methods. The unbiased training of the present invention achieves the highest recall rate @20 among all the methods.

[0098]

[0099]

[0100] This paper proposes a novel one-stage scene graph generation architecture, built by a unified decoder with task-specific queries. Our proposed method provides a viable solution to the weak coupling problem in relation triple prediction. We leverage task-specific queries to locate entities separately and fuse semantic features in triples to share coupled features. Extensive experimental results show that our method outperforms existing one-stage and two-stage methods with fewer parameters, demonstrating the effectiveness of our method.

[0101] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A scene graph generation method based on a unified decoder, characterized in that: The following steps are involved: 1) Image feature extraction; For a given original image Map it to high-level image features Among them, 3 represents 3 color channels, H o is the pixel height of the image, W0 is the pixel width of the image; C represents the number of channels of the feature map X, H, W correspond to the spatial dimensions of the feature map X; Then, the Transformer encoder is used to expand the spatial size and extract more compact features for feature X And add position encoding at each layer Where d is the dimension of feature Y; 2) After extracting the image features, use the decoder to predict the relation triples; The decoder consists of three fixed-size task-specific queries, namely the subject query Q s ∈R N×d 、Object query Q o ∈R N×d and the predicate query Q p ∈R N×d As input, it generates task-specific representations at once using a parameter-sharing decoder; The details are as follows: 2.1) Generate three sets of task-specific queries: subject queries, object queries, and predicate queries; three sets of learned query embeddings Q s , Q o , Q p , each set of queries has N d-dimensional representations, s,i ,q o,i ,q p,i > represents the i-th triple query;​ in, 2.2) For decoder layer l, a multi-layer perceptron MLP is used to connect the query output of the previous decoder layer; 2.3) Run the self-attention mechanism to simulate the interaction of subject / object / predicate within the triple; At a particular decoder layer l, The size of the task-specific three-tuple query is (bs, N, d), and they are reshaped into the dimension of (1, bs × N, d), and then concatenated into a triple query with dimension (3, bs × N, d). Organize each triple into a sequence; Where bs is the number of batch sizes; Run a self-attention mechanism on each triplet and explicitly model the interaction of spatial and semantic information between different subtasks; Since the transformer architecture is position-invariant, the positional encoding shares the same shape as the query Goes through a similar deformation process as the query to form Then added to the input of each attention layer; in, They represent the decoder layer l, Positional encoding; 2.4) capturing the context of the subject, object, and predicate separately through self-attention operations, and extracting visual features from the image representation in parallel through cross-attention operations to model disentangled features specific to each subtask; in, The subject self-attention operation and cross-attention operation are expressed as follows: The output of the last layer of the decoder is to obtain a task-specific representation: Z s ,Z o ,Z p ; 2.5) Share the parameters of the decoder to generate a task-specific representation Z at once x , x∈{s,o,p} is input into the feed-forward network FNN to generate a set of relation triple estimates in, represents the relationship triplet estimation obtained based on the task-specific query and the input image feature Y; 3) The final generated scene graph is obtained based on the visual triples estimated from the relation triples.

2. According to the scene graph generation method based on a unified decoder according to claim 1, it is characterized in that in the step 1), for a given original image I, ResNet is used to map it to a high-level image feature X.

3. The scene graph generation method based on a unified decoder according to claim 1, characterized in that: The loss function used in encoder and decoder training is the triple loss L t , expressed as: L t =L s +L o +L p ; Among them, L s , L o and L p Calculate according to the following formula: where x∈{s,o,p}, c x is the category of subject / object / predicate, i is the i-th triple, is the corresponding serial number of the predicted triple matched by the i-th real triple; The probability of correctly predicting the subject / object / predicate category for a predicted triple that matches a true triple, b x,i is the true bounding box of the subject / object / predicate, for The estimated value of L box is the L1 loss and the generalized IOU loss L GIOU A linear combination of .

4. The scene graph generation method based on a unified decoder according to claim 3, characterized in that: During training, a one-to-many paradigm is used for triple matching, as follows: Use K sets of N learnable task-specific queries, denoted as {Q x,1 ,Q x,2 ,...,Q x,K }, which correspondingly produces K sets of triplet predictions for x∈{s,o,p}. We perform a one-to-one matching within each set, resulting in K predictions for each triplet label.

5. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images