A scene graph generation method based on unified decoder

Abstract

This paper discloses a method for scene graph generation based on a unified decoder, comprising the following steps: 1) image feature extraction; 2) predicting relation triplets using the decoder after extracting the image features; and 3) generating a final scene graph based on the visual triplets estimated from the relation triplets. This paper proposes a novel single-stage scene graph generation architecture, constructed using a unified decoder with task-specific queries. This architecture provides a viable solution to the weak coupling problem in relation triple prediction, leveraging task-specific queries to individually locate entities and fusing semantic features in triples to share coupled features, achieving superior performance with fewer parameters.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a scene graph generation method based on a unified decoder. Background Art

[0002] The scene graph generation task is an extremely important part of visual understanding. It can help computers better understand image content. By extracting objects and relationships in images, it can gain a deeper understanding of scene structure and semantic information, providing richer input for other visual tasks.

[0003] Existing scene graph generation (SGG) methods can be roughly divided into two categories: two-stage based methods and single-stage based methods.

[0004] Two-stage methods typically use a pre-trained object detector (e.g., Faster R-CNN) to generate entity proposals. Each entity is then paired together as a complete graph structure for relational reasoning. To build visual context, some studies use RNNs, GNNs, or attention mechanisms to propagate information between entities and their relations. Others leverage different types of prior knowledge, namely linguistic priors, statistical priors, and knowledge graphs. Motifs found many fixed patterns in SGG, indicating that the relation category is largely determined by the labels of the subject-object pairs. Loss reweighting, data augmentation, and unbiased representation learning are used to alleviate this problem. However, two-stage methods are built on object detectors with a large number of parameters, which incurs a large computational overhead and hinders end-to-end training. In addition, two-stage methods pair all objects to generate triple proposals, resulting in quadratic growth in the solution search space.

[0005] Single-stage methods predict all relations at once by using an encoder-decoder architecture. These single-stage methods bypass all pairings between object proposals and directly decode <subject, predicate, object> triples, which consumes fewer parameters and computation. However, using three decoders to predict the subject, predicate, and object separately makes it difficult to extract the coupled features shared in the triple, while a single decoder struggles to obtain decoupled features of the different components of the triple. Therefore, it is necessary to design a unified decoder that takes a task-specific query as input, not only extracting decoupled visual features separately, but also promoting complex interactions within the triple. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a scene graph generation method based on a unified decoder in view of the defects in the prior art.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a scene graph generation method based on a unified decoder, comprising the following steps:

[0008] 1) Image feature extraction;

[0009] For a given original image Map it to high-level image features

[0010]

[0011] Among them, 3 represents 3 color channels, H o is the pixel height of the image, W0 is the pixel width of the image; C represents the number of channels of the feature map X, H, W corresponds to the spatial dimensions of the feature map X;

[0012] Then, the Transformer encoder is used to expand the spatial size and extract more compact features for feature X And add position encoding at each layer

[0013] Where d is the dimension of the feature map Y;

[0014] 2) After extracting image features, use the decoder to predict relation triplets;

[0015] The decoder consists of three fixed-size task-specific queries, namely the subject query Q s ∈R N×d 、Object query Q o ∈R N×d and predicate query Q p ∈R N×d As input, the decoder generates a task-specific representation at once using parameter sharing;

[0016] The details are as follows:

[0017] 2.1) Generate three sets of task-specific queries: subject queries, object queries, and predicate queries; three sets of learned query embeddings Q s , Q o , Q p , each set of queries has N d-dimensional representations, s,i ,q o,i ,q p,i > represents the i-th triple query;

[0018] 2.2) For decoder layer l, a multilayer perceptron (MLP) is used to connect the query output of the previous decoder layer (layer l-1);

[0019]

[0020] ​2.3) Run the self-attention mechanism to simulate the interaction between the subject, object, and predicate in the triple;

[0021] At a specific decoder layer l, The size of the task-specific three-tuple query is (bs, N, d), and the three-tuple query is reshaped into the dimension of (1, bs × N, d), and then they are concatenated into a triple query with the dimension of (3, bs × N, d). to organize each triple into a sequence;

[0022] Where bs is the number of batch sizes;

[0023] Run a self-attention mechanism on each triplet and explicitly model the interaction of spatial and semantic information between different subtasks;

[0024] Since the transformer architecture is position-invariant, the positional encoding shares the same shape as the query Goes through a similar transformation process as the query to form Then added to the input of each attention layer;

[0025] in, Respectively represent the decoder layer l, Positional encoding;

[0026]

[0027] 2.4) capturing the context of the subject, object, and predicate separately through self-attention operations, and extracting visual features from the image representation in parallel through cross-attention operations to model disentangled features specific to each subtask;

[0028] Among them, the subject self-attention operation and cross-attention operation are expressed as follows:

[0029]

[0030] The output of the last layer of the decoder is to obtain a task-specific representation: Z s ,Z o ,Z p ;

[0031] 2.5) Share the parameters of the decoder to generate a task-specific representation Z at once x , x∈{s, o, p} is input into the feedforward network FNN to generate a set of relationship triples estimation

[0032]

[0033] in, represents the relation triplet estimation obtained based on the task-specific query and the input image feature Y;

[0034] 3) The final scene graph is obtained based on the visual triples estimated from the relation triples.

[0035] According to the above scheme, in step 1), for a given original image I, ResNet is used to map it to a high-level image feature X.

[0036] According to the above scheme, the loss function used in the training of the encoder and decoder is the triple loss L t , expressed as:

[0037] L t =L s +L o +L p ;

[0038] Among them, L s , L o and L p Calculate according to the following formula:

[0039]

[0040] where x∈{s,o,p}, c x is the category of subject / object / predicate, i is the i-th triple, is the corresponding serial number of the predicted triple matched by the i-th real triple; The probability of correctly predicting the subject / object / predicate category for a predicted triple that matches a true triple, b x,i is the true bounding box of the subject / object / predicate, for The estimated value of L box is the L1 loss and the generalized IOU loss L GIOU A linear combination of .

[0041] According to the above scheme, a one-to-many paradigm is used for triple matching in model training, as follows:

[0042] Use K sets of N learnable task-specific queries, denoted as {Q x,1 ,Q x,2 ,...,Q x,K}, for x∈{s,o,p}, this correspondingly produces K groups of triplet predictions, with a one-to-one matching performed within each group, resulting in K predictions for each triplet label.

[0043] The beneficial effects produced by the present invention are:

[0044] 1. The present invention designs a unified decoder that takes three specific queries, namely subject, object and predicate, as input, which can effectively improve the performance compared with one decoder and one type of query.

[0045] 2. This paper proposes a novel single-stage scene graph generation architecture, built from a unified decoder with task-specific queries, providing a viable solution to the weak coupling problem in relation triple prediction. By leveraging task-specific queries to individually localize entities and fusing semantic features across triples to share coupled features, we achieve superior performance with fewer parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0047] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0048] Figure 2 1 is a method principle diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0050] like Figure 1 As shown, a scene graph generation method based on a unified decoder includes the following steps:

[0051] 1) Image feature extraction;

[0052] For a given original image Use a CNN backbone (such as ResNet) to map it to high-level image features

[0053] Among them, 3 represents 3 color channels, H o is the pixel height of the image, W0 is the pixel width of the image; C represents the number of channels of the feature map X, H, W correspond to the spatial dimensions of the low-resolution feature map X;

[0054] Then, the Transformer encoder is used to expand the spatial size and extract more compact features for feature X And add position encoding at each layer

[0055] Where d is the dimension of feature Y;

[0056] The image feature extractor can be formalized as:

[0057] Backbone(I)→X,Encoder(X)→Y;

[0058] 2) After extracting image features, use the decoder to predict relation triplets;

[0059] Decoder(Q s ,Q o ,Q p ; Y)→Z s ,Z o ,Z p ;

[0060]

[0061] The decoder consists of three fixed-size task-specific queries, namely the subject query Q s ∈R N×d 、Object query Q o ∈R N×d and predicate query Q p ∈R N×d As input, it generates a task-specific representation at once using a decoder with shared parameters;

[0062] The details are as follows:

[0063] 2.1) Generate three sets of task-specific queries: subject queries, object queries, and predicate queries; three sets of learned query embeddings Q s , Q o , Q p , each set of queries has N d-dimensional representations, s,i ,q o,i ,q p,i > represents the i-th triple query;

[0064] 2.2) For decoder layer l, a multilayer perceptron (MLP) is used to connect the query output of the previous decoder layer (layer l-1);

[0065]

[0066] 2.3) Run the self-attention mechanism to simulate the interaction between subject / object / predicate in the triple;

[0067] For example, when provided with a relation triple such as <person, ride, horse>, the spatial location of “person” can help locate the object “horse”.

[0068] At a specific decoder layer l, The size of the task-specific three-tuple query is (bs, N, d), and the three-tuple query is reshaped into the dimension of (1, bs × N, d), and then they are concatenated into a triple query with the dimension of (3, bs × N, d).​ Organize each triple into a sequence;

[0069] Where bs is the batch size;

[0070] Run a self-attention mechanism on each triplet and explicitly model the interaction of spatial and semantic information between different subtasks;

[0071] Since the transformer architecture is position-invariant, the positional encoding shares the same shape as the query Goes through a similar deformation process as the query to form Then added to the input of each attention layer;

[0072] in, Respectively represent the decoder layer l, Positional encoding;

[0073]

[0074] 2.4) capturing the context of the subject, object, and predicate separately through self-attention operations, and extracting visual features from the image representation in parallel through cross-attention operations to model disentangled features specific to each subtask;

[0075] Among them, the subject self-attention operation and cross-attention operation are expressed as follows, and the object and predicate forms are the same:

[0076]

[0077] The output of the last layer of the decoder is to obtain a task-specific representation: Z s ,Z o ,Z p ;

[0078] 2.5) Share the parameters of the decoder to generate a task-specific representation Z at once x , x∈{s, o, p} is input into the feedforward network FNN to generate a set of relationship triples estimation

[0079]

[0080] in, represents the relation triplet estimation obtained based on the task-specific query and the input image feature Y;

[0081] The loss function used in encoder and decoder training is triplet loss L t , expressed as:

[0082] L t =L s +Lo +L p ;

[0083] Among them, L s , L o and L p Calculate according to the following formula:

[0084]

[0085] where x∈{s,o,p}, L box is the L1 loss and the generalized IOU loss L GIOU A linear combination of .

[0086] The one-to-many paradigm is used for triple matching in model training, as follows:

[0087] Use K sets of N learnable task-specific queries, denoted as {Q x,1 ,Q x,2 ,...,Q x,K}, for x∈{s,o,p}, this correspondingly produces K groups of triplet predictions, with a one-to-one matching performed within each group, resulting in K predictions for each triplet label.

[0088] 3) The final scene graph is obtained based on the visual triples estimated from the relation triples.

[0089] To demonstrate the effectiveness of our method, we compare it with three types of baselines:

[0090] 1. STA baseline: There is only one decoder and one type of query to predict the entire triplet.

[0091] 2. STS baseline: The approach taken by the present invention is to input task-specific queries into a unified decoder.

[0092] 3. TTS baseline: Use three decoders to predict subject, predicate and object respectively.

[0093] The experimental results are shown in Table 1. Compared with the STA baseline, the STS and TTS baselines achieve better performance, which demonstrates the effectiveness of task-specific queries. In addition, the STS baseline can compete with the TTS baseline with nearly 25% fewer parameters, which means that the multiple decoders are almost redundant.

[0094] Table 1 Baseline comparison experimental results

[0095]

[0096] The above experiments demonstrate the effectiveness of task-specific queries and unified decoders. In addition, we compare the proposed method with a two-stage method based on Faster R-CNN and a single-stage method based on DETR, as shown in Table 2.

[0097] Compared to single-stage methods, our proposed method outperforms the state-of-the-art method IterSG, achieving average recall@50 / 100 and recall@50 / 100 exceeding by 0.5 / 0.8 and 0.8 / 1.1, respectively. SGTR and IterSG employ a top-k strategy to select more likely candidate triples. We reimplement this strategy by selecting the three most likely predicates for each entity pair to evaluate the model (* denotes retaining the top k predicted triples for each true relation triple). Results show that our proposed method surpasses IterSG and SGTR by 1.9 and 7.8% in terms of coordinated recall@100. We also achieve state-of-the-art performance at @20 / 50 / 100. In summary, our proposed method significantly improves performance compared to existing single-stage methods using ResNet-101 as the backbone, while reducing parameters by at least 28%. Compared to two-stage models with large object detectors and consistent use of additional features, our method still outperforms most two-stage methods. Our proposed method, employing a reweighted loss strategy, is competitive with all unbiased two-stage methods. The unbiased training of our invention achieves the highest recall rate@20 among all methods.

[0098]

[0099]

[0100] This paper proposes a novel single-stage scene graph generation architecture, built from a unified decoder with task-specific queries. Our approach provides a viable solution to the weak coupling problem in relation triple prediction. We leverage task-specific queries to individually localize entities and fuse semantic features across triples to share coupled features. Extensive experimental results demonstrate that our approach outperforms existing one-stage and two-stage methods with fewer parameters, demonstrating its effectiveness.

[0101] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A scene graph generation method based on a unified decoder, characterized in that: The following steps are involved: 1) Image feature extraction; For a given original image Map it to high-level image features Among them, 3 represents 3 color channels, H o is the pixel height of the image, W0 is the pixel width of the image; C represents the number of channels of the feature map X, H, W corresponds to the spatial dimensions of the feature map X; Then, the Transformer encoder is used to expand the spatial size and extract more compact features for feature X And add position encoding at each layer Where d is the dimension of feature Y; 2) After extracting image features, use the decoder to predict relation triplets; The decoder consists of three fixed-size task-specific queries, namely the subject query Q s ∈R N×d 、Object query Q o ∈R N×d and predicate query Q p ∈R N×d As input, it generates task-specific representations at once using a parameter-sharing decoder; The details are as follows: 2.1) Generate three sets of task-specific queries: subject queries, object queries, and predicate queries; three sets of learned query embeddings Q s , Q o , Q p , each set of queries has N d-dimensional representations, s,i ,q o,i ,q p,i > represents the i-th triple query;​ in, 2.2) For decoder layer 1, a multi-layer perceptron (MLP) is used to connect the query output of the previous decoder layer; 2.3) Run the self-attention mechanism to simulate the interaction between subject / object / predicate in the triple; At a specific decoder layer l, The size of the task-specific three-tuple query is (bs, N, d), and the three-tuple query is reshaped into the dimension of (1, bs × N, d), and then they are concatenated into a triple query with the dimension of (3, bs × N, d). Organize each triple into a sequence; Where bs is the number of batch sizes; Run a self-attention mechanism on each triplet and explicitly model the interaction of spatial and semantic information between different subtasks; Since the transformer architecture is position-invariant, the positional encoding shares the same shape as the query Goes through a similar deformation process as the query to form Then added to the input of each attention layer; in, Respectively represent the decoder layer l, Positional encoding; 2.4) capturing the context of the subject, object, and predicate separately through self-attention operations, and extracting visual features from the image representation in parallel through cross-attention operations to model disentangled features specific to each subtask; in, The subject self-attention operation and cross-attention operation are expressed as follows: The output of the last layer of the decoder is to obtain a task-specific representation: Z s ,Z o ,Z p ; 2.5) Share the parameters of the decoder to generate a task-specific representation Z at once x , x∈{s, o, p} is input into the feedforward network FNN to generate a set of relationship triples estimation in, represents the relation triplet estimation obtained based on the task-specific query and the input image feature Y; 3) The final scene graph is obtained based on the visual triples estimated from the relation triples.

2. The scene graph generation method based on a unified decoder according to claim 1 is characterized in that in step 1), for a given original image I, ResNet is used to map it to a high-level image feature X.

3. The scene graph generation method based on a unified decoder according to claim 1, characterized in that: The loss function used in encoder and decoder training is triplet loss L t , expressed as: L t =L s +L o +L p ; Among them, L s 、L o and L p Calculate according to the following formula: where x∈{s,o,p}, c x is the category of subject / object / predicate, i is the i-th triple, is the corresponding serial number of the predicted triple matched by the i-th real triple; The probability of correctly predicting the subject / object / predicate category for a predicted triple that matches a true triple, b x,i is the true bounding box of the subject / object / predicate, for The estimated value of L box is the L1 loss and the generalized IOU loss L GIOU A linear combination of .

4. The scene graph generation method based on a unified decoder according to claim 3, characterized in that: During training, a one-to-many paradigm is used for triple matching, as follows: Use K sets of N learnable task-specific queries, denoted as {Q x,1 ,Q x,2 ,...,Q x,K }, for x∈{s,o,p}, this correspondingly produces K groups of triplet predictions, and we perform a one-to-one matching within each group, thereby matching K predictions for each triplet label.

5. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images