Scene graph generation methods, devices and media

By using target semantic priors and graph structure learning theory, a sparse scene graph is constructed, which solves the problem of insufficient relation feature modeling in existing technologies and improves the accuracy and quality of scene graph generation.

CN118070192BActive Publication Date: 2026-11-10SHANGHAI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410300560.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-15
Publication Date
2026-11-10
Estimated Expiration
2044-03-15

AI Technical Summary

Technical Problem

In existing scene graph generation methods, the relationship features between targets are not sufficiently modeled, leading to information loss and irrelevant information transmission during information fusion, which affects the generation effect.

Method used

By constructing relational features based on prior semantic knowledge of the target, and utilizing graph structure learning theory to reduce redundant relations and convey key target information, visual-spatial multimodal relational features and sparse scene graph generation methods are employed to reduce the impact of noise.

Benefits of technology

It improves the accuracy and quality of scene graph generation, reduces the impact of noise information, conveys more effective features, and enhances the semantic feature learning of entities and relationships.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118070192B_ABST
    Figure CN118070192B_ABST
Patent Text Reader

Abstract

This invention relates to a method, device, and medium for generating scene graphs. The method includes: S1, extracting target information from an input image using a target detector, including detection boxes and category information; S2, extracting visual relationship features and spatial relationship features between targets from the target information; S3, fusing the visual relationship features and the spatial relationship features based on semantic priors between targets to form multimodal relationship features; S4, performing binary classification on the multimodal relationship features using a graph structure learning module to construct a sparse scene graph; S5, calculating node weights based on the connection density of the nodes in the sparse scene graph, and aggregating features based on node weights during context message passing; S6, inputting the node features and edge features in the sparse scene graph into a classifier for classification, forming a triplet representation, and thus forming the scene graph. Compared with existing technologies, this invention achieves high-quality scene graph generation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method, apparatus and medium for generating scene graphs. Background Technology

[0002] Scene graph generation (SGG), as an intermediate-level visual understanding task, is crucial for various visual-language cross-modal understanding tasks, such as image captioning, visual question answering (VQA), and image retrieval. Even recently, related research has begun to emerge in the field of robotic environments, and recent work indicates that SGG technology has potential applications in a wider range of fields.

[0003] One of the core challenges in scene graph generation is modeling the relationships between objects. Initially, the field used visual features of joint regions to represent these relationships, relying solely on visual characteristics. Later, explorations began focusing on modeling relationships through the spatial distribution of objects. Spatial distribution not only reflects positional relationships but also describes structural information. Region attributes are used to describe the spatial distribution of objects, including positional, size, distance, and shape relationships. Of course, influenced by human language habits, semantic priors significantly impact the distribution of relationships, and the incorporation of semantic features is quite common in current methods.

[0004] While various modalities have been introduced to construct relational features, refining this information remains a significant challenge in relational modeling. Numerous methods have been proposed to refine the representation of relational features. These can be broadly categorized as follows: First, attention-based methods leverage the powerful associative capabilities of attention mechanisms to learn the relationships between features, refining the representation of features in the scene graph. Second, translational embedding methods model relations as vector representations by embedding target features into a low-dimensional space. Third, conditional random field-based methods utilize the strong statistical correlation between relational predicates and objects to model relational features. Fourth, scene graph generation methods based on convolutional neural networks (CNNs) attempt to extract local and global visual features from images using CNNs and then classify the relationships between objects.

[0005] While many methods can generate scene graphs, these methods still have some problems:

[0006] 1) Insufficient modeling of the relationship features between targets. Many methods use simple feature fusion to form relationship features, resulting in information loss during the fusion process.

[0007] 2) Many methods construct dense scene graphs for reasoning; however, this method allows irrelevant information to be passed through the scene graph, affecting the quality of scene graph generation. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a scene graph generation method, device and medium. By constructing the features of relationships through the prior semantics of the target, and by using graph structure learning theory to reduce redundant relationships, more information of key targets is conveyed to reduce the influence of noise information, thereby accurately learning the semantic features of entities and relationships and improving the quality of generated scene graphs.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] According to a first aspect of the present invention, a scene graph generation method is provided, the method comprising:

[0011] S1. Extract target information from the input image using a target detector, wherein the target information includes a detection box and category information;

[0012] S2. Extract the visual relationship features of the targets and the spatial relationship features between the targets from the target information;

[0013] S3. Based on the semantic prior between targets, the visual relationship features and the spatial relationship features are fused to form visual-spatial multimodal relationship features;

[0014] S4. Use the graph structure learning module to perform binary classification on the multimodal relationship features and construct a sparse scene graph;

[0015] S5. Calculate the node weights based on the connection density of the sparse scene graph nodes, and aggregate features based on the node weights during context message passing.

[0016] S6. Input the node features and edge features in the sparse scene graph into the classifier for classification, form a triplet representation, and form the scene graph.

[0017] Preferably, S1 specifically comprises:

[0018] The features F of the input image I are extracted using a pre-trained convolutional neural network;

[0019] The feature F is input into the Region Proposal Network (RPN) to obtain the candidate bounding box b = (x, y, w, h) and category information c of the target, where (x, y) represents the coordinates of the bounding box and (w, h) represents the width and height of the bounding box.

[0020] Preferably, the convolutional neural network in S1 includes VGG and ResNet.

[0021] Preferably, in step S2, the interest pooling layer extracts the visual features and visual relationship features of the target based on the target region, and forms the spatial relationship features between the targets based on the position information of the bounding boxes, specifically including the following sub-steps:

[0022] S21. Based on the detection bounding box information of paired targets, construct joint region features to represent visual relationship features:

[0023]

[0024] In the formula: ROI represents the ROI pooling layer; the union operation calculates the union region of two bounding boxes;

[0025] S22. Based on the target's location information, calculate the relative position information of the two targets as spatial relationship features:

[0026]

[0027] In the formula: (x,y,w,h)∈b i σ represents the position coordinates of the object; σ represents the linear layer, used to transform the feature space.

[0028] Preferably, step S3 specifically comprises:

[0029] S31. Given subject object i and object object j, calculate the semantic matrix x between paired targets. ij As a representation of semantic relevance between objects:

[0030]

[0031] In the formula: It is matrix multiplication; t i t j This indicates the tag information of the target object;

[0032] S32. Concatenate visual relationship features and spatial relationship features by channel to obtain visual-spatial multimodal relationship features.

[0033]

[0034] S32, through a series of two-dimensional convolutions and semantic matrix x ij Spatial pooling is used to construct channel attention, resulting in semantic association attention vectors. Channel operations are performed using the obtained semantic association attention vector and multimodal relation features to obtain updated visual-spatial multimodal relation features.

[0035]

[0036]

[0037] In the formula: C represents the multimodal relationship feature. The number of channels; σ is the activation function; G conv It is a two-dimensional convolution operation; G pooling It is a two-dimensional pooling operation; × represents the dot product.

[0038] Preferably, S4 specifically comprises:

[0039] Updated visual-spatial multimodal relationship features Input is fed into the graph structure learning module to predict the target o. i and target o j The correlation between them is determined and normalized using softmax.

[0040]

[0041]

[0042] In the formula: τ(·) is the gate function corresponding to the graph structure learning module, which is a module composed of a series of fully connected layers, and softmax maps the values ​​to the range [0,1]; α and β are two learnable parameters.

[0043] Preferably, S5 specifically comprises:

[0044] S51. Calculate the degree of each node o in the sparse scene graph:

[0045]

[0046] S52, Using node o i Visual-spatial multimodal relationship features Initialize hidden state:

[0047] The formal expression for key entity context messaging is:

[0048]

[0049]

[0050]

[0051] In the formula: The characteristics of the node after message passing at layer l; N represents the characteristics of the relationship after message passing at level l; *i Represents node o iThe neighborhood of ; Z is the weighting function of node degree; g e→e g e→r g s g o These are all multilayer perceptrons used to refine the hidden layer table; λ represents a learnable value.

[0052] Preferably, in step S6, the node features and edge features in the sparse scene graph are input into a classifier for classification, forming a triple representation and thus a scene graph. Specifically, the expressions for predicting the target features and the predicting the relationships between targets are as follows:

[0053] The target feature output prediction expression is:

[0054]

[0055] In the formula: It is the node characteristic of the last message transmission, r o It is the output of the predicted target category;

[0056] The output prediction expression for the relationship features between targets is:

[0057]

[0058] In the formula: It is the edge feature of the last message transmission, r r It is the output of the predicted relationship.

[0059] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods described above.

[0060] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1) This invention constructs relational features through prior semantic knowledge of the target, utilizes graph structure learning theory to reduce redundant relations, and conveys more information about key targets to reduce the impact of noise information, thereby accurately learning the semantic features of entities and relations and improving the quality of generated scene graphs.

[0063] 2) The semantically guided multimodal fusion module was used to model the multimodal features between paired targets, forming multimodal relationship features with target semantic priors. In the subsequent feature refinement process, more effective features were transmitted.

[0064] 3) By drawing on graph structure learning theory, graph connections are sparsified before contextual information is passed to avoid the influence of non-existent relationships.

[0065] 4) Utilize the issues arising from node equivalence during the slow transmission of key entity messages to reduce interference caused by noisy data transmission. Attached Figure Description

[0066] Figure 1 This is a flowchart of the method of the present invention;

[0067] Figure 2 This is an overall flowchart of the method in the embodiment;

[0068] Figure 3 This is a diagram illustrating the scene graph generation implementation framework in the embodiment.

[0069] Figure 4 This is a structural diagram of the graph structure learning module in the embodiment. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0071] Example

[0072] like Figure 1 and Figure 2 As shown, this embodiment provides a method for generating scene graphs, which includes the following steps:

[0073] S1. Extract target information from the input image using a target detector, wherein the target information includes detection bounding boxes and category information; specifically, this includes the following sub-steps:

[0074] S11. Input the images from the Visual Genome dataset into a pre-trained convolutional network to extract the image features F = CNN(I). The CNN used here is ResNet-X101 as the backbone, with a final output dimension of 7×7×2048. The VG dataset contains 108,077 data points, including 3.8 million target entities and 230 relation entities, encompassing 200 target categories and 50 relation categories.

[0075] S12. Then, the features are input into the Region Proposal Network (RPN) to obtain the candidate bounding boxes b = (x, y, w, h) and class information c, where (x, y) represent the coordinates of the bounding boxes, and (w, h) represent the width and height of the bounding boxes. In the RPN network, the anchor_size is set to (31, 64, 128, 256, 512), the aspect_ratios is set to (0.23, 0.63, 1.28, 3.51), and the feature dimension of the hidden layer is 256.

[0076] S2. Using the target information in S1, form the visual relationship features of the targets and the spatial relationship features between the targets, specifically including the following sub-steps:

[0077] S21. Based on the detection bounding box information of paired targets, construct a feature model of the joint region to represent visual relationship features:

[0078]

[0079] In the formula: ROI represents the ROI pooling layer, the union operation calculates the union region of the two bounding boxes, the output dimension of the ROI pooling layer is 4096, and the pooling size is 7×7.

[0080] S22. Based on the target's location information, calculate the relative position information of the two targets as spatial relationship features:

[0081]

[0082] In the formula: (x,y,w,h)∈b i σ represents the position coordinates of the object; σ represents the linear layer, which is used to transform the feature space. The transformed spatial relationship features are 512-dimensional.

[0083] S3. Based on the semantic priors between targets, fuse visual relationship features and spatial relationship features to form multimodal relationship features, which specifically includes the following sub-steps:

[0084] S31. Given a subject object i and an object j, calculate the semantic matrix x between the pairs of objects. ij As a representation of semantic relevance between objects:

[0085]

[0086] In the formula: It is matrix multiplication, t i t j This embodiment uses GLOVE to encode the semantic information of the target object, and its weight initialization dimension w represents the label information of the target object. s w o we Finally, x ij The feature dimension is 512.

[0087] S32. By concatenating visual relationship features and spatial relationship features according to channels, a representation of visual and spatial multimodal relationship features is obtained.

[0088]

[0089] S33, through a series of two-dimensional convolutions and x ij Spatial pooling is used to implement channel attention, resulting in attention vectors. Channel operations are performed using the obtained semantic association attention vector and multi-modal other-relation features to update the feature representation of the relation features and select feature patterns with more semantic association.

[0090]

[0091]

[0092] In the formula: C represents the multimodal relationship feature. The number of channels, σ is the activation function, G conv It is a two-dimensional convolution operation, G pooling It is a two-dimensional pooling operation, and the final result is... The value represents the updated relational features, and × represents the dot product. A 3×3 convolutional block is selected to generate a 512-dimensional feature attention vector.

[0093] S4. Borrowing from graph structure learning theory, sparsify the connections in the scene graph to reduce redundant connections. (See attached...) Figure 4 Its structure is shown, specifically including the following sub-steps:

[0094] S41. Construct multimodal relationship features As input, and predict the target o i and target o j The correlation between them was determined, and normalization was performed using softmax.

[0095]

[0096] The graph structure learning module is a module consisting of a series of fully connected layers. Each layer has a ReLU non-linear activation layer, and softmax maps values ​​to the range [0,1]. The output dimension of each linear layer in the graph structure learning module is (512, 256, 2).

[0097] S42. To control the high or low confidence scores, these scores are input into the gating function τ(·):

[0098]

[0099] In the above expression, the τ function controls the range of the fraction, and α and β are two learnable parameters initialized to 2.2 and 0.025, respectively.

[0100] S5. Calculate the importance of the sparse scene graph based on the connection density of the nodes in the graph, and aggregate features based on the importance of the current node in the context message passing.

[0101] S51. Calculate the degree of each node o in the sparse scene graph:

[0102]

[0103] S52, using o i and To initialize the hidden state: Therefore, the formal expression for contextual message passing of key entities is, where o i , Both are 512-dimensional feature vectors:

[0104]

[0105]

[0106]

[0107] In the formula: This represents the characteristics of the node after message passing at layer l. N represents the characteristics of the relationship after message passing at level l. *i Represents o i The neighborhood of a node. Z is the weighted function of node degree, g e→e g e→r g s g o These are all multilayer perceptrons used to refine the hidden layer representation; λ represents a learnable value, which we set to 0.5 initially.

[0108] S6. Input the node features and edge features in the graph into the classifier for classification, and identify the corresponding triplet representations.<subject,predicate,object> To create a scene diagram, the following sub-steps are involved:

[0109] S61. Predict the output based on the obtained target features:

[0110]

[0111] In the formula: It is the node characteristic of the last message transmission, r o The output is the predicted target category, W∈R 512 ×200 It is a weight matrix;

[0112] S62. Predict the output based on the obtained relationship features between the targets:

[0113]

[0114] In the formula: It is the edge feature of the last message transmission, r r It is a predictive relation, where W∈R 512×50 is the dimension of the weight matrix.

[0115] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0116] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0117] The processing unit executes the various methods and processes described above, such as methods S1 to S6. For example, in some embodiments, methods S1 to S6 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S6 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S6 by any other suitable means (e.g., by means of firmware).

[0118] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0119] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating scene graphs, characterized in that, The method includes: S1. Extract target information from the input image using a target detector, wherein the target information includes a detection box and category information; S2. Extract the visual relationship features of the targets and the spatial relationship features between the targets from the target information; S3. Based on the semantic prior between targets, the visual relationship features and the spatial relationship features are fused to form visual-spatial multimodal relationship features, specifically: S31. Given a subject object and object Calculate the semantic matrix between paired targets. As a representation of semantic relevance between objects: (3) In the formula: ⨂ represents matrix multiplication; , This indicates the tag information of the target object; S32. Concatenate visual relationship features and spatial relationship features by channel to obtain visual-spatial multimodal relationship features. : (4) S32, through a series of two-dimensional convolutions and semantic matrices Spatial pooling is used to construct channel attention, resulting in semantic association attention vectors. Channel operations are performed using the obtained semantic association attention vector and multimodal relationship features to obtain updated visual-spatial multimodal relationship features. : (5) (6) In the formula: C represents the multimodal relationship feature. The number of channels; It is an activation function; It is a two-dimensional convolution operation; It is a two-dimensional pooling operation; Represents the dot product; S4. Utilize the graph structure learning module to perform binary classification on the multimodal relationship features, and construct a sparse scene graph, specifically as follows: Updated visual-spatial multimodal relationship features Input is fed into the graph structure learning module to predict the target. and target The correlation between them, and using Perform normalization processing; (7) (8) In the formula: Here is the gate function corresponding to the graph structure learning module, which is a module composed of a series of fully connected layers. Map the values ​​to the range [0, 1]. These are two learnable parameters; S5. Calculate the node weights based on the connection density of the sparse scene graph nodes, and aggregate features based on the node weights during context message passing. S6. Input the node features and edge features in the sparse scene graph into the classifier for classification, form a triplet representation, and form the scene graph.

2. The scene graph generation method according to claim 1, characterized in that, Specifically, S1 is: The input image is extracted using a pre-trained convolutional neural network. Features ; Features Input the data into the Region Proposal Network (RPN) to obtain candidate bounding boxes for the target. and category information ,in( Represents the coordinates of the bounding box. Represents the width and height of the bounding box.

3. The scene graph generation method according to claim 2, characterized in that, The convolutional neural networks in S1 include VGG and ResNet.

4. The scene graph generation method according to claim 1, characterized in that, In step S2, the interest pooling layer extracts the visual features and visual relationship features of the target based on the target region, and forms the spatial relationship features between targets based on the position information of the bounding box. Specifically, it includes the following sub-steps: S21. Based on the detection bounding box information of paired targets, construct joint region features to represent visual relationship features: (1) In the formula: ROI represents the ROI pooling layer; the union operation calculates the union region of two bounding boxes; S22. Based on the target's location information, calculate the relative position information of the two targets as spatial relationship features: (2) In the formula: This indicates the object's position coordinates. This represents a linear layer used to transform the feature space.

5. The scene graph generation method according to claim 1, characterized in that, Specifically, S5 is: S51. Calculate each node in a sparse scene graph. Degree: (9) S52, Using Nodes Visual-spatial multimodal relationship features Initialize hidden state: , ; The formal expression for key entity context messaging is: (10) (11) (12) In the formula: Representing the Characteristics of nodes after layer message passing; Representing the Characteristics of relationships after layer message passing; Represents a node The neighborhood; It is a weighting function for node degree; , , , Both are multilayer perceptrons, used to refine the hidden layer table; This represents a learnable value.

6. The scene graph generation method according to claim 5, characterized in that, In step S6, the node features and edge features in the sparse scene graph are input into a classifier for classification, forming a triplet representation and thus a scene graph. Specifically, the expressions for predicting target features and predicting the relationships between targets are as follows: The target feature output prediction expression is: (13) In the formula: It is the node characteristic of the last message transmission. It is the output of the predicted target category; The output prediction expression for the relationship features between targets is: (14) In the formula: It is the edge feature of the last message transmission. It is the output of the predicted relationship.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.