A spatial relationship recognition method based on graph convolutional neural network reasoning

Abstract

The present invention discloses a spatial relationship recognition method based on graph convolutional neural network reasoning. The method is characterized by constructing a spatial relationship graph of three spatial features based on a multi-spatial feature extraction method, enhancing the extracted spatial features using a linear layer network, extracting spatial features to construct a spatial relationship graph, and then using a graph convolutional network to update nodes on the edges of the spatial relationship graph, extracting features of the structural relationship between nodes for spatial relationship recognition. Compared with the existing technology, the present invention has the ability to obtain spatial relationships in scenarios with scale and direction changes, while realizing knowledge transfer between object types, thereby learning spatial relationships of specific categories to more accurately and comprehensively recognize complex spatial relationships, so as to meet the changes in scale and direction of spatial relationships. It can be generalized to new scenarios and is not limited by application scenarios. It has good application prospects and commercial value.

Description

Technical Field

[0001] The present invention relates to the field of spatial relationship technology in computer vision, and in particular to a spatial relationship recognition method based on graph convolutional neural network reasoning. Background Art

[0002] The spatial relationship between objects in an image provides important information that helps to understand the image. Therefore, spatial relationship recognition has received increasing attention and has been applied to many computer vision tasks, such as image caption generation [1,2,10–12], image retrieval [13–16], target detection [17–20], action recognition [21–25], activity detection [26–30], and human interaction recognition [5,31–34]. The spatial relationship recognition task refers to the determination of the spatial relationship between different objects in the image by analyzing the relative position, direction, and distance between them in the field of computer vision. Specifically, the goal of this task is to identify the relationship between different objects, such as topological spatial relationships such as inclusion, intersection, and proximity, as well as information such as directional spatial relationships and distance spatial relationships between them. The spatial relationship contained in the image is represented by a relation triple. A relation triple refers to a data structure that represents the relationship between objects in the spatial relationship recognition task. It consists of three elements: subject, spatial predicate, and object, and can be expressed as<subject,spatialpredicate,object> The subject represents the starting object in the relationship, the spatial predicate represents the type of spatial relationship, and the object represents the end object in the relationship.

[0003] Spatial relationships play a crucial role in machine learning and computer vision. However, these relationships are often complex and can vary in scale and orientation, making it difficult for models to generalize to new scenarios. A common example is the relationship between a person and a car, which can change depending on the distance and angle from which the objects are viewed. Scale variation refers to the fact that the size of an object changes as its distance increases. For example, if we observe a car from a distance, it may appear as a small dot, while if we observe it up close, it may appear larger. This scale variation can make it difficult for models to generalize to new scenarios. This is because the model may have only been trained on cars of a few scales, while in real-world applications, the model needs to be able to recognize cars of various scales. Another common issue is orientation variation. In many cases, the orientation of objects can affect their relationships. For example, if we want to distinguish between a car and a truck, we may need to consider the vehicle's orientation, as cars and trucks can have very similar shapes. However, if the model is trained only on the vehicle's shape without considering its orientation, it may struggle to handle new scenarios.

[0004] In addition to scale and orientation issues, object category diversity is also a common challenge, as objects of different categories may have different spatial relationships. These spatial relationships can play an important role in the performance and generalization ability of the model. For example, the spatial relationship between a person and a chair may be different from the spatial relationship between a person and a car. Therefore, the model needs to be able to learn these category-specific spatial relationships.

[0005] The existing spatial relationship models are unable to learn the spatial relationships of specific categories. The spatial relationships cannot change in scale and direction. Differences in object categories lead to differences in relationships. That is, objects of different categories may have different spatial relationships, making it difficult for the model to generalize to new scenarios, and its application is greatly limited. Summary of the Invention

[0006] The purpose of the present invention is to provide a spatial relationship recognition method based on graph convolutional neural network reasoning to address the deficiencies of the existing technology. It adopts a spatial relationship graph based on spatial attention to construct a spatial relationship graph that integrates spatial and semantic features, and uses a graph convolutional neural network to update the nodes of the edges of the spatial relationship graph to obtain enhanced features, thereby realizing knowledge transfer between object types, learning the spatial relationships of specific categories, and extracting the features of the structural relationships between nodes for spatial relationship recognition. The method is simple and has good use effect. It can obtain spatial relationships in scenarios with scale and direction changes to more accurately and comprehensively identify complex spatial relationships. It can be generalized to new scenarios to meet the changes in scale and direction of spatial relationships. It is not limited by application scenarios and has good application scenarios and commercial value.

[0007] The specific technical solution for achieving the purpose of the present invention is: a spatial relationship recognition method based on graph convolutional neural network reasoning, which is characterized in that the method specifically includes the following steps:

[0008] S1. Construction of three spatial features

[0009] 1) The first spatial feature is (tl x ,tl y ,w,h) represents the width and height spatial features, where w and h are the width and height of the bounding box; tl x is the horizontal coordinate of the upper left corner of the bounding box; tl y The vertical coordinate of the upper left corner of the bounding box.

[0010] 2) The second spatial feature is an 8D vector that combines position, width and height information (tl x ,tl y ,br x ,br y ,x center ,y center ,w,h), where br x ,br y ,x center ,y center They are the horizontal coordinate of the lower right corner of the bounding box, the horizontal coordinate of the lower right corner of the bounding box, the horizontal coordinate of the center of the bounding box, and the vertical coordinate of the center of the bounding box.

[0011] 3) The third spatial feature considers the area of the bounding box as a 5-dimensional vector [53, 70, 72, 84], expressed as: Where W and H are the width and height of the image, A and A img are the areas of the object and image respectively.

[0012] S2. Construction of spatial relationship graph

[0013] 1) Create a node in the graph for each object detected in the image. The node can contain information about the object's location, size, and other properties.

[0014] 2) Add edges between nodes to represent the spatial relationship between objects, and use spatial attention features as edges. Using spatial attention features as edges has three benefits:

[0015] (i) It can reduce computational costs and make the model easier to handle;

[0016] (ii) the context of surrounding objects can be more effectively incorporated;

[0017] (iii) It can obtain the importance information of different objects and features in the image, which helps to identify the most significant spatial relationships.

[0018] 3) Construction of the spatial relationship graph. Once the nodes and edges are created, the graph can be constructed. Instead of using a fully connected graph to model the spatial relationship between objects, each subject is connected to all objects, and each object is connected to all subjects.

[0019] S3. Reasoning of Graph Neural Networks

[0020] Connect all subject features with each object feature, and connect all object features with each subject feature, learn the spatial relationship between the subject and the object, and thus enhance the subject features and object features. The process is as follows:

[0021] 1) Input the features of the subject and object in the image into the graph convolutional neural network.

[0022] 2) The input subject features and object features are linearly transformed through a fully connected layer.

[0023] 3) Using the spatial attention features of the encoded subject-object spatial relationship, the subject node features and object node features are updated.

[0024] 4) Perform an add operation on the updated subject node features and object node features.

[0025] 5) The graph features after the addition operation are input into two linear connection layers with Norm operation. These linear connection layers can recognize the spatial relationship between objects in the image after learning, and the output features are the enhanced subject features and object features.

[0026] Compared with the existing technology, the present invention has the ability to more accurately and comprehensively identify complex spatial relationships. The method is simple and has good use effect. It can obtain spatial relationships in scenarios with scale and direction changes to more accurately and comprehensively identify complex spatial relationships. It can be generalized to new scenarios to meet the changes in scale and direction of spatial relationships. It is not limited by application scenarios and has good application scenarios and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the spatial relationship recognition model of the present invention;

[0028] Figure 2 It is a spatial relationship diagram of the present invention;

[0029] Figure 3 Schematic diagram of graph convolutional neural network. DETAILED DESCRIPTION

[0030] The present invention is further described and explained in detail below with specific implementation:

[0031] Example 1

[0032] See Figure 1 , follow the steps below to identify spatial relationships:

[0033] S1. Construction of three spatial features

[0034] 1) The first spatial feature is (tl x ,tl y ,w,h) represents the width and height spatial features, where w and h are the width and height of the bounding box; tl x is the horizontal coordinate of the upper left corner of the bounding box; tl y The vertical coordinate of the upper left corner of the bounding box.

[0035] 2) The second spatial feature is an 8D vector that combines position, width and height information (tl x ,tl y ,br x ,br y ,x center ,y center ,w,h), where br x ,br y ,x center ,y center They are the horizontal coordinate of the lower right corner of the bounding box, the horizontal coordinate of the lower right corner of the bounding box, the horizontal coordinate of the center of the bounding box, and the vertical coordinate of the center of the bounding box.

[0036] 3) The area of the bounding box considered by the third spatial feature is a 5-dimensional vector [53,70,72,84], expressed as: Where W and H are the width and height of the image, A and A img are the areas of the object and image respectively.

[0037] S2. Construction of spatial relationship graph

[0038] 1) Create a node in the graph for each object detected in the image. The node can contain information about the object's location, size, and other properties.

[0039] 2) Add edges between nodes to represent the spatial relationship between objects, and use spatial attention features as edges. Using spatial attention features as edges has three benefits:

[0040] (i) It can reduce computational costs and make the model easier to handle;

[0041] (ii) the context of surrounding objects can be more effectively incorporated;

[0042] (iii) It can obtain the importance information of different objects and features in the image, which helps to identify the most significant spatial relationships.

[0043] 3) Construction of spatial relationship graph

[0044] See Figure 2 Once the nodes and edges are created, a spatial relationship graph can be constructed. Instead of using a fully connected graph to model the spatial relationship between objects, each subject is connected to each object, and each object is connected to each subject.

[0045] S3. Reasoning of Graph Neural Networks

[0046] Connect all subject features with each object feature, and connect all object features with each subject feature, learn the spatial relationship between subject and object, and thus enhance subject features and object features.

[0047] See Figure 3 , the reasoning of the graph neural network is as follows:

[0048] 1) Input the features of the subject and object in the image into the graph convolutional neural network.

[0049] 2) The input subject features and object features are linearly transformed through a fully connected layer.

[0050] 3) Use the spatial attention features that encode the spatial relationship of the subject-object pair to update the subject node features and object node features.

[0051] 4) Perform an add operation on the updated subject node features and object node features.

[0052] 5) The graph features after the addition operation are input into two linear connection layers with Norm operation. These linear connection layers can recognize the spatial relationship between objects in the image after learning, and the output features are the enhanced subject features and object features.

[0053] The above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A spatial relationship recognition method based on graph convolutional neural network reasoning, characterized in that: A spatial relationship graph that integrates spatial and semantic features is constructed based on spatial attention. A graph convolutional network is used to update nodes along the edges of the spatial relationship graph, thereby extracting features of the structural relationships between nodes for spatial relationship recognition. The specific steps include: 1) Construct three spatial features to represent the spatial relationships of objects in the bounding box-based image, and use a linear layer network to enhance the extracted spatial features; 2) Using the spatial features obtained by the two-dimensional graph representation method and the semantic features obtained by extracting semantic knowledge, we construct the edge features and node features of the spatial relationship graph. Then, we use the graph convolutional neural network for reasoning to obtain enhanced features. The step 1) constructs three spatial features to represent the spatial information of the bounding box, as follows: 1) The first spatial feature is Width and height spatial features in and is the width and height of the bounding box; is the horizontal coordinate of the upper left corner of the bounding box; is the vertical coordinate of the upper left corner of the bounding box; 2) The second spatial feature is an 8D vector that combines position, width, and height information ,in are the horizontal coordinate of the lower right corner of the bounding box, the horizontal coordinate of the lower right corner of the bounding box, the horizontal coordinate of the center of the bounding box, and the vertical coordinate of the center of the bounding box; 3) The third spatial feature is The area of the bounding box is a 5-dimensional vector where and is the width and height of the image; and are the areas of the object and image respectively; The step 2) of constructing the spatial relationship diagram specifically includes: 1) Node creation: Create a node in the graph for each object detected in the image. The node may contain information about the object's location, size, and other properties. 2) Edge creation: Add spatial attention features as edges between nodes to represent the spatial relationship between objects; 3) Construction of spatial relationship graph: Connect each subject with all objects, and connect each object with all subjects to form a spatial relationship graph.

2. The spatial relationship recognition method based on graph convolutional neural network reasoning according to claim 1 is characterized in that The reasoning of the graph convolutional neural network in step 2) is to connect the features of all subjects with the features of each object, and connect the features of all objects with the features of each subject, learn the spatial relationship between the subject and the object, and thus enhance the subject features and object features. The reasoning process of the graph convolutional neural network is as follows: 1) Input the features of the subject and object in the image into the graph convolutional neural network; 2) The input subject features and object features are linearly transformed through a fully connected layer; 3) Using the spatial attention features of the encoded subject-object spatial relationship, the subject node features and object node features are updated; 4) Perform an add operation on the updated subject node features and object node features; 5) The graph features after the addition operation are input into two linear connection layers with a Norm operation for learning, and the output is features that can identify the spatial relationship between objects in the image, namely the enhanced subject features and object features.

Images