A spatial relationship recognition method based on multi-feature fusion Transformer

Abstract

The present invention discloses a spatial relationship recognition method based on a multi-feature fusion Transformer, which is characterized by adopting a two-bit image representation method, modeling the overlapping or intersecting relationship between objects by constructing a binary image based on a bounding box, and extracting spatial features using CNN; the extracted spatial features are connected to construct a new feature input spatial attention model, a spatial semantic Transformer encoder layer model, and a visual semantic Transformer model, which respectively perform spatial feature optimization, spatial relationship recognition enhancement, and spatial relationship recognition. Compared with the existing technology, the present invention has the advantages of distinguishing important spatial relationships, integrating the global spatial semantics and visual semantic information between objects, and using multiple modal information of visual features, spatial features, and semantic features to more accurately and comprehensively recognize complex spatial relationships. The model is simple and easy to train, improves user experience, can be generalized to new scenarios, and has good application scenarios and commercial value.

Description

Technical Field

[0001] The present invention relates to the technical field of spatial relationship recognition in computer vision, and in particular to a spatial relationship recognition method based on multi-feature fusion Transformer. Background Art

[0002] The spatial relationship between objects in an image provides important information that helps to understand the image. Therefore, spatial relationship recognition has received increasing attention and has been applied to many computer vision tasks, such as image caption generation [1,2,10–12], image retrieval [13–16], target detection [17–20], action recognition [21–25], activity detection [26–30], and human interaction recognition [5,31–34]. The spatial relationship recognition task refers to the determination of the spatial relationship between different objects in the image by analyzing the relative position, direction, and distance between them in the field of computer vision. Specifically, the goal of this task is to identify the relationship between different objects, such as topological spatial relationships such as inclusion, intersection, and proximity, as well as information such as directional spatial relationships and distance spatial relationships between them. The spatial relationship contained in the image is represented by a relation triple. A relation triple refers to a data structure that represents the relationship between objects in the spatial relationship recognition task. It consists of three elements: subject, spatial predicate, and object, and can be expressed as<subject,spatialpredicate,object> The subject represents the starting object in the relationship, the spatial predicate represents the type of spatial relationship, and the object represents the end object in the relationship.

[0003] Spatial relationship recognition has attracted increasing attention. Complex spatial semantic relationships exist between objects, and these relationships rely not only on their geometric properties, such as position and orientation, but also on common sense knowledge. For example, if the sentence "fish on the water" is represented by the relation triple <fish, on..., water>, it does not conform to common sense. Conversely, "fish in the water" is represented by the relation triple <fish, in..., water>, which conforms to common sense. Therefore, the required model must not only be able to recognize and understand the relationships between objects in different scenarios, but also be able to account for the influence of common sense knowledge.

[0004] The objects and spatial relationships contained in images are very complex, and many methods fail to reduce this complexity by distinguishing the importance of relationships. When there are many objects in an image, a large number of spatial relationships may form between them. For N objects, there will be N(N-1) spatial relationships. However, from a human perspective, these spatial relationships have varying degrees of importance, and people typically focus on important or significant objects and relationships. The attention mechanism is similar to how humans distinguish information, retaining only the important information and ignoring the less important ones. Spatial semantics are complex, and the spatial semantics between objects depends not only on geometric properties such as position and orientation, but also on common sense knowledge, requiring the model to have a certain degree of robustness.

[0005] Existing spatial relationship models cannot distinguish the importance of spatial relationships and objects. In an image, all objects and relationships are considered equally important without distinguishing their importance. As a result, recognition is ineffective when objects and spatial relationships are very complex. Summary of the Invention

[0006] The purpose of the present invention is to provide a spatial relationship recognition method based on multi-feature fusion Transformer to address the deficiencies of the existing technology. The method adopts the attention mechanism and the Transformer model to solve the problem that image spatial relationship recognition fails to distinguish the importance of spatial relationships and objects. It uses multiple modal information of visual features, spatial features and semantic features to more accurately and comprehensively identify complex spatial relationships. At the same time, the attention mechanism and the Transformer model are used to better solve the problem that image spatial relationship recognition fails to distinguish the importance of spatial relationships and objects, greatly improving the performance of the model. The model is simple and easy to train, which greatly improves the user experience and can be generalized to new scenarios. It has good application prospects and commercial value.

[0007] The specific technical solution for achieving the purpose of the present invention is: a spatial relationship recognition method based on multi-feature fusion Transformer, which specifically includes the following steps:

[0008] S1. Extract spatial features using two-bit image representation method

[0009] 1) Extract the subject object proposal to model the spatial relationship between object pairs.

[0010] 2) Focus on the union of the two bounding boxes corresponding to the subject and object, that is, the attention window.

[0011] 3) Remove the part outside the focused union and expand the union of the two bounding boxes corresponding to the subject and object to a fixed size.

[0012] 4) Set the pixel values inside the bounding box to 1 and the pixel values outside the bounding box to 0 to generate two binary images corresponding to the two bounding boxes to capture the spatial relationship between the subject and the object.

[0013] 5) Use convolutional layer and pooling layer to extract the final spatial feature S os , the spatial feature S os The generation is expressed by the following formula (a):

[0014] S os =Flat(Pool(Conv(Pool(Conv(B os ))))) (a).

[0015] Among them, B os It is a dual-channel two-dimensional spatial relationship diagram between object o and object s; Flat is the operation of converting spatial features into two-dimensional features; Pool is the pooling operation; Conv is the convolution operation.

[0016] S2. Modeling of spatial attention model

[0017] 1) After constructing the spatial features and semantic features, we connect the spatial features and semantic features and construct the feature X by the following formula (b): ors :

[0018] X ors =concat(L s , S os , L o ) (b).

[0019] Among them, L s is the semantic feature of the subject, L o It is the semantic feature of the object.

[0020] 2) X ors Projected into the query, key, and value embedding space, it is expressed by the following formulas (c) to (e):

[0021] query=W q X ors (c);

[0022] key=W k X ors (d);

[0023] value=W v X ors (e).

[0024] Among them, W q 、Wk and W v X ors Projected to the corresponding projection vector in the query, key, and value embedding space.

[0025] 3) Calculate the attention using the scaled dot product and normalize it using the following formula (f):

[0026]

[0027] Among them, a is the calculated attention; key T Indicates the transposition of key; d k is the dimension of key; is the vector dot product; Softmax is the result presented in the form of probability; d k is the dimension of query and key; query is the embedding space.

[0028] 4) Enhance the spatial features by using the following formula (g):

[0029] X ors =X ors +matmul(value, a) (g).

[0030] Among them, matmul is a multiplication operation.

[0031] S3. Modeling of the Spatial Semantic Transformer Encoder Layer Model

[0032] 1) Concatenate subject semantic features, spatial features, and object semantic features into a new feature.

[0033] 2) The new features are input into the self-attention layer to model the relationships between all elements in the input sequence to capture global context information. The self-attention mechanism allows the model to dynamically calculate the relationship between each input element and other elements when processing it, and use these relationships to weighted average the representation of each element.

[0034] 3) Perform residual connections and layer normalization. Residual connections allow the model to skip the information bottleneck between layers and more easily learn the mapping between input and output. The layer normalization adopts LayerNorm normalization to solve the problems of gradient disappearance and gradient explosion, thereby improving the training speed and performance of the model.

[0035] 4) The results of residual connection and layer normalization are input into the feedforward network layer. The feedforward network layer consists of one or more fully connected layers, and each fully connected layer consists of two parts: linear transformation and activation function.

[0036] S4. Modeling of the Visual Semantic Transformer Model

[0037] 1) Concatenate subject semantic features, visual features, and object semantic features into a new feature.

[0038] 2) Input the new features into the Transformer encoder layer. Each layer contains a multi-head self-attention mechanism and a feedforward neural network. The specific steps are as follows:

[0039] i) The self-attention mechanism is used to calculate the attention weights between the elements of the input sequence, which are then used to calculate the weighted sum of the input vectors;

[0040] ii) The result is fed into the feed-forward network to obtain the final output of the encoder layer.

[0041] 3) The output of the Transformer encoder layer is fed into the Transformer decoder layer. Each layer consists of a multi-head self-attention mechanism, a multi-head attention mechanism on the encoder output, and a feed-forward network. The specific steps are as follows:

[0042] i) Use the multi-head self-attention mechanism to calculate the attention weights between the elements of the input sequence;

[0043] ii) Use a multi-head attention mechanism on the encoder output to compute a weighted sum of the encoder outputs and combine it with the output of the self-attention mechanism;

[0044] iii) The result is fed into the feed-forward network to obtain the final output of the encoder layer.

[0045] Compared with the existing technology, the present invention can distinguish important spatial relationships, integrate the global spatial semantics and visual semantic information between objects, and more accurately and comprehensively identify complex spatial relationships. The model is simple and easy to train, which can significantly improve the user experience and can be generalized to new scenarios. It has good application scenarios and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the spatial relationship recognition model constructed by the present invention;

[0047] Figure 2 This is a process diagram of generating spatial features using binary image representation in the present invention. DETAILED DESCRIPTION

[0048] See Figure 1The present invention utilizes semantic knowledge between objects to enhance spatially related information between objects. It also constructs a spatial Transformer module based on the spatial attention module to integrate the global spatial semantics and visual semantic information of the object, thereby distinguishing important spatial relationships and identifying complex spatial relationships more accurately and comprehensively. A two-bit image representation method is adopted to model the overlapping or intersecting relationships between objects by constructing a binary image based on a bounding box, and then CNN is used to extract spatial features. Semantic knowledge model and visual knowledge model are then used to extract semantic knowledge and visual knowledge. The semantic knowledge model uses the word vector model word2vec to construct language features about subject categories and object categories, and the visual knowledge model uses Faster RCNN to extract visual features in the image. The semantic features and visual features are then connected to construct new features, which are then input into the spatial attention model, the spatial semantic Transformer encoder layer model, and the visual semantic Transformer model. The spatial attention model uses spatial semantic knowledge and attention mechanism to optimize spatial features; the spatial semantic Transformer encoder layer model uses the input spatial semantic knowledge to obtain global context information, thereby enhancing the recognition of spatial relationships. Its core component is the self-attention mechanism defined by the spatial attention model; the visual semantic Transformer model inputs the input language features and visual features into the Transformer to identify spatial relationships, which can significantly improve the user experience.

[0049] The present invention is further described and explained in detail below with specific implementation:

[0050] Example 1

[0051] See Figure 2 , follow the steps below to identify spatial relationships:

[0052] S1. Extract spatial features using two-bit image representation method

[0053] 1) Extract the subject object proposal to model the spatial relationship between object pairs.

[0054] 2) Focus on the union of the two bounding boxes corresponding to the subject and object, that is, the attention window.

[0055] 3) Remove the part outside the focused union and expand the union of the two bounding boxes corresponding to the subject and object to a fixed size.

[0056] 4) Set the pixel values inside the bounding box to 1 and the pixel values outside the bounding box to 0 to generate two binary images corresponding to the two bounding boxes to capture the spatial relationship between the subject and the object.

[0057] 5) Use convolutional layer and pooling layer to extract the final spatial feature S os , the spatial feature S os The generation is expressed by the following formula (a):

[0058] S os =Flat(Pool(Conv(Pool(Conv(B os ))))) (a).

[0059] Among them, B os It is a dual-channel two-dimensional spatial relationship diagram between object o and object s; Flat is the operation of converting spatial features into two-dimensional features; Pool is the pooling operation; Conv is the convolution operation.

[0060] S2. Modeling of spatial attention model

[0061] 1) After constructing the spatial features and semantic features, construct feature X by connecting the spatial features and semantic features ors , the feature X ors Constructed by the following formula (b):

[0062] X ors =concat(L s , S os , L o ) (b).

[0063] Among them, L s is the semantic feature of subject; L o It is the semantic feature of the object.

[0064] 2) X ors Projected into the query, key, and value embedding space, it is expressed by the following formulas (c) to (e):

[0065] query=W q X ors (c);

[0066] key=W k X ors (d);

[0067] value=W v X ors (e).

[0068] Among them, Wq 、W k and W v X ors Projected to the corresponding projection vector in the query, key, and value embedding space.

[0069] 3) Calculate the attention using the scaled dot product and normalize it using the following formula (f):

[0070]

[0071] Among them, a is the calculated attention; key T Indicates the transposition of key; d k is the dimension of key; is the vector dot product; Softmax is the result presented in the form of probability; d k is the dimension of query and key; query is the embedding space.

[0072] 4) The spatial feature X is calculated by the following formula (g): ors Enhancements:

[0073] X ors =X ors +matmul(value, a) (g).

[0074] Among them, matmul is a multiplication operation.

[0075] S3. Modeling of the Spatial Semantic Transformer Encoder Layer Model

[0076] 1) Concatenate subject semantic features, spatial features, and object semantic features into a new feature.

[0077] 2) The new features are fed into the self-attention layer to model the relationships between all elements in the input sequence to capture global context. The self-attention mechanism allows the model to dynamically calculate the relationship between each input element and other elements when processing it, and use these relationships to weighted average the representation of each element.

[0078] 3) Perform residual connection and layer normalization. The residual connection allows the model to skip the information bottleneck between layers and more easily learn the mapping between input and output. The layer normalization adopts LayerNorm normalization.

[0079] Layer normalization helps solve the problems of gradient disappearance and gradient explosion, and improves the training speed and performance of the model.

[0080] 4) The results of residual connection and layer normalization are input into the feedforward network layer. The feedforward network layer consists of one or more fully connected layers, and each fully connected layer consists of two parts: linear transformation and activation function.

[0081] S4. Modeling of the Visual Semantic Transformer Model

[0082] 1) Concatenate subject semantic features, visual features, and object semantic features into a new feature.

[0083] 2) The new features are input into the encoder layer of the Transformer, each of which contains a multi-head self-attention mechanism and a feed-forward neural network.

[0084] i) The self-attention mechanism is used to calculate the attention weights between the elements of the input sequence, which are then used to calculate the weighted sum of the input vectors;

[0085] ii) The result is fed into the feed-forward network to obtain the final output of the encoder layer.

[0086] 3) The output of the Transformer encoder layer is input into the Transformer decoder layer. Each layer contains a multi-head self-attention mechanism, a multi-head attention mechanism on the encoder output, and a feed-forward network.

[0087] i) Use the multi-head self-attention mechanism to calculate the attention weights between the elements of the input sequence;

[0088] ii) Use a multi-head attention mechanism on the encoder output to compute a weighted sum of the encoder outputs and combine it with the output of the self-attention mechanism;

[0089] iii) The result is fed into the feed-forward network to obtain the final output of the encoder layer.

[0090] The above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A spatial relationship recognition method based on multi-feature fusion Transformer, characterized by: The spatial relationship recognition specifically includes the following steps: 1) Using two-dimensional image representation methods, semantic knowledge models, and visual knowledge models to extract spatial relationships, semantic features, and visual features between objects; 2) Semantic features and visual features are connected to construct new features, and the new features are input into the spatial attention model, the spatial semantic Transformer encoder layer model and the visual semantic Transformer model. The spatial attention model uses spatial semantic knowledge and attention mechanism to optimize spatial features; the spatial semantic Transformer encoder layer model uses the input spatial semantic knowledge to obtain global context information and enhance the recognition of spatial relationships; the visual semantic Transformer model inputs the input language features and visual features into the Transformer to recognize spatial relationships.

2. The spatial relationship recognition method based on multi-feature fusion Transformer according to claim 1 is characterized in that The specific process of the two-bit image representation method in step 1) is as follows: 1) Extract the subject object proposal to model the spatial relationship between object pairs; 2) Focus on the union of the two bounding boxes corresponding to the subject and object, that is, the attention window; 3) Remove the part outside the focused union and expand the union of the two bounding boxes corresponding to the subject and object to a fixed size; 4) Set the pixel values inside the bounding box to 1 and the pixel values outside the bounding box to 0, generating two binary images corresponding to the two bounding boxes to capture the spatial relationship between the subject and the object; 5) Use convolutional layer and pooling layer to extract the final spatial feature S os , the spatial feature S os The generation is expressed by the following formula (a): S os =Flat(Pool(Conv(Pool(Conv(B os ))))) (a); Among them, B os It is a dual-channel two-dimensional spatial relationship diagram between object o and object s; Flat is the operation of converting spatial features into two-dimensional features; Pool is the pooling operation; Conv is the convolution operation.

3. The spatial relationship recognition method based on multi-feature fusion Transformer according to claim 1 is characterized in that: The specific process of step 2) inputting the new features into the spatial attention model is as follows: 1) Connect the constructed spatial features and semantic features, and construct the feature X by the following formula (b): ors : X ors =concat(L s ,S os ,L o ) (b); Among them, L s is the semantic feature of subject; L o is the semantic feature of the object; 2) X ors Projected into the query, key, and value embedding space, it is expressed by the following formulas (c) to (e): query=W q X ors (c); key=W k X ors (d); value=W v X ors (e); Among them, W q 、W k and W v X ors Projected to the corresponding projection vector in the query, key, and value embedding space; 3) Calculate the attention using the scaled dot product and normalize it using the following formula (f): Among them, a is the calculated attention; key T Indicates the transposition of key; d k is the dimension of key; is the vector dot product; Softmax is the result presented in the form of probability; d k is the dimension of query and key; query is the embedding space; 4) Spatial feature X ors Enhanced by the following formula (g): X ors =X ors +matmul(value,a) (g); Among them, matmul is a multiplication operation.

4. The spatial relationship recognition method based on multi-feature fusion Transformer according to claim 1 is characterized in that: The specific process of step 2) inputting the new features into the spatial semantic Transformer encoder layer model is as follows: 1) Connect the subject semantic features, spatial features, and object semantic features into a new feature; 2) The new features are input into the self-attention layer to model the relationships between all elements in the input sequence and capture global context information. The self-attention mechanism allows the model to dynamically calculate the relationship between each input element and other elements when processing the input element, and use these relationships to weighted average the representation of each element; 3) Performing residual connections and layer normalization, wherein the residual connections allow the model to skip the information bottleneck between layers and learn the mapping between input and output; The layer normalization adopts LayerNorm normalization to improve the training speed and performance of the model; 4) The results of residual connection and layer normalization are input into a feed-forward network layer consisting of one or more fully connected layers, wherein the fully connected layer consists of a linear transformation and an activation function.

5. The spatial relationship recognition method based on multi-feature fusion Transformer according to claim 1 is characterized in that The specific process of step 2) inputting the new features into the visual semantic Transformer model is as follows: 1) Concatenate subject semantic features, visual features, and object semantic features into a new feature; 2) The new features are fed into the Transformer’s encoder layer. Each encoder layer consists of a multi-head self-attention mechanism and a feedforward neural network. The self-attention mechanism operates as follows: 2-1: Use the self-attention mechanism to calculate the attention weights between the elements of the input sequence, and then calculate the weighted sum of the input vectors; 2-2: Input the above calculation results into the feedforward network to obtain the final output of the encoder layer; 3) The output of the Transformer encoder layer is fed into the Transformer decoder layer. Each encoder layer consists of a multi-head self-attention mechanism, a multi-head attention mechanism on the encoder output, and a feed-forward network. The self-attention mechanism operates as follows: 3-1: Use the multi-head self-attention mechanism to calculate the attention weights between the elements of the input sequence; 3-2: Use the multi-head attention mechanism on the encoder output to calculate the weighted sum of the encoder output and combine the weighted sum with the output of the self-attention mechanism; 3-3: Input the above results into the feedforward network to obtain the final output of the encoder layer.

Images