Scene graph generation method based on space attention depth convolution and feature enhancement

By introducing spatial attention depth convolution, image map feature enhancement and cross attention fusion modules into the scene graph generation method, the problems of insufficient feature extraction and neglect of context information in the prior art are solved, and the quality of scene graph generation and The model's ability to understand image content.

CN118941809APending Publication Date: 2024-11-12HEBEI UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410961322.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing scene graph generation method cannot highlight important areas during feature extraction, ignore context information, and the model complexity and calculation cost are high, making it difficult to accurately capture scene details of complex interactions and diverse relationships.

Method used

The spatial attention depth convolution and feature enhancement method is adopted to enhance the feature expression of the convolution neural network through the spatial attention module. The image map feature enhancement module captures global context information and introduces the cross attention fusion module to interact with the characteristics and relationships of the object. feature.

Benefits of technology

It improves the model's adaptability and generalization ability to diversified data, significantly improves the generation quality of scene maps and the depth of understanding of semantic content, and enhances the overall understanding of image content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941809A_ABST
    Figure CN118941809A_ABST
Patent Text Reader

Abstract

The invention discloses a scene graph generation method based on space attention depth convolution and feature enhancement. The method comprises the following steps: firstly, extracting an object and visual features, bounding box coordinates and an initial prediction label of the object from an input image by using a target detection model; then, an object encoder obtains the context of the object according to the visual features of the object, the bounding box coordinates and the initial prediction label encoding; extracting global image features by using an image map feature enhancement module, and splicing the global image features with the object context to obtain an enhanced object context; the object decoder obtains a final prediction label of the object according to the enhanced object context; finally, the enhanced object context, the visual features of the object and the final prediction label are input into a relation encoder, and the edge context of the object pair is obtained; and coding and enhancing the edge context of the object pair, and performing predicate relation prediction according to the subject feature and the predicate feature of the object pair. According to the method, important areas or details in the image are particularly concerned, and meanwhile, the understanding of the model on the image content is enhanced through the global context information, so that the generation quality of the scene graph is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of visual image processing, and specifically is a scene graph generation method based on spatial attention deep convolution and feature enhancement. Background Art

[0002] Scene graph is an efficient structured representation that can capture comprehensive semantic knowledge. It is the key to deeply understand visual scenes. It can serve as a bridge between visual scenes and text descriptions, providing a promising alternative for tasks that combine vision and language. It is widely used in knowledge management, reasoning, retrieval, recommendation and other fields. Scene graph generation requires the classification and location of relationships between objects to improve the performance of object detection and visual relationship detection.

[0003] The scene graph generation method based on deep learning is a technology that automatically recognizes and understands objects and their relationships from a scene graph. It aims to construct a graph representation, where nodes represent objects in the scene graph and edges represent semantic relationships between objects. However, there are still some problems with the existing scene graph generation methods: First, in the traditional scene graph generation model, on the one hand, the ordinary convolution operation treats all regions of the input feature map equally, without paying special attention to or strengthening the important regions or details in the feature map, which means that in the feature extraction process, all spatial positions are given the same importance, and the regions that are more critical to the task cannot be highlighted. On the other hand, some models use highly complex network structures to capture and understand the details in the image. Although it can improve the model's ability to capture image details, it will also lead to a significant increase in model complexity and computational cost. Second, the object context obtained by the object context encoder of the existing scene graph generation model mainly contains the local feature information of each object. This local feature information is obtained by considering the object's own attributes and its relative relationship with other objects. Simply relying on local features may ignore important context information, resulting in the model being unable to fully understand the role and interaction of objects in the scene. Third, in the encoding stage of most scene graph generation models, there is a lack of effective mechanisms to integrate information about objects and relationships, which makes it difficult for the model to accurately capture details when dealing with scenes containing complex interactions and diverse relationships, resulting in limited performance of scene graph generation. To this end, the present invention proposes a scene graph generation method based on spatial attention deep convolution and feature enhancement. Summary of the invention

[0004] In view of the deficiencies in the prior art, the technical problem that the present invention intends to solve is to provide a scene graph generation method based on spatial attention deep convolution and feature enhancement.

[0005] The present invention solves the technical problem by adopting the following technical solution:

[0006] A scene graph generation method based on spatial attention deep convolution and feature enhancement, characterized in that it includes the following steps:

[0007] In the first step, the object detection model is used to extract objects, their visual features, bounding box coordinates, and initial predicted labels from the input image.

[0008] In the second step, the object encoder encodes the object context based on the object’s visual features, bounding box coordinates, and initial predicted label. The object encoder is expressed as:

[0009] O i =Enc obj ([V i ; B i ],Map(l i )) (9)

[0010] Where V i , B i and l i are the visual features, bounding box coordinates, and initial predicted labels of the i-th object, respectively. i Represents the i-th object context, Enc obj represents the object encoder, Map(·) represents mapping the predicted label to the embedding space, and [;] represents the concatenation operation;

[0011] The image atlas feature enhancement module is used to extract global image features. First, an image graph is constructed based on the visual features of all objects to obtain an adjacency matrix. The graph convolutional network is used to iteratively update node features. The node message passing during the iteration process is expressed as:

[0012]

[0013] In the formula, represents the demagnetization of node i at the tth iteration, A ij represents the connection weight between node i and node j in the adjacency matrix, represents the feature of node j at the t-1th iteration, N(i) represents the set of neighbor nodes of node i, and ε represents a constant;

[0014] The node message is further processed by the kernel function, expressed as:

[0015]

[0016] In the formula, It is a node message After further processing, W1 and b1 are the weight and bias of the first nonlinear layer in the kernel function, W2 and b2 are the weight and bias of the second nonlinear layer in the kernel function, and ReLU(·) represents the ReLU activation function;

[0017] The further processed node message is input into the gated recurrent unit to update the node features. The updated node features are gated pooling to obtain the global image features. The gated pooling operation is expressed as:

[0018]

[0019] G=W p P g +b p (18)

[0020] In the formula, is the updated feature of node i in the tth iteration, α i is the attention score of node i, σ represents the Sigmoid activation function, W α , b α are the weight matrix and bias term of the linear transformation from node features to attention scores, P g is the weighted sum of all node features in the image graph g, W p , b p are the weight matrix and bias term of the linear transformation from global representation to image features, respectively, G is the global image feature;

[0021] The global image feature G is concatenated with the object context to obtain an enhanced object context. The enhanced object context passes through the object decoder to obtain the final predicted label of the object.

[0022] l' i =argmax(Softmax(Dec obj ([ O i;G]))) (19)

[0023] In the formula, l' i represents the final predicted label of the i-th object, Dec obj is the object decoder, Softmax(·) represents the Softmax function, and argmax(·) represents the argmax function;

[0024] In the third step, the enhanced object context, the visual features of the object, and the final predicted label are input into the relation encoder to obtain the side context of the object pair; the relation encoder is expressed as:

[0025] E ij =Enc rel (V i ;Linear([O i ;G])],Map(l' i )) (20)

[0026] In the formula, E ij Represents an object pair (o i ,o j )’s edge context, Enc rel is a relational encoder; Linear(·) represents a linear transformation to achieve feature dimensionality reduction;

[0027] The object context and the side context of the object pair are input into the cross attention fusion module for fusion, and then multiplied element-wise with the side context of the object pair to obtain the enhanced side context of the object pair;

[0028] The enhanced side context of the object pair is segmented through the fully connected layer to obtain the subject feature and object feature of the object pair; the subject feature, object feature and joint feature of the object pair are then multiplied to obtain the relation feature of the object pair; the relation feature of the object pair is input into the relation decoder to output the predicate relation prediction label of the object pair.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. A spatial attention deep convolution module is proposed and integrated into the Faster R-CNN model to enhance the feature expression capability of convolutional neural networks, especially in the case of diverse object sizes and shapes. When the object size changes, the spatial attention module can adaptively adjust its focus to better capture the features of objects of different scales. This targeted feature learning helps the model show better adaptability and generalization when facing diverse data. At the same time, by using deep separable convolution, the model can process more feature information with fewer parameters.

[0031] 2. An image graph feature enhancement module is proposed. The image graph feature enhancement module uses the visual features of all objects in the image to form a graph structure, and performs message passing to update node features; all node features are then aggregated into global image features through gated pooling operations to capture the overall contextual information of the image. By concatenating global image features with object context, the object decoder can take into account both the object's own attributes and the object's role in the entire image when decoding each object, thereby enhancing the model's overall understanding of the scene, which can significantly improve the model's understanding of image content, especially in identifying the relationship between objects, thereby improving the quality of scene graph generation.

[0032] 3. Propose a cross-attention fusion module. The cross-attention fusion module allows the model to rely not only on the features of the object itself but also on the relational features related to the object when encoding each object, thereby realizing the interaction between object features and relational features, and dynamically adjusting the attention allocation according to the relationship between the object and the side context, so that the model can focus on the most critical information for scene understanding. Therefore, the cross-attention fusion module not only improves the generation quality of the scene graph, but also enhances the model's understanding depth of the semantic content of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the overall flow chart of the present invention;

[0034] Figure 2 is a structural diagram of the target detection model of the present invention;

[0035] Figure 3 It is a structural diagram of the depth-separable convolution of the present invention;

[0036] Figure 4 is a structural diagram of the spatial attention module of the present invention;

[0037] Figure 5 It is a structural diagram of the image atlas feature enhancement module of the present invention;

[0038] Figure 6 It is a structural diagram of the cross-attention fusion module of the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods, but the protection scope of the present application is not limited thereto.

[0040] The present invention is a scene graph generation method based on spatial attention deep convolution and feature enhancement (referred to as method, see Figures 1 to 6 ), including the following steps:

[0041] The first step is to improve the Faster-RCNN network and use the improved Faster-RCNN network as a target detection model for target detection to extract objects from the input image. i and the visual features V of the object i , Bounding box coordinates B i and the initial predicted label L i ;

[0042] The Faster-RCNN network uses ResNet-101 as the backbone network to extract image features. The ResNet-101 network is constructed using residual blocks. Each residual block contains multiple convolutional layers and a skip connection, which allows the network to learn deeper features while maintaining gradient flow. However, the ordinary convolution operation in the residual block treats all areas of the input feature map equally, and there is no mechanism to pay special attention to or strengthen the important areas or details in the feature map. This means that all spatial positions are given the same importance during the feature extraction process, and the areas that are more critical to the task cannot be highlighted. There are deficiencies in fine-grained attention, which makes the model lack a spatial attention mechanism and the generalization ability of feature extraction is limited, and there are limitations in feature expression. In order to solve the problem of lack of strengthening processing of specific areas in the feature map, the Spatial Attention Depthwise Convolution Module (SADCM) is used to replace the 3×3 convolution in each residual block of the ResNet-101 network to obtain the improved Faster-RCNN network. The overall structure is as follows: Figure 2 shown.

[0043] Compared with traditional convolution, the spatial attention deep convolution module includes depthwise separable convolution (DSC) and spatial attention module (SAM). Depthwise separable convolution is an efficient convolution operation method, which is mainly used to reduce the number of model parameters and calculations while maintaining or improving model performance. In ordinary convolution, the convolution filter spans all input channels, and each filter produces an output channel. The depthwise separable convolution divides the convolution process into depthwise convolution and pointwise convolution. Its structure is as follows: Figure 3 As shown. The depth convolution is to convolve each input channel with a separate convolution kernel. If there are K input channels, there will be K convolution kernels. The output of the depth convolution still has K channels, but each channel only contains information from the corresponding input channel. The output feature map F of the kth channel obtained by the depth convolution is k It is expressed as:

[0044] F k =ReLU(BN(C k *K k )); k∈{1,2,…,K} (1)

[0045] In the formula, C k is the kth input channel, * indicates the convolution operation; K kis the convolution kernel of the kth channel, with a size of 3×3; BN(·) represents the normalization operation, and ReLU(·) represents the ReLU activation function;

[0046] Point-by-point convolution uses a 1×1 convolution kernel to convolve the result of depth convolution. This process does not increase the information of the spatial dimension, but combines the features from different input channels. The number of convolution kernels of point-by-point convolution is equal to the number of output channels. The calculation formula of point-by-point convolution is:

[0047] X = ReLU(BN(F*W)) (2)

[0048] Where, F={F1,F2,…,F K} is the output feature map of the depthwise convolution, W is the weight matrix of the 1×1 convolution kernel, and X is the output feature map of the point-by-point convolution, that is, the output feature map of the depthwise separable convolution.

[0049] The main purpose of the spatial attention module is to improve the network's sensitivity to spatial information in the input feature map. The spatial attention module uses the attention weights of each position in the feature map for modeling, which can highlight important features and suppress unimportant information, thereby enhancing the network's attention to key spatial positions in the input feature map. Figure 4 As shown in Figure 1, the input feature map of the spatial attention module undergoes the intra-channel maximum pooling and intra-channel average pooling operations to obtain the intra-channel maximum value feature map and the intra-channel average value feature map. The intra-channel maximum pooling operation captures the most significant features at each spatial position by calculating the maximum value along the channel dimension, while the intra-channel average pooling operation provides an overall overview of the features at each spatial position by calculating the average value along the channel dimension. Therefore, the intra-channel maximum value feature map and the intra-channel average value feature map represent the most important and average spatial information in the input feature map, respectively. Then, the intra-channel maximum value feature map and the intra-channel average value feature map are concatenated along the channel dimension to form a fused feature map X that combines the maximum and average spatial information. cat Next, a two-dimensional convolutional layer is used to fusion feature map X cat The two-dimensional convolutional layer is configured with two input channels and one output channel to learn the weight of the spatial position; the output feature map of the two-dimensional convolutional layer is then converted into an attention map through the Sigmoid activation function. The value of each element of the attention map is between 0 and 1, indicating the importance of the corresponding spatial position. Finally, the attention map is multiplied element by element with the input feature map of the spatial attention module to obtain the output feature map of the spatial attention module, so that the network can highlight important spatial areas and suppress unimportant areas, thereby effectively improving the network's sensitivity and recognition ability to important spatial features. The expression of the spatial attention module is as follows:

[0050]

[0051] X cat =[max_out;avg_out] (5)

[0052] X conv =conv2d(X cat ) (6)

[0053] A atten =σ(X conv ) (7)

[0054]

[0055] In the formula, X c represents the cth channel information of the input feature map X of the spatial attention module, C represents the number of channels, max_out is the maximum value feature map in the channel, avg_out is the average value feature map in the channel, [;] represents the concatenation operation, X cat represents the fused feature map, conv2d(·) represents the two-dimensional convolution operation, X conv represents the output feature map of the two-dimensional convolutional layer, σ represents the Sigmoid activation function, A atten represents the attention map, represents the element-wise multiplication operation, and Y represents the output feature map of the spatial attention module.

[0056] In the second step, the object context is obtained by encoding the object's visual features, bounding box coordinates, and initial predicted labels. The object context is enhanced using global image features, and the enhanced object context is decoded to obtain the final label prediction of the object.

[0057] The visual features, bounding box coordinates, and initial predicted labels of each object are transferred to the object encoder to obtain the object context. The object encoder uses a three-layer Transformer encoder. The self-attention mechanism of the Transformer encoder enables the model to handle long-distance dependency problems. Through multiple layers of Transformer encoders, the model can learn deeper features. Each layer further captures and synthesizes information based on the previous layer, thereby obtaining richer context information. The object encoder is represented as:

[0058] O i =Enc obj ([V i ; B i ],Map(l i )) (9)

[0059] In the formula, O i Represents the i-th object context, Enc objrepresents the object encoder and Map(·) represents mapping the predicted labels to the embedding space.

[0060] The object context obtained by the object encoder mainly contains the local feature information of the object. This information is obtained by considering the object's own attributes and its relative relationship with other objects. Simply relying on local features may ignore important contextual information, resulting in the model being unable to fully understand the role and interaction of objects in the scene. To this end, the object encoder's understanding of object features in the image is enhanced through the image atlas feature enhancement module, thereby improving the accuracy and consistency of scene graph generation. The image atlas feature enhancement module fuses the feature representation of global context information based on the visual features of all objects to obtain the global image feature G.

[0061] like Figure 5 As shown in the figure, the image atlas feature enhancement module first constructs an image graph (im_graph) based on the visual features of all objects extracted by target detection. The nodes of the image graph correspond to the visual features of an object in the input image. At the same time, the adjacency matrix A of the image graph is constructed to store the connection relationship between the nodes. If there is an edge connecting nodes i and j, the element A of the adjacency matrix ij =1, otherwise A ij =0. The adjacency matrix A fully represents all objects in the input image and the connection relationship between objects. The graph convolutional network (GNN) is used to iteratively update node features to capture the dependencies between nodes. In the iterative process, message passing is implemented through the adjacency matrix to pass information from one node to all its neighboring nodes. The node features are updated by aggregating the information of neighboring nodes. The adjacency matrix provides key information for message passing. In the iterative process, node message passing is implemented by matrix multiplication. The adjacency matrix is ​​multiplied with the neighbor node features to aggregate the information from the neighbor nodes. The node message passing process is expressed as:

[0062]

[0063] In the formula, represents the message of node i at the tth iteration, A ij represents the connection weight between node i and node j in the adjacency matrix, represents the feature of node j at the t-1th iteration, N(i) represents the set of neighbor nodes of node i, and ε represents a constant used to prevent division by zero errors;

[0064] Node Message It is further processed by the kernel function; the kernel function consists of two nonlinear layers, each of which includes a fully connected layer and an activation function; the kernel function enhances the model's ability to handle complex patterns by introducing nonlinear transformations, and the stacking of kernel functions can increase the model depth and allow for deeper abstraction; the kernel function allows nonlinear transformations to occur each time a message is passed, allowing the graph convolutional network to learn more complex and useful graph structure features. The further processing of node messages is expressed as:

[0065]

[0066] In the formula, It is a node message After further processing, W1 and b1 are the weight and bias of the first nonlinear layer, and W2 and b2 are the weight and bias of the second nonlinear layer.

[0067] The further processed node message is input into the gated recurrent unit GRU, and the node features of the previous iteration and the neighbor node information aggregated in the current iteration are integrated through the GRU gating mechanism to update the node features. GRU contains a reset gate and an update gate to control the flow of information. The process of updating node features using GRU is expressed as:

[0068]

[0069] In the formula, is the output of the reset gate, W γ 、b γ are the weight matrix and bias term of the reset gate respectively, is the output of the update gate, W z 、b z are the weight matrix and bias term of the update gate respectively, is a candidate hidden state, W h 、b h are the weight matrix and bias term of the candidate hidden state, tanh(·) represents the hyperbolic tangent activation function, ⊙ represents the Hadamard product, They are the updated features of node i in the t-1th and tth iterations respectively.

[0070] After message passing, each node feature already contains its neighborhood information. Next, the updated node features are globally pooled through a gated pooling operation to generate the global features of the image graph. First, the attention score of each node is calculated through a gating mechanism. This score determines the importance of each node in the global pooling process. Then, the node features are weighted using these attention scores and aggregated on all nodes in the image graph to obtain the weighted sum of all node features in the image graph, i.e., the global representation of the image graph. Finally, the global representation of the image graph is passed through a linear transformation layer to obtain the global image feature G. The calculation formula of the gated pooling operation is as follows:

[0071]

[0072] G=W p P g +b p (18)

[0073] In the formula, α i is the attention score of node i, W α 、b α are the weight matrix and bias term of the linear transformation from node features to attention scores, P g is the weighted sum of all node features in the image graph g, W p 、b p are the weight matrix and bias term of the linear transformation from global representation to image features, respectively.

[0074] Finally, the global image feature G is concatenated with the object context to obtain the enhanced object context, and the global image feature is used to enhance the expressiveness of the object context. The enhanced object context passes through the object decoder to obtain the final predicted label of the object.

[0075] l' i =argmax(SOftmax(Dec obj ([O i ; G]))) (19)

[0076] In the formula, l' i Represents object o i The final predicted label, Dec obj is the object decoder, Softmax(·) represents the Softmax function, and argmax(·) represents the argmax function.

[0077] The third step is to encode and enhance the side context of the object pair, and predict the predicate relationship based on the subject and predicate features of the object pair;

[0078] The enhanced object context, the visual features of the object, and the final predicted label are input into the relation encoder to obtain the side context of the object pair; the relation encoder is expressed as:

[0079] E ij =Enc rel (V i ;Linear([O i ;G])],Map(l' i )) (20)

[0080] In the formula, E ij Represents an object pair (o i ,o j )’s edge context, Enc rel is a relational encoder; Linear(·) represents a linear transformation to achieve feature dimensionality reduction.

[0081] When dealing with complex visual scenes, the relationship between objects depends not only on their individual features, but also on the interactions between them. Cross-attention helps the model better understand the contextual dependencies in the scene by establishing direct connections between the feature representations of different objects, especially in the case of complex backgrounds and multiple interacting objects. Therefore, the cross-attention fusion module is introduced to capture the connections between objects. The cross-attention fusion module combines the object context i As a query, the edge context E of the object pair ij As keys and values, the criss-cross attention fusion module is represented as:

[0082] cross_attn = CrossAttention( i ,E ij ,E ij ) (twenty one)

[0083] Where cross_attn represents the output feature of the cross attention fusion module, CrossAttention(·) represents the cross attention fusion module;

[0084] Then, the output features of the cross-attention fusion module are combined with the side context E ij By performing element-by-element multiplication, we obtain the enhanced side context of the object pair, so that the model can establish the connection between object features and relationship features at the feature level, as shown in formula (22):

[0085]

[0086] In the formula, Represents an object pair (o i ,o j ) Enhanced edge context;

[0087] Since the object pair contains two objects, the enhanced side context of the object pair is divided into two parts through the fully connected layer to obtain the subject feature and object feature of the object pair. The process is shown in equations (23) and (24):

[0088]

[0089] In the formula, FC represents the fully connected layer, Head_text ij,s 、Tail_text ij,O They are object pairs (o i ,o j )’s subject and object features;

[0090] According to the object pair index, the subject feature, object feature and joint feature of the object pair are multiplied to obtain the relational feature of the object pair. The relational feature of the object pair is used as the input of the relation decoder to obtain the predicate relation prediction label of the object pair, as shown in equations (25) and (26):

[0091] R ij =Head_text ij,S Tail_text ij,O ·u ij (25)

[0092] p i→j =argmax(Softmax(Dec rel (R ij ))) (26)

[0093] In the formula, u ij Represents an object pair (o i ,o j ), that is, the joint feature of object o i With o j Characteristics of the overlapping part; R ij Represents the object pair R ij The relationship characteristics, p i→j Represents an object pair (o i ,o j )’s predicate relation prediction label, reflecting the object o i With o j The predicate relationship between .

[0094] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A scene graph generation method based on spatial attention deep convolution and feature enhancement, It is characterized in that The following steps are involved: In the first step, the object detection model is used to extract objects, their visual features, bounding box coordinates, and initial predicted labels from the input image. In the second step, the object encoder encodes the object context based on the object’s visual features, bounding box coordinates, and initial predicted label. The object encoder is expressed as: O i =Enc obj ([V i ;B i ],Map(l i )) (9) Where V i , B i and l i are the visual features, bounding box coordinates, and initial predicted labels of the i-th object, respectively. i Represents the i-th object context, Enc obj represents the object encoder, Map(·) represents mapping the predicted label to the embedding space, and [;] represents the concatenation operation; The image atlas feature enhancement module is used to extract global image features. First, an image graph is constructed based on the visual features of all objects to obtain an adjacency matrix. The graph convolutional network is used to iteratively update node features. The node message passing during the iteration process is expressed as: In the formula, represents the demagnetization of node i at the tth iteration, A ij represents the connection weight between node i and node j in the adjacency matrix, represents the feature of node j at the t-1th iteration, N(i) represents the set of neighbor nodes of node i, and ε represents a constant; The node message is further processed by the kernel function, expressed as: In the formula, It is a node message After further processing, the message W 1 、b 1 are the weight and bias of the first nonlinear layer in the kernel function, W 2 、b 2 are the weight and bias of the second nonlinear layer in the kernel function, respectively. ReLU(·) represents the ReLU activation function. The further processed node message is input into the gated recurrent unit to update the node features. The updated node features are gated pooling to obtain the global image features. The gated pooling operation is expressed as: G=W p P g +b p (18) In the formula, is the updated feature of node i in the tth iteration, α i is the attention score of node i, σ represents the Sigmoid activation function, W α , b α are the weight matrix and bias term of the linear transformation from node features to attention scores, P g is the weighted sum of all node features in the image graph g, W p , b p are the weight matrix and bias term of the linear transformation from global representation to image features, respectively, G is the global image feature; The global image feature G is concatenated with the object context to obtain an enhanced object context; the enhanced object context passes through the object decoder to obtain the final predicted label of the object; l′ i =argmax(Softmax(Dec obj ([O i ;G]))) (19) In the formula, l′ i represents the final predicted label of the i-th object, Dec obj is the object decoder, Softmax(·) represents the Sofimax function, and argmax(·) represents the argmax function; In the third step, the enhanced object context, the visual features of the object, and the final predicted label are input into the relation encoder to obtain the side context of the object pair; the relation encoder is expressed as: E ij =Enc rel ([V i ;Linear([O i ;G])],Map(l′ i )) (20) In the formula, E ij Represents an object pair (o i , o j )’s edge context, Enc rel is a relational encoder; Linear(·) represents a linear transformation to achieve feature dimensionality reduction; The object context and the side context of the object pair are input into the cross attention fusion module for fusion, and then multiplied element-wise with the side context of the object pair to obtain the enhanced side context of the object pair; The enhanced side context of the object pair is segmented through the fully connected layer to obtain the subject feature and object feature of the object pair; the subject feature, object feature and joint feature of the object pair are then multiplied to obtain the relation feature of the object pair; the relation feature of the object pair is input into the relation decoder to output the predicate relation prediction label of the object pair.

2. The scene graph generation method based on spatial attention deep convolution and feature enhancement according to claim 1, It is characterized in that In the first step, the Faster-RCNN network is improved by replacing the 3×3 convolutions in each residual block of the original ResNet-101 network with the spatial attention deep convolution module to obtain the target detection model; the spatial attention deep convolution module consists of two parts: the depthwise separable convolution and the spatial attention module. The depthwise separable convolution is expressed as: F k =ReLU(BN(C k *K k ));k∈{1,2,...,K}(1) X = ReLU(BN(F*W))(2) In the formula, C k is the kth input channel, * indicates the convolution operation, K k is the convolution kernel of the kth channel, N(·) represents the normalization operation, F is the output feature map of the depthwise convolution, W is the weight matrix, and X is the output feature map of the depthwise separable convolution; The spatial attention module is represented as: X c at=[max_out;avg-out](5) X conv =conv2d(X cat ) (6) A atten =σ(X conv ) (7) Where, X c represents the cth channel information of feature map X, C represents the number of channels, max_out is the maximum value feature map in the channel, avg_out is the average value feature map in the channel, X cat represents the fused feature map, conv2d(·) represents the two-dimensional convolution operation, X conv Represents the output feature map of the two-dimensional convolutional layer, A atten represents the attention map, represents the element-wise multiplication operation, and Y represents the output feature map of the spatial attention module.

3. The scene graph generation method based on spatial attention deep convolution and feature enhancement according to claim 1 or 2, It is characterized in that In the second step, the process of updating node features by the gated recurrent unit is expressed as: In the formula, is the output of the reset gate, W γ , b γ are the weight matrix and bias term of the reset gate respectively, is the output of the update gate, W z , b z are the weight matrix and bias term of the update gate respectively, is a candidate hidden state, W h , b h are the weight matrix and bias term of the candidate hidden state, tanh(·) represents the hyperbolic tangent activation function, ⊙ represents the Hadamard product, is the updated feature of node i in the t-1th iteration.

4. The scene graph generation method based on spatial attention deep convolution and feature enhancement according to claim 1, It is characterized in that The kernel function includes two nonlinear layers, each of which includes a fully connected layer and an activation function.

5. The scene graph generation method based on spatial attention deep convolution and feature enhancement according to claim 1 or 4, It is characterized in that In the criss-cross attention fusion module, the object context is taken as the query and the side context of the object pair is taken as the key and value.