Scene graph generation method based on depth visual features

Abstract

The invention discloses a scene graph generation method based on depth visual features, and the method comprises the steps: inputting a training image in a data set into a target detector, and generating subject and object instance labels of a to-be-predicted relation; extracting subject and object instance features by adopting a semantic extractor, and performing feature fusion on the subject and object instance features to obtain relation features; performing feature extraction and coding on the depth map corresponding to each training image to form depth visual feature representation; fusing the relation feature and the depth vision feature representation to generate a depth vision relation feature representation; and carrying out cross entropy loss calculation on label distribution represented by the generated depth vision relation characteristics and real relation labels, then carrying out model training, and outputting a trained model until all training pictures are read. According to the method, the depth visual features are fused, the influence of background interference information is reduced, and the accuracy of relation prediction of the scene graph generation model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and in particular relates to a scene graph generation method based on deep visual features. Background Art

[0002] As an important topic in the field of artificial intelligence, scene graph generation deeply integrates the techniques of image processing and natural language processing, and is one of the core tasks in computer vision. The goal of scene graph generation is to identify and classify different object instances from a given image and accurately predict the visual relationship between these instances. Each pair of instance relationships is usually described by a noun and a predicate and presented in the form of a triple, such as "man-holding-plate". Among them, "man" is the subject instance, "plate" is the object instance, and "holding" is the relationship between them. By parsing all such triples in the image, a complete scene graph can be constructed, providing a structured semantic understanding of the image. Scene graph generation can not only improve the efficiency of image information acquisition, but also provide support for a variety of advanced computer vision applications. This includes but is not limited to image caption generation, visual question answering, visual navigation, and content retrieval of images and videos. For example, in the visual question answering task, the generated scene graph can help the model better understand the context of the question and provide more accurate answers. In addition, the generation of scene graphs also helps to improve the efficiency and accuracy of content-based image retrieval. Through structured image representation, the search system can better understand and match the user's query intent.

[0003] Existing scene graph generation methods first perform instance recognition and feature extraction on the input image through a pre-trained target detector model, and then use the scene graph generation model to combine the features of the subject instance and the object instance to predict the relationship between them. Although most current scene graph generation models have made significant progress in reducing the head-tail relationship prediction bias by balancing the predicate distribution, using re-weighting technology, and enhancing semantic information, they have not yet achieved ideal results in the accuracy of relationship prediction. The reason is that there is a lot of background interference in the image, and the model misclassifies this interference information, resulting in the inability of the scene graph generation method to accurately predict the relationship. This kind of misclassification caused by background interference may appear in many forms. For example, a portrait reflected in a mirror may be mistakenly identified by the model as an actual object, such as Figure 1 As shown in (a), there is actually only one person in the image, but the model mistakenly identifies the person in the mirror as another person and infers the corresponding wrong relationship. For example, an image on a TV or computer screen may be mistakenly considered to be part of a real scene, such as Figure 1(b) shows that there is no real person in the image, but the model recognizes the person on the TV and tries to predict the relationship between the person and the table. In addition, people on billboards or photos may also be mistakenly identified by the model as real people, such as Figure 1 (c) shows that there is a clear portrait in the photo on the wall, which is mistakenly identified by the model as a real person. These background interference phenomena will seriously affect the accuracy of the model's relationship prediction and introduce a large number of noisy relationships in the generated scene graph, misleading downstream tasks to focus on the image content.

[0004] As a task of predicting the relationship between instances in an image, scene graph generation relies heavily on the model accurately identifying instance information in the image. Existing scene graph generation methods have difficulty identifying background interference information, which causes the model to make incorrect relationship predictions on the interference information, resulting in a large number of noise relationships in the generated scene graph. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a scene graph generation method based on deep visual features to further improve the accuracy of the scene graph generation model in relationship prediction.

[0006] Technical solution: The training flow chart of FADV-SGG proposed in the present invention is as follows: Figure 5 As shown. In the training process, the batch size can be adjusted according to the performance of the computer. The training process of the present invention is described as follows:

[0007] Step 1: Input the training images in the dataset into the object detector to generate the subject and object instance labels of the relationship to be predicted, and proceed to step 2;

[0008] Step 2: Use a semantic extractor to extract instance features of the subject and object, further model them to generate refined subject and object instance features, fuse the subject and object instance features to obtain relationship features, and proceed to step 3;

[0009] Step 3: Use FullyAdaptive Transformer to extract and encode the depth map corresponding to each training image to form a deep visual feature representation, and then proceed to step 4;

[0010] Step 4: Fuse the relationship features obtained in step 2 with the deep visual features obtained in step 3 to generate a deep visual relationship feature representation, and proceed to step 5;

[0011] Step 5: Generate the label distribution of the deep visual relationship feature representation in step 4, calculate the cross entropy loss with the true relationship label, and then perform model training and proceed to step 6;

[0012] Step 6: If the set batch size is reached, return to step 2; if all training images have been read, go to step 7.

[0013] Step 7: Output the trained model and end.

[0014] Preferably, the training images in step 1 are all from the standard dataset VisualGenome150 (VG 150) for scene graph generation. The dataset consists of 108,077 images, including the 150 most common instance classes and 50 predicate classes. The present invention divides 70% of the images in the dataset into a training set and the remaining 30% of the images into a test set. The object detector uses a pre-trained Faster R-CNN model to obtain subject and object instance labels through the object detector.

[0015] Preferably, the steps of generating the relationship features in step 2 of the present invention are as follows:

[0016] Step 2-1: Use the semantic extractor to extract the main instance features f from the instance labels obtained in step 1 s and object instance features f o , as shown in Formula 1. Where w s , w o are the learnable weight parameters of the subject and object instances, T represents the matrix transpose, c s , c o They respectively represent the word embeddings of subject and object labels generated by the Glove model.

[0017]

[0018] Step 2-2: Refine the instance features to obtain instance feature representation, as shown in Formula 2. Where, and are the instance feature representations of the subject and object instances, FC(·), LN(·), and ReLU(·) represent the fully connected layer, layer normalization operation, and ReLU activation function, respectively. Finally, the residual connection is used to improve the expressiveness of the instance features.

[0019]

[0020] Step 2-3: According to formula 3, the feature representation of the subject and object instances is fused to obtain the relationship feature F so .

[0021]

[0022] Preferably, the steps of encoding the depth map in step 3 to obtain the depth visual features of the present invention are as follows:

[0023] Step 3-1: Input the depth map (VG-Depth) corresponding to the training image in step 1, and use the initial convolutional layer to extract the initial features v of the depth map i , obtain v through the conditional position encoding network CPE i The position information of d , as shown in Formula 4.

[0024] V d =CPE(v i )+v i (4)

[0025] Step 3-2: Use the fine-grained downsampling strategy shown in Formula 5 to perform the visual feature V d Modeling to generate fine-grained visual features This method can minimize the loss of global information. BN(·) and DW(·) are batch normalization and deep convolution operations respectively. 1*1 (·) is a 1×1 convolution.

[0026]

[0027] Step 3-3: Use the multi-head self-attention mechanism MSA for fine-grained visual features Perform global adaptive aggregation, as shown in Formula 6. Where V g ' lobal is the result of global aggregation, Express Perform a linear projection, is the matrix multiplication operation, w d is a learnable weight parameter;

[0028] V g ' lobal =MSA(Q d ,K d ,V d ) (6)

[0029] Step 3-4: Generate context-aware weights through the Sigmoid activation function and combine them with the deep convolution DW(·) to adaptively aggregate local information, as shown in Formula 7. Where ⊙ represents the element-wise product, V lo ' cal It is the result of local aggregation.

[0030] V lo ' cal =DW(Q d )⊙Sigmoid(DW(Q d )) (7)

[0031] Step 3-5: Fusion of V through bidirectional adaptive interaction mechanisms from “local to global” and “global to local” g ' lobal and V lo ' cal , so that local features and global features can contain each other's information. The local to global interaction process is shown in Formula 8, and the global to local interaction process is shown in Formula 9. Among them, V local is the local feature after fusion, V global is the global feature after fusion.

[0032] V local =V lo ' cal ⊙Sigmoid(V g ' lobal ) (8)

[0033] V global =V g ' lobal ⊙Sigmoid(V lo ' cal ) (9)

[0034] Step 3-6: To combine local features with global features, use matrix multiplication The element-wise product ⊙ and linear projection achieve the mixing between global and local features, as shown in Formula 10. Where w gl is the learnable weight parameter, V g_l The result after mixing.

[0035]

[0036] Step 3-7: Use the convolutional feed-forward network ConvFFN and the downsampling operation DS(·) to transform V g_l Further modeling to obtain the final deep visual feature representation Among them, LN(·) is the layer normalization operation.

[0037]

[0038] Preferably, the specific steps of generating the deep visual relationship feature representation in step 4 of the present invention are as follows:

[0039] Step 4-1: The relationship feature F obtained in step 2 so and deep visual feature representation Splicing along the y-axis and further refinement through the network layer to obtain deep visual relationship feature representation As shown in Formula 12, [·,·] is the concatenation operation, FC(·), LN(·), and ReLU(·) represent the fully connected layer, layer normalization operation, and ReLU activation function, respectively.

[0040]

[0041] Preferably, the steps of calculating the cross entropy loss in step 5 of the present invention are as follows:

[0042] Step 5-1: Use semantic extractor to extract predicate features f p , as shown in formula 10. Where w p is the learnable weight parameter of the predicate, T represents the matrix transpose, c p is the word embedding representing the predicate label.

[0043] f p =w p T c p (13)

[0044] Step 5-2: Representing deep visual relationship features and the predicate feature f p After normalization, the label distribution F of the deep visual relationship p It is calculated by formula 14. Among them, ||·|| represents the L2 norm, ⊙ is the element-by-element product operation, exp is the exponential operation, and γ is a learnable scaling factor used to adjust the range of the distribution of deep visual relationship labels.

[0045]

[0046] Step 5-3: Calculate the label distribution F of the deep visual relationship through the cross entropy loss function as shown in Formula 15 p and the true relation label F t The loss between them is used as the relationship loss L for model training rel Among them, F tj and F pj are the true relation label and deep visual relation label distribution of the jth relation sample, respectively, and N is the total number of relation samples on each image.

[0047]

[0048] Preferably, the present invention is tested separately under three subtasks, namely, predicate classification (PredicateClassification, PredCls), scene graph classification (Scene Graph Classification, SGCls) and scene graph detection (Scene Graph Detection, SGDet).

[0049] In the predicate classification task, the model takes the bounding box information and labels of all instances in the image as input, and the goal is to predict the relationship category between instances, focusing on understanding the interactions between entities without involving the challenge of instance recognition.

[0050] The scene graph classification task is similar to the predicate classification task in that its input also includes the bounding box information of all instances, but does not contain the instance category information. Therefore, the model not only needs to identify the category of each instance, but also predict the relationship between instances. This task is more challenging because the accuracy of instance classification directly affects the effect of relationship prediction.

[0051] Scene graph detection is the most challenging task. In this task, the input image does not contain bounding boxes and category information of any instances. The model needs to independently identify the locations and categories of all instances and predict the relationships between them. Instance recognition from scratch may lead to deviations from manually annotated data, increase the difficulty of relationship prediction, and thus affect the overall accuracy.

[0052] In order to test the output of the trained model in the scene graph generation method based on deep visual features, the test process of the present invention includes the following steps:

[0053] Step 1: Input the test image and the corresponding depth map into the above-mentioned trained model for relationship prediction, and then proceed to step 2;

[0054] Step 2: Compare the relationship prediction results with the true predicate labels, evaluate the results using currently commonly used evaluation indicators, and proceed to step 3;

[0055] Step 3: Output the relationship prediction results and evaluation results, and end.

[0056] Preferably, the test image in step 1 of the test process is selected from the test set divided in the VG150 data set, and the specific steps are as follows:

[0057] Step 1-1: Perform the predicate classification task. The model uses the position and category label information of all instances provided by the test set. Given that the position and category of each instance are known, the model focuses on predicting the relationship between these instances.

[0058] Step 1-2: Perform scene graph classification tasks. The model cannot use the category label information of the instances in the image, but needs to independently predict the category of each instance. These prediction results are then used for relationship prediction, that is, to continue predicting the relationship between instances based on the results of instance classification.

[0059] Steps 1-3: Perform scene graph detection tasks. The model cannot access the location and category label information of instances in the image. It first needs to identify and locate each instance in the image, and then predict the category of these instances. After completing these steps, the model further predicts the relationship between instances. This process is more complex and challenging than the previous two tasks because it needs to understand the image content from scratch.

[0060] Preferably, in step 2 of the test process, in order to compare the relationship prediction results with the true predicate labels, the present invention uses the widely recognized evaluation standard Recall@K (R@K) in the current scene graph generation field to accurately evaluate the test effect of the model. The higher the value of this indicator, the better the test result, indicating that the model performs better when dealing with diverse background interference information. The Recall@K indicator measures the frequency with which the model correctly identifies the predicate category in its top K predicted relationships. Specifically, the present invention uses the R@50 and R@100 indicators for evaluation.

[0061] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0062] (1) The present invention proposes a scene graph generation method based on deep visual features. The method adopts a latest lightweight ViT model FullyAdaptive Transformer (FAT) to encode the visual features of the depth map (VG-Depth) generated by the monocular depth estimator. By allowing the scene graph generation model to learn the visual features of the depth map, background interference information can be identified, thereby improving the accuracy of relationship prediction. The FAT model models local and global visual features and their bidirectional interactions in a context-aware manner through the fully adaptive self-attention (FASA) mechanism, and then uses a fine-grained downsampling strategy to enhance the downsampling self-attention mechanism to obtain a more fine-grained global perception capability. The present invention provides important visual information for scene graph generation through the deep visual features generated by the model, which helps the model to more accurately understand the image content and identify background interference information. These deep visual features will be integrated with the relationship features to form a more refined relationship feature representation to improve the accuracy of the model's relationship prediction. Specifically, the present invention proposes a fully adaptive depth visual scene graph generation network FADV-SGG (Fully Adaptive Depth Visual Scene Graph Generation Network), the overall framework of which is shown in the following figure: Figure 3 As shown in Figure 2, the fully adaptive attention module structure is as follows: Figure 4 As shown. Figure 2 Compared with the prior art that represents relationship features through DG-BGN refinement, the present invention integrates deep visual features, reduces the influence of background interference information, and improves the accuracy of relationship prediction of the scene graph generation model.

[0063] (2) The present invention conducts experiments on the VG150 dataset, which contains the 150 most common instance classes and 50 predicate classes. The present invention conducts indicator evaluation in three subtasks: predicate classification, scene graph classification, and scene graph detection. In the predicate classification task, the method of the present invention finally obtains the following evaluation results: R@50 is 67.9%, and R@100 is 69.6%; in the scene graph classification task, the method of the present invention finally obtains the following evaluation results: R@50 is 40.7%, and R@100 is 41.3%; in the scene graph detection task, the method of the present invention finally obtains the following evaluation results: R@50 is 31.9%, and R@100 is 35.9%. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of incorrect relationship prediction caused by background interference information.

[0065] Figure 2This is the overall network framework diagram of the prior art using DG-BGN to refine relationship features.

[0066] Figure 3 It is the overall network framework diagram of the scene graph generation method based on deep visual features of the present invention, and the gray area is the innovation point of the present invention.

[0067] Figure 4 is the fully adaptive attention module diagram in FADV-SGG of the present invention, and ⊙ is the element-by-element product operation.

[0068] Figure 5 Flowchart of the training phase of the fully adaptive deep visual scene graph generation network FADV-SGG of the present invention. DETAILED DESCRIPTION

[0069] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0070] Embodiment 1:

[0071] This embodiment uses the VG150 dataset to complete the scene graph generation task based on deep visual features. The VG150 dataset contains 108,077 images, covering the 150 most common instance categories and 50 predicate categories. In order to reasonably divide the dataset, we assign 70% of the images to the training set and the remaining 30% to the test. Both training and testing are performed on the deepin20.4 operating system, and the experimental environment is torch1.7.1+cu110. The FADV-SGG model of the present invention is trained for 50K iterations using the SGD optimizer, with the initial learning rate set to 0.001 and the batch size of 4. All experiments are performed on hardware equipped with NVIDIAGeForce RTX 4090GPU to ensure the efficiency of the training and testing process.

[0072] The present invention uses Faster R-CNN to detect instances in images. As a key milestone in the field of target detection, the architecture technology of Faster R-CNN has reached a mature stage, providing valuable inspiration and a solid foundation for many subsequently developed target detection models. Thanks to its excellent performance and flexibility, Faster R-CNN has been widely used in multiple computer vision tasks, including but not limited to object detection, image segmentation, and face detection. In addition, it is also widely used to solve various practical problems, such as visual processing of autonomous driving systems, dynamic target tracking in security monitoring, and medical image analysis and other key areas, demonstrating its wide influence in modern technology applications.

[0073] The present invention is trained and tested on the scene graph generation baseline model Motifs, and compared with the prior art DG-BGN. The Recall@K (R@K), an evaluation standard widely recognized in the current scene graph generation field, is used to accurately evaluate the performance of the model. The higher the value of this indicator, the better the test result, indicating that the model performs better when dealing with diverse background interference information. The Recall@K indicator measures the frequency with which the model correctly identifies the predicate category in its top K predicted relationships. Specifically, the present invention uses the R@50 and R@100 indicators for evaluation. The comparison results of the R@K indicator between the method of the present invention and the existing method on the VG150 dataset are shown in Table 1.

[0074] Table 1 Comparison results of the R@K index between the proposed method and the existing method on the VG150 dataset

[0075]

[0076] According to the results in Table 1, our method can achieve superior performance in relation prediction on the three subtasks. Compared with DG-BGN in the PredCls task, FADV-SGG improves R@50 and R@100 by 0.3% and 0.5% respectively; compared with DG-BGN in the SGCls task, FADV-SGG improves R@50 and R@100 by 0.6% and 0.3% respectively. In the SGDet task, FADV-SGG improves R@50 and R@100 by 0.7% and 0.4% respectively. These results show that the method FADV-SGG of the present invention can achieve a high level in the most basic evaluation criterion Recall@K, which improves the accuracy of relation prediction.

[0077] Embodiment 2:

[0078] This embodiment will introduce an applicable scenario of the present invention.

[0079] In an intelligent security system, surveillance cameras continuously capture real-time images and transmit them to the central processing unit for analysis. However, in practical applications, surveillance images often contain a lot of background interference information, such as reflections, screen display content, and images on billboards. These interference information will cause the system to make incorrect classifications and relationship predictions during analysis, thus affecting the accuracy and efficiency of the security system.

[0080] The scene graph generation method based on deep visual features proposed in this invention can effectively solve this problem. First, the system obtains the image captured by the surveillance camera and performs preprocessing, including image resizing, color correction, etc., and uses the Faster R-CNN model to extract features from the image.

[0081] Next, the method of the present invention is used to generate a detailed and accurate scene graph for each image, which can significantly improve the accuracy of relationship prediction, especially when processing images with a lot of background interference information. For example, when there are portraits reflected in mirrors, images on TV or computer screens, and portraits on billboards or photos in the image, the model can accurately distinguish these interference information and avoid misclassification, thereby generating true and reliable relationship prediction results.

[0082] The generated scene graph is then used in the security analysis of the system. Even in the face of complex backgrounds and interference information, the model can still provide accurate relationship predictions to ensure the system's timely response to potential threats and abnormal situations.

[0083] Finally, the system stores the generated scene graph and analysis results in a database for subsequent query and retrieval. Through the method of the present invention, the intelligent security system can significantly improve the accuracy of relationship prediction and the overall performance of the system when processing images with complex backgrounds and more interference information, thereby improving the reliability and effectiveness of the security system.

[0084] In summary, the present invention can effectively improve the accuracy of relationship prediction, reduce the occurrence of misclassification, and ensure the efficient operation of the system in scenarios with a lot of background interference information.

Claims

1. A scene graph generation method based on deep visual features, characterized in that: The following steps are involved: Step 1: Input the training images in the dataset into the object detector to generate the subject and object instance labels of the relationship to be predicted, and proceed to step 2; Step 2: Based on the subject and object instance labels of the relationship to be predicted, a semantic extractor is used to extract subject and object instance features, which are further modeled to generate refined subject and object instance features, and the subject and object instance features are fused to obtain relationship features, and then proceed to step 3; Step 3: Use FullyAdaptive Transformer to extract and encode the depth map corresponding to each training image to form a deep visual feature representation, and then proceed to step 4; Step 4: Fuse the relationship feature obtained in step 2 with the deep visual feature representation obtained in step 3 to generate a deep visual relationship feature representation, and proceed to step 5; Step 5: Generate the label distribution of the deep visual relationship feature representation in step 4, calculate the cross entropy loss with the true relationship label, and then perform model training and proceed to step 6; Step 6: If the set batch processing size is reached, return to step 2; if all training images have been read, go to step 7; Step 7: Output the trained model and end.

2. The method for generating a scene graph based on deep visual features according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 2-1: Use the semantic extractor to extract the main instance features f from the instance labels obtained in step 1 s and object instance feature f o , as shown in Formula 1, where w s , w o are the learnable weight parameters of the subject and object instances, T represents the matrix transpose, c s , c o Respectively represent the word embeddings of subject and object labels generated by the Glove model; Step 2-2: Refine the instance features to obtain instance feature representation, as shown in Formula 2, where: and are the instance feature representations of the subject and object instances, respectively. FC(·), LN(·), and ReLU(·) represent the fully connected layer, layer normalization operation, and ReLU activation function, respectively. Finally, the residual connection is used to improve the expression ability of the instance feature. Step 2-3: According to formula 3, the feature representation of the subject and object instances is fused to obtain the relationship feature F so ; 。 3. The method for generating a scene graph based on deep visual features according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3-1: Input the depth map VG-Depth corresponding to the training image in step 1, and use the initial convolutional layer to extract the initial features v of the depth map i , obtain v through the conditional position encoding network CPE i The position information of d , as shown in formula 4; V d =CPE(v i )+v i (4) Step 3-2: Use the fine-grained downsampling strategy shown in Formula 5 to downsample the visual feature V d Modeling to generate fine-grained visual features Minimize the loss of global information; BN(·) and DW(·) are batch normalization and deep convolution operations, respectively. 1*1 (·) is a 1×1 convolution; Step 3-3: Use the multi-head self-attention mechanism MSA for fine-grained visual features Perform global adaptive aggregation, as shown in Formula 6; where V g ' lobal is the result of global aggregation, Express Perform a linear projection, is the matrix multiplication operation, w d is a learnable weight parameter; V g ′ lobal =MSA(Q d ,K d ,V d ) (6) Step 3-4: Generate context-aware weights through the Sigmoid activation function and combine them with the deep convolution DW(·) to adaptively aggregate local information, as shown in Formula 7; where ⊙ represents the element-wise product, V lo ' cal It is the result of local aggregation; V′ local =DW(Q d )⊙Sigmoid(DW(Q d )) (7) Step 3-5: Fusion of V′ through local-to-global and global-to-local bidirectional adaptive interaction mechanisms global and V′ local , so that local features and global features contain each other's information; the local to global interaction process is shown in Formula 8, and the global to local interaction process is shown in Formula 9; where V local is the local feature after fusion, V global is the global feature after fusion; V local =V′ local ⊙Sigmoid(V′ global ) (8) V global =V g ′ lobal ⊙Sigmoid(V lo ′ cal ) (9) Step 3-6: To combine local features with global features, use matrix multiplication The element-wise product ⊙ and linear projection achieve the mixing between global and local features, as shown in Formula 10; where w gl is the learnable weight parameter, V g_l The result after mixing; Step 3-7: Use the convolutional feed-forward network ConvFFN and the downsampling operation DS(·) to transform V g_l Further modeling to obtain the final deep visual feature representation Among them, LN(·) is the layer normalization operation; 4. The method for generating a scene graph based on deep visual features according to claim 1, characterized in that: Step 4 is as follows: The relationship feature F obtained in step 2 so and deep visual feature representation Splicing along the y-axis and further refinement through the network layer to obtain deep visual relationship feature representation As shown in Formula 12, [·,·] is the concatenation operation, FC(·), LN(·), and ReLU(·) represent the fully connected layer, layer normalization operation, and ReLU activation function, respectively.

5. The method for generating a scene graph based on deep visual features according to claim 1, characterized in that: Step 5 specifically includes the following steps: Step 5-1: Use semantic extractor to extract predicate features f p , as shown in formula 10; where w p is the learnable weight parameter of the predicate, T represents the matrix transpose, c p is the word embedding representing the predicate label; f p =w p T c p (13) Step 5-2: Representing deep visual relationship features and the predicate feature f p After normalization, the label distribution F of the deep visual relationship p It is calculated by formula 14, where ||·|| represents the L2 norm, ⊙ is the element-by-element product operation, exp is the exponential operation, and γ is a learnable scaling factor used to adjust the range of the distribution of deep visual relationship labels. Step 5-3: Calculate the label distribution F of the deep visual relationship through the cross entropy loss function as shown in Formula 15 p and the true relation label F t The loss between them is used as the relationship loss L for model training rel ; Among them, F tj and F pj are the true relation label and deep visual relation label distribution of the jth relation sample, respectively, and N is the total number of relation samples on each image; 6. A testing method, characterized in that: The method for testing the output of the trained model in the scene graph generation method based on deep visual features as described in any one of claims 1 to 5 comprises the following steps: Step 1: Input the test image and the corresponding depth map into the above-mentioned trained model for relationship prediction, and then proceed to step 2; Step 2: Compare the relationship prediction results with the true predicate labels, evaluate the results using evaluation metrics, and proceed to step 3; Step 3: Output the relationship prediction results and evaluation results, and end.

7. The testing method according to claim 6, characterized in that: The test image in step 1 is selected from the test set divided in the VG150 dataset.

8. The testing method according to claim 6, characterized in that: In step 2, in order to compare the relationship prediction results with the true predicate labels, the Recall@K evaluation standard widely recognized in the current scene graph generation field is used to accurately evaluate the test effect of the model; the higher the value of this indicator, the better the test result, indicating that the model performs better when dealing with diverse background interference information; the Recall@K indicator measures the frequency with which the model correctly identifies the predicate category in its top K predicted relationships.

9. The testing method according to claim 8, characterized in that: The evaluation criterion Recall@K uses the R@50 or R@100 indicator.

Images