A method for introducing heterogenization constraints in scene graph generation

Abstract

The application discloses a method for introducing a heterogenization constraint in scene graph generation, which introduces a heterogenization constraint in the process of generating a scene graph through a heterogenization model, uses visual features of scene objects as object representations, discovers potential relationships of the objects by link prediction, generates mixed relationship features between the objects through a graph neural network, and uses gradient reversal confusion to weaken the relevance of the mixed relationship features and the object representations, so as to ensure the heterogeneity of the two; subsequently, the mixed relationship features are reconstructed by using an automatic encoder module, and finally, the reconstructed mixed relationship features are modeled, so that the relationship labels with heterogeneity are transmitted to the scene graph generation, and are used for generating a heterogenized scene graph. The application successfully ensures the heterogeneity of the generated scene graph by explicitly decoupling the relevance of the object feature space and the relationship feature space in the training process of a traditional scene graph generation method.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing technology and relates to scene understanding and object relationship detection in images. It is used to discover, locate and classify object pairs in images and their spatial and semantic relationships, and uses heterogeneous scene graphs for structured representation, which is a method to introduce heterogeneous constraints in scene graph generation. Background Technology

[0002] Scene graph generation (SGG) aims to detect objects and their relationships in a scene represented by a graph structure. It plays a fundamental role in many downstream applications, such as caption generation. [1] Multimodal dialogue [2] and visual Q&A [3] Etc. Early relationship prediction [4] Objects and relationships are treated as isolated entities. They only utilize local context in relationship prediction, which is insufficient to form a comprehensive understanding of the scene graph. To fully leverage global context, current SGG methods... [5,6] Integrating feature representations of objects and relationships improves the performance of object and relationship prediction. However, current SGG methods typically fuse object and relationship features into the same feature space, which can lead to large intra-class differences and ambiguous inter-class differences in features, resulting in mostly homogeneous scene graphs with a single type of relationship. Furthermore, interrelated object and relationship features can also lead to semantic ambiguity.

[0003] Therefore, a method is needed to decouple the feature spaces of objects and relationships during scene graph generation, thereby ensuring the heterogeneity of the generated scene graph. Compared with traditional homogeneous scene graph and semi-heterogeneous (implicit heterogeneous) scene graph methods, heterogeneous scene graphs can effectively hinder the flow of non-homogeneous correlation information between objects and relationships, thereby alleviating the problems of large intra-class differences and ambiguous inter-class differences, and improving the quality of scene graph generation.

[0004] Most existing research on scene graph generation can be categorized methodologically into four types or any combination thereof. Firstly, some studies explore the simplest and most effective basic feature transformation and combination methods in model design. [7] Features are extracted by a convolutional network and then input into a classifier for direct relationship prediction. [8] The success of these methods is often attributed to complex feature engineering. [9] Or a carefully designed loss function

[10] Secondly, the RNN-based architecture treats the scene graph as a special sequence with correlations.

[11] These models typically convert graphical structures directly into linear sequences, resulting in the loss of complex topological logic information.

[12] Some special methods, such as VCTree, utilize tree-structured RNNs to avoid distortion of the graph structure.

[13] Third, GNN-based methods are naturally suited for scene graph generation tasks. [14、15、16、17] Many modern GNNs can gather information from neighboring nodes and are robust to generating zero-sample relation triple combinations. This robustness is crucial for SGG, as even in the largest visual genome datasets, only 4.22% of relation triples are non-zero samples. Finally, some SGG methods employ popular strategies from knowledge graphs.

[18] The use of external knowledge is also categorized into this type because it often involves constructing knowledge graphs based on common sense. [19，20] .

[0005] In addition to basic SGG generation methods, current research focuses on plug-and-play methods within the general SGG architecture, and the method described in this invention belongs to this category. TDE is a well-known plug-and-play plugin for SGG inference; this method does not affect the model training process but directly manipulates the generated scene graph.

[21] However, since this method aims to address the long-tail distribution problem in SGG, it significantly reduces the recall metric, meaning that the real-world data distribution is ignored. PUM is another inference-stage plugin that does not affect model training; it optimizes relation prediction results by whitening the data distribution.

[22] However, it requires converting the general feature representation into a Gaussian distribution, which limits its ease of use. EMB is a model training method that proposes an energy-based model to replace binary cross-entropy loss.

[23] This method is effective, but it requires a disruptive modification to the original SGG method.

[0006] References:

[0007] [1] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. 2018. Exploring Visual Relationship for Image Captioning. In ECCV.

[0008] [2]L Liao,Y Ma,X He,R Hong,and T Chua.2018.Knowledge-aware multimodaldialogue systems.In ACM MM.

[0009] [3]S Antol,A Agrawal,J Lu,M Mitchell,D Batra,C Lawrence Zitnick,and DParikh.2015.VQA:Visual Question Answering.In ICCV.

[0010] [4]J.Johnson,R.Krishna,M.Stark,L.Li,D.A.Shamma,M.S.Bernstein,andL.Fei-Fei.2015.Image retrieval using scene graphs.In CVPR.

[0011] [5]Long Chen,Hanwang Zhang,Jun Xiao,Xiangnan He,Shiliang Pu,and Shih-Fu Chang.2019.Counterfactual critic multi-agent training for scene graphgeneration.In ICCV.

[0012] [6]Yikang Li,Wanli Ouyang,Bolei Zhou,Kun Wang,and XiaogangWang.2017.Scene graph generation from objects,phrases and region captions.InICCV.

[0013] [7]Hengyue Liu,Ning Yan,Masood Mortazavi,and Bir Bhanu.2021.FullyConvolutional Scene Graph Generation.In CVPR.

[0014] [8]Hai Wan,Yonghao Luo,Bo Peng,and Wei-Shi Zheng.2018.RepresentationLearning for Scene Graph Completion via Jointly Structural and VisualEmbedding.In IJCAI.

[0015] [9]Mengshi Qi,Weijian Li,Zhengyuan Yang,Yunhong Wang,and JieboLuo.2019.Attentive Relational Networks for Mapping Images to Scene Graphs.InCVPR.

[0016]

[10] Ji Zhang,Kevin J.Shih,Ahmed Elgammal,Andrew Tao,and BryanCatanzaro.2019.Graphical Contrastive Losses for Scene Graph Parsing.In CVPR.

[0017]

[11] Roei Herzig,Moshiko Raboh,Gal Chechik,Jonathan Berant,and AmirGloberson.2018.Mapping Images to Scene Graphs with Permutation-InvariantStructured Prediction.In NuerIPS.

[0018]

[12] Rowan Zellers,Mark Yatskar,Sam Thomson,and Yejin Choi.2018.NeuralMotifs:Scene Graph Parsing with Global Context.In CVPR.

[0019]

[13] Kaihua Tang,Hanwang Zhang,Baoyuan Wu,Wenhan Luo,and WeiLiu.2019.Learning to Compose Dynamic Tree Structures for Visual Contexts.InCVPR.

[0020]

[14] Rongjie Li,Songyang Zhang,Bo Wan,and Xuming He.2021.BipartiteGraph Network with Adaptive Message Passing for Unbiased Scene GraphGeneration.In CVPR.

[0021]

[15] Yikang Li,Wanli Ouyang,Bolei Zhou,Jianping Shi,Chao Zhang,andXiaogang Wang.2018.Factorizable Net:An Efficient Subgraph-Based Framework forScene Graph Generation.In ECCV.

[0022]

[16] Wenbin Wang,Ruiping Wang,Shiguang Shan,and XilinChen.2019.Exploring Context and Visual Pattern of Relationship for SceneGraph Generation.In CVPR.

[0023]

[17] Jianwei Yang,Jiasen Lu,Stefan Lee,Dhruv Batra,and DeviParikh.2018.Graph R-CNN for Scene Graph Generation.In ECCV.

[0024]

[18] Hanwang Zhang,Zawlin Kyaw,Shih-Fu Chang,and Tat-SengChua.2017.Visual Translation Embedding Network for Visual RelationDetection.In CVPR.

[0025]

[19] Weitao Wang,Ruyang Liu,Mingle Wang,Sen Wang,Xiaojun Chang,andYang Chen.2020.Memory-Based Network for Scene Graph with UnbalancedRelations.In ACM MM.

[0026]

[20] Alireza Zareian, Zhecan Wang, Haoxuan You, and Shih-FuChang. 2020. Learning Visual Commonsense for Robust Scene Graph Generation. InECCV.

[0027]

[21] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased Scene Graph Generation From Biased Training. In CVPR.

[0028]

[22] Gengcong Yang, Jingyi Zhang, Yong Zhang, Baoyuan Wu, and Yujiu Yang. 2021. Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation. In CVPR.

[0029]

[23] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. 2021. Energy-Based Learning for Scene Graph Generation. In CVPR. Summary of the Invention

[0030] The problem this invention aims to solve is: to capture high-level semantic information of a scene from a complex image visual scene, to construct feature representations of objects and relationships in the scene through a heterogeneous approach, thereby discovering, locating and classifying object pairs and their spatial and semantic relationships in a video, and to use scene graphs for structured representation.

[0031] The technical solution of this invention is as follows: a method for introducing heterogeneity constraints in scene graph generation. The scene graph generation model obtains object labels through object detection and relationship labels between objects through relationship prediction to generate a scene graph. A heterogeneity model is used to introduce heterogeneity constraints during the scene graph generation process. The heterogeneity model decouples the feature spaces of objects and relationships during scene graph generation and passes the heterogeneity results to the scene graph generation model to generate a heterogeneous scene graph. The training implementation of the heterogeneity model is as follows:

[0032] 1) Use the visual features of objects extracted by the scene graph generation model as object representations, use link prediction to initially determine whether there is a relationship between each pair of objects, initialize the graph topology of the scene, and obtain the feature map.

[0033] 2) Use a graph neural network to generate mixed relationship features between objects based on feature maps. The message passing process of the graph neural network is x′ i =σ(ω1·mean) j∈N′(i) (x j )+ω2·x i ), where x i Let N'(i) represent the feature of graph node i, N′(i) represent the set of neighboring nodes of graph node i that do not contain self-loops, ω1 and ω2 represent the learning parameters of the graph neural network, σ is the non-linear activation function, and mean represents the mathematical mean of the feature.

[0034] 3) Perform gradient inversion on the mixed relation features. First, use a multilayer perceptron to learn the part of the mixed relation features that is strongly correlated with the object representation. Predict the object label from the mixed relation features. Then, use gradient inversion to make the perceptron lose its ability to predict the object label. And radiate the generation of mixed relation features through the gradient backpropagation mechanism to weaken the correlation between mixed relation features and object representation, thereby ensuring the heterogeneity of the two.

[0035] 4) Reconstructing mixed relationship features using an autoencoder: An autoencoder consists of an encoder and a decoder. The encoder is used as a classifier to classify mixed relationship features and generate mixed relationship labels, and the decoder is used to reconstruct the mixed relationship features.

[0036] 5) Use Gaussian mixture modeling to model the distribution of mixed relation labels, and pass the heterogeneous mixed relation labels to the scene graph generation model to generate a heterogeneous scene graph.

[0037] This invention is based on Visual Genome (VG-150), the largest dataset in the scene graph generation task, and is evaluated on this benchmark. Experimental results show that the method of this invention is superior to existing methods, and the effectiveness of the method is confirmed by ablation experiments. Component analysis proves that the method of this invention has better performance and interpretability.

[0038] Compared with similar methods such as TDE, PUM and EMB mentioned in the background art, this invention proposes a model training method for heterogeneous scene graphs. It introduces heterogeneous constraints in scene graph generation, incrementally inserts them into the training process, and improves performance by >10% without any additional inference cost.

[0039] The key benefit of this invention is that it proposes a model training method for generating heterogeneous scene graphs. By training a heterogeneous model, heterogeneous constraints are introduced during scene graph generation. This heterogeneous model decouples the feature spaces of objects and relationships during scene graph generation. By capturing high-level semantic information from complex image visual scenes, and constructing feature representations of objects and relationships in the scene through heterogeneity, it discovers, locates, and classifies object pairs and their spatial and semantic relationships within the scene, and uses scene graphs for structured representation. This invention has been evaluated on the VG-150 dataset, and experimental results show that the proposed method outperforms existing methods. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method proposed in this invention.

[0041] Figure 2 This is a flowchart of the network model of the method of the present invention.

[0042] Figure 3 These are the experimental results of embodiments of the present invention.

[0043] Figure 4 The figures shown are illustrations of the results of an embodiment of the method of the present invention. Figure 4 In the diagram, (a)-(f) represent different scene images and their scene graphs, with the left side showing the input image and the detection results of objects in the image, and the right side showing the scene graph generated by this invention. Detailed Implementation

[0044] This invention proposes a method for introducing heterogeneity constraints in scene graph generation. Its aim is to capture high-level semantic information of complex image visual scenes, construct feature representations of objects and relationships within the scene through heterogeneity, thereby discovering, locating, and classifying object pairs and their spatial and semantic relationships in videos, and using scene graphs for structured representation. For example... Figure 2 As shown, this invention initializes the graph topology of the scene through link prediction, learns relation representations under context conditions using an improved graph neural network, ensures the heterogeneity of the scene graph using gradient inversion, ensures the retention of relation representation information using an autoencoder, and models and transmits heterogeneity using a Gaussian mixture model.

[0045] The implementation of the present invention is illustrated below through a preferred embodiment, the process of which is as follows:Figure 1 As shown.

[0046] 1. Generate target detection results and use link prediction to make a preliminary judgment on whether there is a relationship between each pair of objects.

[0047] The proposed method in this invention is a heterogeneous training approach for the traditional SGG method. It introduces heterogeneous constraints during scene graph generation using a heterogeneous model. Therefore, object detection is performed first in the traditional SGG process. For each object in the scene, object detection provides its initial predicted object label and visual feature representation. Subsequently, this invention utilizes link prediction to use the visual feature representation as input to the heterogeneous model, initially determining the probability of a correlation between each pair of objects.

[0048] Taking an image as an example, the link prediction process of this invention is as follows:

[0049] 1.1) For each pair of objects that may be related in the same scene, their visual features are stitched together and represented as follows: Given a single-layer fully connected network fc1, predict the probability e of a direct relationship between two independent objects. p ;

[0050] 1.2) For each F ij Calculate all in the same scene The self-attention weights between objects are multiplied by the weights and features and then input into a single-layer fully connected network fc2 to calculate the conditional probability e of the association between each pair of objects in the current scene. a ;

[0051] 1.3) For all f = {f1, f2, ..., f...} i}, perform global context encoding, input a single-layer fully connected network fc3 to obtain the global joint probability e of the set of possible associations in the current scene. c ;

[0052] 1.4) Predict the probability P of the final association relationship between each pair of objects: P = σ(ω(e) c +e a +e p )).

[0053] 2. Using graph neural networks to generate relation representations

[0054] Object detectors can only generate visual representations of objects, while relationships are invisible. Therefore, they need to be learned and transformed from the visual representations of objects. Generally, using graph learning neural networks to consider all object nodes in a scene simultaneously can introduce contextual information and enhance the expressive power of relationship representations. However, graph neural networks often suffer from oversmoothing, meaning that the representations of all relationship nodes in the same connected graph tend to converge. Therefore, as a further improvement, this invention proposes an improved graph neural network, the specific improvement of which is as follows:

[0055] 2.1) For a general graph neural network, its message passing process can be represented as x′ i =σ(ω1·mean) j∈N(i) (x j ), where N(i) represents the set of neighborhood nodes containing self-loops of graph node i;

[0056] 2.2) Update the message passing procedure to Where N′(i) represents the set of neighboring nodes of graph node i that do not contain self-loops.

[0057] 3. Perform gradient inversion on the relation representation:

[0058] To ensure the heterogeneity of object representations and relation representations, this invention employs a gradient inversion method. A multilayer perceptron (MLP) is used, which attempts to learn to predict object labels using relation representations, thus learning the strongly correlated parts of the relation representations with the object representations. After initial learning, the effect of gradient inversion is gradually strengthened, causing the perceptron to lose its ability to predict object labels. Furthermore, the generation of mixed relation features is achieved through gradient backpropagation, minimizing the correlation between mixed relation features and object representations, thereby ensuring their heterogeneity. In essence, this invention decouples the feature spaces of objects and relations. The specific steps of this process are as follows:

[0059] 3.1) Perform gradient separation for each independent object representation and remove it from the computation graph. The computation graph represents a specific structure in the deep learning engine. This is a technical term in this field and will not be described in detail here.

[0060] 3.2) For the object representation after gradient separation, input it into a multilayer perceptron to predict the possible object categories;

[0061] 3.3) For the object category output by the multilayer perceptron, its positive gradient g is calculated using binary cross-entropy;

[0062] 3.4) For the positive gradient g, calculate its reverse gradient.

[0063] 3.5) During training, continuously update the value of iter and increase the weight of the inverted gradient.

[0064] 4. Reconstruct relation representations using an autoencoder:

[0065] Considering that mapping mixed relation features to relation labels is essentially an irreversible projection from high-dimensional data to low-dimensional data, this invention introduces an autoencoder. An autoencoder consists of an encoder and a decoder. The encoder typically encodes high-dimensional vectors into low-dimensional vectors, and the decoder typically decodes low-dimensional vectors into high-dimensional vectors. This invention uses the encoder as a classifier to classify mixed relation features and generate mixed relation labels, and then uses the decoder to reconstruct the mixed relation features, thereby minimizing information loss during the mapping process from high-dimensional data to low-dimensional data. The specific process is as follows:

[0066] 4.1) Using the mixed relation features R as input to the encoder, calculate the mixed relation label for each scene. The specific numerical distribution;

[0067] 4.2) As input to the decoder, the hybrid relation features are reconstructed, denoted as R′;

[0068] 4.3) Calculate the reconstruction loss The autoencoder is trained using reconstruction loss.

[0069] 5. Using Gaussian mixture modeling to convey heterogeneity:

[0070] Since the method proposed in this invention is geared towards general SGG generation methods, in order to ensure the lightweight and non-destructive implantation characteristics of the heterogeneous model training method proposed in this invention, this invention utilizes a Gaussian mixture model to model the output data distribution of the encoder, thereby establishing a correlation mapping with traditional, general SGG methods. In this way, the heterogeneity characteristics preserved during the heterogeneous model training process are propagated to traditional SGG generation. The specific process is as follows:

[0071] 5.1) The optimization objective of Gaussian mixture modeling is expressed as: in Let P represent the mixed relation label, τ represent the confidence function, and P represent the confidence function. oi,oj This represents a single relation label, meaning that SGG obtains relation labels between objects through relation prediction. By minimizing the Euclidean distance between the mixed relation labels and the single relation labels generated by SGG, heterogeneity is transferred from the mixed relation labels to the scene graph generated by SGG.

[0072] 5.2) Calculate the optimization loss Train a Gaussian mixture model.

[0073] The heterogeneous model implementation process of this invention is a model training method. The existing scene graph generation process is "constructing a scene graph model - model training - generating a scene graph". The scene graph generation process under the method of this invention is "constructing a scene graph model - heterogeneous model training - generating a heterogeneous scene graph". Among them, "heterogeneous model training" is the core content of this invention. Constructing a scene graph model is an existing technology, referring to various general scene graph models. Generating a scene graph is the capability of the scene graph model. This invention inserts a heterogeneous model into the existing scene graph model to introduce heterogeneous constraints and assist the existing scene graph generation model in generating heterogeneous scene graphs.

[0074] To verify the effectiveness of the proposed model training method for generating heterogeneous scene graphs, this invention was tested on VG-150, the largest SGG dataset currently available. To verify the generality of the method, training and testing were performed on seven benchmark SGG methods. Figure 3 The quantitative results show that the present invention does not suffer from large intra-class differences or ambiguous inter-class differences, and its method is superior to existing methods. The first group presents experimental results for five recent SGG methods; the first row of groups 2-8 presents experimental results for seven benchmark SGG methods; the second row of groups 2-8 presents experimental results for the seven benchmark SGG methods after using the model training method proposed in this invention. The last group represents the average result of the proposed method on each test metric.

[0075] The evaluation process employed three common tasks from SGG: Predicate Classification (PredCls), Scene Graph Classification (SGCls), and Scene Graph Detection (SGDet). PredCls typically ignores the performance of the object detection model, directly providing manually labeled object locations and categories on the image; SGCls generally only provides the true location of objects, not their true category; SGDet most closely approximates the real scene, providing only the original image without any additional information. The evaluation metrics used were the most commonly used metrics in SGG research: Mean Recall@K (mR@K) and Recall@K (R@K). K was set to 50 and 100, respectively. R@K is the earliest and most widely accepted metric, evaluating the recall of the top K predictions. mR@K addresses the long-tail effect, calculating the recall of all predicate categories separately and then averaging the results.

[0076] like Figure 4This demonstrates the final generation effect of the heterogeneous scene graph generated by the method of the present invention, which assists in the generation of the traditional SGG model. Figure 4 (a)-(f) represent different scene images and their scene graphs, respectively. The left side shows the input image and the detection results of objects in the image. The box lines in the figure are a detection illustration, indicating that object detection is performed on the image. The right side shows the scene graph generated by the present invention, which shows the objects and the relationships between them. There are more than one type of relationship in the figure. The various scene images show that the present invention can ensure the heterogeneity of the generated scene graph.

Claims

1. A method for introducing heterogeneity constraints in scene graph generation, wherein the scene graph generation model obtains object labels through object detection, obtains relationship labels between objects through relationship prediction, and generates a scene graph, characterized by: A heterogeneous model is introduced to introduce heterogeneous constraints during the scene graph generation process. This model decouples the feature spaces of objects and relationships during scene graph generation, and the heterogeneous results are passed to the scene graph generation model to generate heterogeneous scene graphs. The training implementation of the heterogeneous model is as follows: 1) Use the visual features of objects extracted by the scene graph generation model as object representations, use link prediction to initially determine whether there is a relationship between each pair of objects, initialize the graph topology of the scene, and obtain the feature map. 2) Use a graph neural network to generate mixed relationship features between objects based on feature maps. The message passing process of the graph neural network is x′ i =σ(ω1·mean) j∈N′(i) (x j )+ω2·x i ), where x i Let N'(i) represent the feature of graph node i, N′(i) represent the set of neighboring nodes of graph node i that do not contain self-loops, ω1 and ω2 represent the learning parameters of the graph neural network, σ is the non-linear activation function, and mean represents the mathematical mean of the feature. 3) Perform gradient inversion on the mixed relation features. First, use a multilayer perceptron to learn the part of the mixed relation features that is strongly correlated with the object representation. Predict the object label from the mixed relation features. Then, use gradient inversion to make the perceptron lose its ability to predict the object label. And radiate the generation of mixed relation features through the gradient backpropagation mechanism to weaken the correlation between mixed relation features and object representation, thereby ensuring the heterogeneity of the two. 4) Reconstructing mixed relationship features using an autoencoder: An autoencoder consists of an encoder and a decoder. The encoder is used as a classifier to classify mixed relationship features and generate mixed relationship labels, and the decoder is used to reconstruct the mixed relationship features. 5) Use Gaussian mixture modeling to model the distribution of mixed relation labels, and pass the heterogeneous mixed relation labels to the scene graph generation model to generate a heterogeneous scene graph.

2. The method for introducing heterogeneity constraints in scene graph generation according to claim 1, characterized in that: The link prediction process is as follows: 1.1) For each pair of objects that may be related in the same scene, their visual features are stitched together and represented as follows: Where i represents the i-th graph node, and e represents the probability e that two independent objects are directly related, predicted by a single-layer fully connected network. p ; 1.2) For each F ij Calculate all in the same scene The self-attention weights between objects are multiplied by the self-attention weights and features, and then passed through a single-layer fully connected network to calculate the conditional probability e of the association between each pair of objects in the current scene. a ; 1.3) For all f = {f1, f2, ..., f...} i Global context encoding is performed, and the global joint probability e of the set of possible relationships in the current scene is obtained through a single-layer fully connected network. c ; 1.4) Predict the probability P of the final association relationship between each pair of objects: P = σ(ω(e) c +e a +e p )), where σ represents the nonlinear activation function and ω represents the parameters that need to be learned in the process.

3. The method for introducing heterogeneity constraints in scene graph generation according to claim 1, characterized in that: Gradient reversal of relation representations specifically involves: 3.1) Perform gradient separation for each individual object representation and remove it from the computation graph; 3.2) For the object representation after gradient separation, input it into the multilayer perceptron to predict the object category; 3.3) For the object category output by the multilayer perceptron, its positive gradient g is calculated using binary cross-entropy; 3.4) For the positive gradient g, calculate its reverse gradient. Where ∈ represents the artificially preset inversion efficiency parameter. The larger ∈ is, the faster the inverted gradient g′ will approach g. iter represents the number of iterations. 3.5) During training, continuously update the value of iter and increase the weight of the inverted gradient.

4. The method for introducing heterogeneity constraints in scene graph generation according to claim 1, characterized in that it involves reconstruction. The relation representation is specifically as follows: 4.1) Using the relation representation features R as input to the encoder, calculate the hybrid relation label for each scene. The specific numerical distribution; 4.2) As input to the decoder, the reconstructed relation representation is denoted as R′; 4.3) Calculate the reconstruction loss The autoencoder is trained using reconstruction loss.

5. A method for introducing heterogeneity constraints in scene graph generation according to claim 1, characterized in that: Using Gaussian mixture modeling to convey heterogeneity specifically involves: 5.1) The optimization objective of Gaussian mixture modeling is expressed as: in Let P represent the mixed relation label, τ represent the confidence function, and P represent the confidence function. oi,oj Representing a single relation label, heterogeneity is transferred from the mixed relation label to the scene graph generated by SGG by minimizing the Euclidean distance between the mixed relation label and the single relation label generated by SGG; 5.2) Calculate the optimization loss Train a Gaussian mixture model.

Images