A method for generating unbiased scene graphs that mitigate long-tailed distributions

Abstract

The application discloses a kind of unbiased scene graph generation methods for alleviating long tail distribution, comprising the following steps: S1, model construction;S2, data preprocessing;S3, object feature extraction;S4, graph learning structure (GLS) construction;S5, regional message passing network (RMPN);S6, pseudo-label generation;S7, loss function definition;S8, model training;S9, pseudo-label and triple generation;S10, iterative training and optimization.The unbiased scene graph generation methods for alleviating long tail distribution, in the application, the correlation between entities is calculated using GLS and the relationship graph is optimized, enhancing the relationship representation of head and tail classes, while RMPN improves the semantic representation of objects and relationships through information transmission mechanism, pseudo-labels are generated based on unlabeled relationships in the training set through a pseudo-label generation mechanism, and pseudo-triples are generated to enhance the learning samples of tail classes through a high-confidence screening mechanism.

Description

Technical Field

[0001] This invention relates to the field of scene graph generation technology, specifically to an unbiased scene graph generation method that alleviates long-tail distribution. Background Technology

[0002] Scene graph generation is a core task in the field of computer vision. Its purpose is to extract entities from visual scenes and identify the semantic relationships between these entities. With the deep integration of visual language models with fields such as autonomous driving and intelligent security, scene graph generation (SGG) technology has become the core foundation of high-order visual reasoning. By structurally representing objects in images and their semantic relationships, SGG provides interpretable semantic support for tasks such as visual question answering and cross-modal retrieval.

[0003] In complex open-world scenarios, the naturally occurring long-tail relation distribution leads to serious prediction bias in existing methods. According to statistics, in the mainstream dataset VisualGenome, the top 20% of high-frequency predicates account for 82.3% of the labeled data, while the tail 50% of relation categories only cover 3.1% of the instances. This data imbalance significantly reduces the reliability of the model in safety-critical scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide an unbiased scene graph generation method that alleviates the problem of prediction bias in existing scene graph generation caused by the naturally existing long-tail relationship distribution, as mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for generating unbiased scene graphs to alleviate long-tail distribution, comprising the following steps:

[0006] S1. Model Construction: Construct an unbiased scene graph generation model based on Graph Learning Structure (GLS) and Region Message Passing Network (RMPN);

[0007] S2. Data preprocessing: Obtain image data from the VisualGenome (VG) dataset, including object and relation annotations, and divide the dataset into 80% training set and 20% test set according to the standard partitioning method.

[0008] S3. Object Feature Extraction: The Faster R-CNN model is used to detect objects in the image, extract object region features, and ROIAlign is used to improve the feature resolution of small targets.

[0009] S4. Graph Learning Structure (GLS) Construction: Construct a relation graph based on Graph Learning Structure (GLS), including the connection between objects and relations, calculating the correlation between entities, and performing edge self-supervised optimization to enhance the relation representation of head and tail categories;

[0010] S5. Regional Message Passing Network (RMPN): The Regional Message Passing Network (RMPN) is used to extract regional features at different scales and improve the semantic representation of objects and relationships through the information passing mechanism.

[0011] S6. Pseudo-label generation: Based on the unlabeled relations in the training set, pseudo-labels are generated using a relation classifier, and combined with a high-confidence screening mechanism, pseudo-triples are generated to enhance the learning samples of the tail category.

[0012] S7. Loss function definition: Set the basic supervised loss (L_sup), pseudo-label loss (L_pseudo), and graph structure constraint loss (L_graph), and integrate them into the total loss function for training;

[0013] S8. Model Training: Configure training parameters and perform iterative training to optimize the model's performance on the head and tail categories.

[0014] S9. Pseudo-labels and triplet generation: During training, based on the pseudo-label generation mechanism and the triplets output by the model, the model's generalization ability to tail categories is further optimized through dynamic weight adjustment.

[0015] S10. Iterative training and optimization: Repeat S4-S9 until the loss function converges to obtain a trained unbiased scene graph generation model, which effectively solves the long-tail distribution problem.

[0016] Preferably, in step S1, the unbiased scene graph generation model includes a graph learning structure module, a region message passing network module, and a pseudo-label generation module. The graph learning structure module is used to learn the correlation between entities and optimize the relationship graph; the region message passing network module is used to pass information between different regions to enhance the representation of entities and relationships; and the pseudo-label generation module is used to generate relationships between unlabeled entity pairs.

[0017] By adopting the above technical solution, the collaborative work of the three modules enhances the scene graph generation model's ability to handle complex relationships.

[0018] Preferably, the image data obtained in step S2 includes object and relation annotations, the image dataset includes a training set and a test set, the training set accounts for 80%, the test set accounts for 20%, and data augmentation is performed on the dataset, the data augmentation includes rotation and cropping;

[0019] In step S3, the Faster R-CNN model is used to perform object detection on the image, extract the features of each object, and improve the resolution of the target features through ROIAlign. The object features are then input into the GLS and RMPN modules for further processing.

[0020] The above technical solution lays a solid foundation for subsequent model learning and prediction.

[0021] Preferably, in the GLS of step S4, to capture the correlation between entities and improve the prediction of long-tailed distribution relationships, the main goal of the GLS is to construct a graph structure by enhancing the representation of nodes and edges in the graph, where nodes represent objects and edges represent relationships. The correlation between nodes is learned through a graph attention network (GAT), as shown in the following formula:

[0022]

[0023] in, N represents the updated feature vector of node i. i Indicates the neighboring nodes of node i. Let be the degree of node i. Let W be the degree of node j. (l) Let σ be the weights of the graph convolutional layer, and σ be the activation function. Let j be the original feature vector of the neighbor node j at layer l;

[0024] An edge self-supervised mechanism is adopted to optimize the edge representation, remove unnecessary edges, and supplement missing edges. The formula is as follows:

[0025]

[0026] Among them, L GLS Let o be the edge self-supervised loss function; i and r j w represents objects and relations respectively. i This represents the weights obtained through learning.

[0027] By adopting the above technical solution, the ability of the model to handle long-tailed distribution relationships can be enhanced by optimizing the graph structure.

[0028] Preferably, in step S5, the semantic representation of each node is further enhanced through an information propagation mechanism, and the reasoning ability of relationships between nodes is improved. Step S5 enhances the model's ability to predict complex relationships through multiple information transmissions and cross-scale feature fusion. Each node generates a more accurate relationship representation after multiple message passing steps. The representation formula of the message passing process is as follows:

[0029]

[0030] in, Let be the feature representation of node i after being updated at layer k+1. This represents the message passing matrix at step k. Let be the representation of neighbor node j at step k.

[0031] By adopting the above technical solution, the model performance is improved in three aspects: enhancing semantic representation, improving relational reasoning, and enhancing complex relational prediction, thereby making scene graph generation more accurate.

[0032] Preferably, in step S6, pseudo-labels are generated and optimized by predicting unlabeled relation triples to further expand the training dataset. The generation of pseudo-labels mainly involves inferring missing relations in the model. These relations are tail-category relations, increasing the amount of tail-category data, and introducing high-confidence pseudo-labels to avoid the influence of noise. The pseudo-label generation process is as follows:

[0033] The model infers unlabeled relationships based on object pairs and learned relationship graphs;

[0034] Use a relation classifier to predict these unlabeled triples;

[0035] And generate pseudo-labels based on the prediction results;

[0036] The pseudo-labels are the predicted relationships assigned by the model to unlabeled entity pairs. The formula for generating the pseudo-labels is as follows:

[0037]

[0038] in, This represents the relation label predicted by the model, and Conf represents the model's confidence level. This represents the predicted probability.

[0039] By adopting the above technical solutions, expanding training data, enhancing tail category learning, and ensuring data quality, the performance of the model on low-frequency relationships can be improved.

[0040] Preferably, the loss function in step S7 includes basic supervision loss, pseudo-label loss, and graph structure constraint loss. These three losses work synergistically to ensure the model can balance the learning of head and tail categories. The supervision of head category learning uses cross-entropy loss, enabling the model to predict labeled relation triples as accurately as possible. The formula for the basic supervision loss is as follows:

[0041]

[0042] Among them, L sup Based on the monitoring loss, yi For real labels, p i This represents the predicted probability distribution.

[0043] For relation triples generated by pseudo-labels, the quality of pseudo-labels is constrained by calculating the difference between pseudo-labels and real labels, using KL divergence to help the model learn effectively on tail categories. The pseudo-label loss formula is as follows:

[0044] L pseudp =KL(p pseudo ||p true )#Formula 6

[0045] Among them, L pseudp For pseudo-label loss, KL() is used for divergence calculation, p pseudo p is the predicted distribution of the pseudo-labels. true The distribution of the true labels;

[0046] Graph structure regularization ensures the connectivity and consistency of the graph structure. Similar entities and relationships should have similar representations, and there should be no misleading connections between unrelated entities. The graph structure constraint loss formula is as follows:

[0047]

[0048] Among them, L graph For graph structure constraint loss, w i,j r represents the weight between entity i and entity j. i and r j These represent the relationship between entity i and entity j, respectively.

[0049] Combining the basic supervision loss, pseudo-label loss, and graph structure constraint loss, the final loss function is as follows:

[0050] L total =L sup +α1L pseudo +α2L graph #Formula8

[0051] Here, α1 and α2 are parameters that balance different loss terms, with α1 being the weight that controls the pseudo-label loss and α2 being the weight that controls the graph structure loss.

[0052] By employing the above technical solution and combining multiple loss functions, the overall performance of the model can be improved.

[0053] Preferably, in step S9, priority is given to stabilizing classification accuracy, weighted cross-entropy loss is used to suppress overfitting, pseudo-label contrast loss is used to enhance feature discrimination, and dynamic weight adjustment is combined to ensure that the model can balance the accuracy of high-frequency relationships and the recall of low-frequency relationships under complex long-tail distributions.

[0054] By adopting the above technical solution and adjusting the weights dynamically, the accuracy of high-frequency relationships and the recall of low-frequency relationships can be guaranteed even under complex long-tail distributions.

[0055] Preferably, after one iteration of the iterative training in step S10, the model based on multiple unbiased scene graphs is validated using the validation set in the VG dataset to prevent the model from overfitting.

[0056] By adopting the above technical solution and iteratively verifying the model, overfitting can be prevented.

[0057] Compared with the prior art, the beneficial effects of the present invention are: the method for generating unbiased scene graphs that alleviates long-tail distribution:

[0058] 1. In this invention, GLS is used to calculate the correlation between entities and optimize the relationship graph, thereby enhancing the relationship representation of head and tail categories. At the same time, RMPN improves the semantic representation of objects and relationships through the information transmission mechanism. Through the pseudo-label generation mechanism, pseudo-labels are generated based on the unlabeled relationships in the training set. Combined with the high confidence screening mechanism, pseudo triples are generated to enhance the learning samples of tail categories. This enables the model to effectively learn low-frequency relationships when processing long-tailed distributed data, improve the processing ability of tail category data, and thus balance the learning effect of the model on head and tail categories.

[0059] 2. In this invention, basic supervision loss, pseudo-label loss, and graph structure constraint loss are set and integrated into a total loss function for training. The different loss functions work together to ensure that the model takes into account different aspects during the learning process. It can accurately predict labeled relation triples and effectively learn tail categories, while ensuring the connectivity and consistency of the graph structure, thereby improving the accuracy of scene graph generation. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the process structure of the present invention;

[0061] Figure 2 This is a schematic diagram of the overall structure of the scene graph-based model for mitigating long-tail distribution in this invention.

[0062] Figure 3 This is a schematic diagram of the graph learning structure and the regional information transmission module structure of the present invention;

[0063] Figure 4This is a schematic diagram of the test result structure of the method of the present invention and other test models. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Please see Figures 1-4 The present invention provides a technical solution: a method for generating unbiased scene graphs to alleviate long-tail distribution.

[0066] S1. Model Construction: Construct an unbiased scene graph generation model based on Graph Learning Structure (GLS) and Region Message Passing Network (RMPN);

[0067] S2. Data preprocessing: Obtain image data from the VisualGenome (VG) dataset, including object and relation annotations, and divide the dataset into 80% training set and 20% test set according to the standard partitioning method.

[0068] S3. Object Feature Extraction: The Faster R-CNN model is used to detect objects in the image, extract object region features, and ROIAlign is used to improve the feature resolution of small targets.

[0069] S4. Graph Learning Structure (GLS) Construction: Construct a relation graph based on Graph Learning Structure (GLS), including the connection between objects and relations, calculating the correlation between entities, and performing edge self-supervised optimization to enhance the relation representation of head and tail categories;

[0070] S5. Regional Message Passing Network (RMPN): The Regional Message Passing Network (RMPN) is used to extract regional features at different scales and improve the semantic representation of objects and relationships through the information passing mechanism.

[0071] S6. Pseudo-label generation: Based on the unlabeled relations in the training set, pseudo-labels are generated using a relation classifier, and combined with a high-confidence screening mechanism, pseudo-triples are generated to enhance the learning samples of the tail category.

[0072] S7. Loss function definition: Set the basic supervised loss (L_sup), pseudo-label loss (L_pseudo), and graph structure constraint loss (L_graph), and integrate them into the total loss function for training;

[0073] S8. Model Training: Configure training parameters and perform iterative training to optimize the model's performance on the head and tail categories.

[0074] S9. Pseudo-labels and triplet generation: During training, based on the pseudo-label generation mechanism and the triplets output by the model, the model's generalization ability to tail categories is further optimized through dynamic weight adjustment.

[0075] S10. Iterative training and optimization: Repeat S4-S9 until the loss function converges to obtain a trained unbiased scene graph generation model, which effectively solves the long-tail distribution problem.

[0076] like Figure 1 , Figure 2 and Figure 3 As shown, an unbiased scene graph generation method using the following steps is used to alleviate long-tail distribution: A scene graph generation model based on multi-scale feature fusion and graph learning is constructed. Images containing object and relation annotations are collected from the Visual Genome dataset, with images appearing less than 50 times. Low-frequency relations are filtered out, retaining 80% and 20% of the training set containing long-tail distribution. Faster R-CNN is used to detect object bounding boxes and categories in the images. 7×7 region features are extracted from the original image (s=0), 2x (s=1), and 4x (s=2) downsampled feature maps. Multi-scale features are unified to 256 dimensions using 1×1 convolution. The extracted multi-scale features are used to extract textual description features of object regions using the CLIP model, concatenated with visual features to enhance the semantic representation of low-frequency relations. Each object in the output supplemented features is treated as a graph node. Cosine similarity is calculated by initializing the weights of fully connected edges, removing edges with weights below a threshold, and retaining Top-K connections. Latent semantic edges are added based on CLIP text-image alignment scores, according to predicate categories (space / attribute / action). Independent message passing paths are allocated. High-confidence relationships predicted by the model are selected from the validation set. The matching degree between image regions and text descriptions, such as "person-carrying-backpack," is calculated using CLIP. Pseudo-labels with consistency scores > 0.6 are retained. The basic supervision loss, pseudo-label loss, and graph sparsity constraint are calculated separately. The basic supervision loss is calculated using weighted cross-entropy loss, and the pseudo-label loss is aligned with the pseudo-label feature space using contrastive loss. In stage 1, the graph structure is fixed, and only feature extraction and classification heads are optimized. In stage 2, the graph learning module and message passing network are jointly optimized, and the pseudo-label library is dynamically updated. The loss weights are adjusted according to the long-tail performance of the validation set. If the recall rate of low-frequency relationships improves, the pseudo-label loss weight is increased. The pseudo-label loss weight is increased, and the graph constraint weight is decreased. The above operations are repeated until a trained unbiased scene graph generation model with mitigated long-tail distribution is established. The obtained detection images are input into the trained unbiased scene graph generation model with mitigated long-tail distribution for scene graph generation effect testing.

[0077] In step S1, the unbiased scene graph generation model includes a graph learning structure module, a region message passing network module, and a pseudo-label generation module. The graph learning structure module is used to learn the correlation between entities and optimize the relationship graph; the region message passing network module is used to pass information between different regions to enhance the representation of entities and relationships; and the pseudo-label generation module is used to generate relationships between unlabeled entity pairs.

[0078] like Figure 2 As shown, the overall unbiased scene graph generation model generates object region features and category labels from object bounding boxes detected in images using Faster R-CNN, and simultaneously extracts predicate candidate features for subsequent relational semantic modeling. For the encoder module, multi-scale fusion of object features enhances the representation ability of small targets. The graph learning structure constructs a fully connected relation graph based on object and predicate features, with nodes representing objects and edge weights initialized by the cosine similarity of predicate features. The region message passing network allocates independent message passing paths according to the predicate type to avoid semantic confusion. The pseudo-label generation and unbiased prediction select high-confidence predictions from the validation set, and add them to the training set after CLIP cross-modal validation, focusing on supplementing low-frequency relation samples to alleviate long-tail distribution bias.

[0079] The image data acquired in step S2 includes object and relation annotations. The image dataset consists of a training set and a test set, with the training set accounting for 80% and the test set accounting for 20%. Data augmentation is performed on the dataset, including rotation and cropping. In step S3, the Faster R-CNN model is used to perform object detection on the images, extracting features for each object and improving the resolution of the target features through ROIAlign. The object features are then input into the GLS and RMPN modules for further processing.

[0080] Object detection and initial feature extraction use Faster R-CNN to measure object bounding boxes and class labels in the input image. Multi-scale feature maps are extracted from the following three layers of the backbone network, such as ResNet-50: original image resolution (s=0): preserves high-resolution details such as small object edges; 2x downsampling (s=1): captures mid-range context; 4x downsampling (s=2): extracts global semantics; ROIAlign region feature alignment: for each detected object region, the ROIAlign operation is performed on the feature maps of s=0, s=1, and s=2 respectively to generate aligned 7×7 region features.

[0081] In the S4 step of GLS, to capture the correlations between entities and improve the prediction of long-tailed distribution relationships, the main goal of GLS is to construct a graph structure by enhancing the representation of nodes and edges in the graph. Nodes represent objects, edges represent relationships, and the correlations between nodes are learned through a graph attention network (GAT), as shown in the following formula:

[0082]

[0083] Where, N i Indicates the neighboring nodes of node i. W represents the node degree. (l) σ represents the weights of the graph convolutional layer, and σ is the activation function.

[0084] An edge self-supervised mechanism is adopted to optimize the edge representation, remove unnecessary edges, and supplement missing edges. The formula is as follows:

[0085]

[0086] Among them, o i and r j w represents objects and relations respectively. i This represents the weights obtained through learning;

[0087] The semantic representation of low-frequency relationships is enhanced by text-visual feature alignment. The CLIP text encoder is used to generate linguistic description text for each object region and the description text is converted into dense vectors that are aligned with visual features.

[0088] In step S5, the semantic representation of each node is further enhanced through the information propagation mechanism, and the reasoning ability of relationships between nodes is improved. Step S5 enhances the model's ability to predict complex relationships through multiple information transmissions and cross-scale feature fusion. Each node generates a more accurate relationship representation after multiple message passing steps. The formula for the message passing process is as follows:

[0089]

[0090] in, This indicates that the message passing at step k is assisting. Let j be the representation of neighbor node j at step k;

[0091] The robustness of relational reasoning is enhanced through a dynamic graph optimization mechanism, which optimizes the output of each object feature H. i ∈R 256 For each graph node, the weights of the fully connected edges are initialized based on cosine similarity, and the calculation formula is as follows: After calculating the weights, additional text-image alignment scores are calculated for the CLIP-supplemented features to enhance the weights of low-frequency relationships. During dynamic graph pruning, a threshold θ = 0.35 is set to remove edges with weights below this value, and the top-30 high-weight connections for each node are retained (corresponding to "retaining top-K connections") to ensure the sparsity and efficiency of the graph structure. When supplementing latent semantic edges, the CLIP model is used to generate text descriptions of object pairs, and the text-image alignment score S is calculated. clip ∈[0,1], when Sclip When the weight is greater than 0.6, add this semantic edge to the graph, with the following weight: This is to address the problem of missed detections caused by insufficient samples.

[0092] In step S6, pseudo-labels are generated and optimized. The training dataset is further expanded by predicting unlabeled relation triples. Pseudo-label generation primarily involves inferring missing relations in the model; these relations are tail-category relations, increasing the amount of data for tail-category relations, and introducing high-confidence pseudo-labels to avoid the influence of noise. The pseudo-label generation process is as follows:

[0093] The model infers unlabeled relationships based on object pairs and learned relationship graphs;

[0094] Use a relation classifier to predict these unlabeled triples;

[0095] And generate pseudo-labels based on the prediction results;

[0096] Pseudo-labels are predicted relationships assigned by the model to unlabeled entity pairs. The formula for generating pseudo-labels is as follows:

[0097]

[0098] in, This represents the relation label predicted by the model, and Conf represents the model's confidence level. Indicates the predicted probability;

[0099] Enhanced relational semantic representation through predicate-type driven multi-path message passing; transforming the relative positions of object pairs into 4-dimensional vectors based on spatial relationships; its attention calculation formula is as follows:

[0100]

[0101] In the formula, g ij W represents a four-dimensional vector. g This represents a learnable geometrically encoded weight matrix. Representing spatial attention weights, for path fusion and output, gating fusion is used to weight and merge the outputs of the three paths, as shown in the formula:

[0102]

[0103] Wherein, γ1, γ2, and γ3 are learnable gating weights with initial values ​​of 0.4, 0.3, and 0.3 respectively. Finally, the gating weights are stably trained using LayerNorm and the updated node features are output to the pseudo-label generation module.

[0104] The loss function in step S7 includes basic supervision loss, pseudo-label loss, and graph structure constraint loss. These three losses work together to ensure the model can balance learning the head and tail categories. The supervision for learning the head category uses cross-entropy loss, which enables the model to predict labeled relation triples as accurately as possible. The formula for basic supervision loss is as follows:

[0105]

[0106] Among them, y i For real labels, p i This represents the predicted probability distribution.

[0107] For relation triples generated by pseudo-labels, the KL divergence is used to constrain the quality of pseudo-labels by calculating the difference between pseudo-labels and real labels, which helps the model to learn effectively on tail categories. The pseudo-label loss formula is as follows:

[0108] L pseudp =KL(p pseudo ||p true )#Formula 6

[0109] Where, p pseudo p is the predicted distribution of the pseudo-labels. true The distribution of the true labels;

[0110] Graph structure regularization ensures the connectivity and consistency of the graph structure. Similar entities and relationships should have similar representations, and there should be no misleading connections between unrelated entities. The graph structure constraint loss formula is as follows:

[0111]

[0112] Among them, w i,j r represents the weight between entity i and entity j. i and r j These represent the relationship between entity i and entity j, respectively.

[0113] Combining the basic supervision loss, pseudo-label loss, and graph structure constraint loss, the final loss function is as follows:

[0114] L total =L sup +α1L pseudo +α2L graph #Formula8

[0115] Wherein, α1 and α2 are parameters for balancing different loss terms, α1 is the weight that controls the pseudo-label loss, and α2 is the weight that controls the graph structure loss.

[0116] High-quality pseudo-labels are generated through a cross-modal validation mechanism. Relationship triples predicted by the model are obtained from the validation set, with confidence scores p∈[0,1]. Subsequent preliminary filtering retains relations with confidence scores higher than a threshold (p>0.7), generating a candidate pseudo-label set. When validating cross-modal consistency, the CLIP image encoder is used to extract visual features V of the object region. i ∈R 512 And calculate the visual-text alignment score: Only when S clip If the value is greater than 0.6, retain the pseudo-tag;

[0117] By employing a multi-objective joint optimization approach to balance model performance and generalization ability of long-tail relationships, the weighted cross-entropy loss is calculated based on the long-tail distribution in the labeled data, assigning differentiated weights to high-frequency and low-frequency relationships. The calculation formula is as follows:

[0118] Where R represents the total number of categories, W r y represents the weight of class r. r p represents the true label of class r. r Let represent the predicted probability of class r. For pseudo-label loss, its calculation formula is:

[0119]

[0120] Where h i and h j The feature vectors after CLIP alignment have a temperature coefficient τ = 0.1, used to align the feature space distribution of pseudo-labels and labeled data, mitigating long-tail bias.

[0121] For graph sparsity constraints, L1 regularization is used to penalize redundant edge weights, thereby enhancing the interpretability of the graph structure.

[0122]

[0123] Where the coefficient λ = 0.05, and ε represents all edges in the dynamic graph;

[0124] Finally, for the overall loss function, a weighted summation is performed to combine the three types of losses and balance the supervision signal and model complexity. The calculation formula is: L total =0.5L CE +0.3L CL +0.2L L1 .

[0125] In step S9, priority is given to stabilizing classification accuracy. Weighted cross-entropy loss is used to suppress overfitting. Pseudo-label contrast loss is used to enhance feature discrimination. Combined with dynamic weight adjustment, the model is ensured to balance the accuracy of high-frequency relationships and the recall of low-frequency relationships under complex long-tail distribution. After one iteration of iterative training in step S10, the model based on multiple unbiased scene graph generation is validated using the validation set in the VG dataset to prevent the model from overfitting.

[0126] In the first stage, the frozen graph learning module and message passing network parameters are used to optimize multi-scale feature alignment capabilities through the feature encoder, and classification weights are updated using labeled data, prioritizing the stabilization of high-frequency relationship predictions. In the second stage, the joint training module dynamically adjusts edge weights and optimizes predicate-aware path weights. Based on the long-tail performance of the validation set, the loss weights are dynamically optimized. If the recall rate of low-frequency relationships does not increase sufficiently (<2%), the pseudo-label loss weights are increased; conversely, if the accuracy of high-frequency relationships decreases, the graph constraint weights are reduced. This process leads to a well-trained unbiased scene graph generation model that mitigates the long-tail distribution. After each training iteration, the validation machine from the VG dataset is used to verify the effectiveness of the unbiased scene graph generation model, preventing overfitting.

[0127] By calculating R@K and mR@K, the model's ability to predict visual relationships can be measured. Furthermore, by comparing the model's R@K and mR@K with the visualization results generated from the actual scene graphs and other models, its position and advantages among similar models can be clarified. Based on the test results, the model's shortcomings can be identified, providing direction and basis for further optimization and improvement of the model.

[0128] This technical solution was compared with other test models, and the model performance comparison results are shown in Table 1 and... Figure 4 As shown, a higher R@K metric indicates a higher proportion of correctly predicted top K candidate relations. The mR@K metric is the average of the independent recall rates for each predicate category, providing a fair assessment of the long-tail distribution.

[0129] Table 1

[0130]

[0131] Working principle: An unbiased scene graph generation model is constructed by fusing a graph learning structure and a region message passing network. Image data is obtained from the Visual Genome dataset and divided into training and test sets. Data augmentation operations such as rotation and cropping are performed, and Faster is utilized. The R-CNN model, combined with ROIAlign, extracts object features, providing a data foundation for subsequent relationship modeling. The GLS module learns the correlation between entities through a graph attention network, optimizes the relationship graph, and enhances the prediction of long-tailed relationships. The RMPN module passes information multiple times and integrates cross-scale features, strengthening node semantic representation and relationship reasoning capabilities, and improving the prediction ability of complex relationships. Based on unlabeled relationships in the training set, pseudo-labels are generated using a relationship classifier. Combined with a high-confidence screening mechanism, the tail category learning samples are expanded to alleviate the problem of imbalanced long-tailed data. Basic supervision loss, pseudo-label loss, and graph structure constraint loss are set and combined into a total loss function. During training, weighted cross-entropy loss is used to suppress overfitting, and pseudo-label contrast loss is used to enhance feature discrimination. The weights are dynamically adjusted to balance the accuracy of high-frequency relationships and the recall of low-frequency relationships. The model is iteratively trained, and a validation set is used to verify the model after each iteration to prevent overfitting, until the loss function converges, resulting in a trained model used to accurately generate scene graphs.

[0132] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

Claims

1. A method for generating unbiased scene graphs to alleviate long-tail distribution, characterized in that: Includes the following steps: S1. Model Construction: Construct an unbiased scene graph generation model based on Graph Learning Structure (GLS) and Region Message Passing Network (RMPN); S2. Data preprocessing: Obtain image data from the VisualGenome (VG) dataset, including object and relation annotations, and divide the dataset into training and test sets; S3. Object Feature Extraction: The Faster R-CNN model is used to detect objects in the image, extract object region features, and ROIAlign is used to improve the feature resolution of small targets. S4. Graph Learning Structure (GLS) Construction: Construct a relation graph based on Graph Learning Structure (GLS), including the connection between objects and relations, calculating the correlation between entities, and performing edge self-supervised optimization to enhance the relation representation of head and tail categories; S5. Regional Message Passing Network (RMPN): The Regional Message Passing Network (RMPN) is used to extract regional features at different scales and improve the semantic representation of objects and relationships through the information passing mechanism. S6. Pseudo-label generation: Based on the unlabeled relations in the training set, pseudo-labels are generated using a relation classifier, and combined with a high-confidence screening mechanism, pseudo-triples are generated to enhance the learning samples of the tail category. S7. Loss function definition: Set the basic supervised loss (L_sup), pseudo-label loss (L_pseudo), and graph structure constraint loss (L_graph), and integrate them into the total loss function for training; S8. Model Training: Configure training parameters and perform iterative training to optimize the model's performance on the head and tail categories. S9. Pseudo-labels and triplet generation: During training, based on the pseudo-label generation mechanism and the triplets output by the model, the model's generalization ability to tail categories is further optimized through dynamic weight adjustment. S10. Iterative training and optimization: Repeat S4-S9 until the loss function converges to obtain a well-trained unbiased scene graph generation model, which effectively solves the long-tail distribution problem.

2. The method for generating unbiased scene graphs to alleviate long-tail distribution according to claim 1, characterized in that: In step S1, the unbiased scene graph generation model includes a graph learning structure module, a region message passing network module, and a pseudo-label generation module. The graph learning structure module is used to learn the correlation between entities and optimize the relationship graph; the region message passing network module is used to pass information between different regions to enhance the representation of entities and relationships; and the pseudo-label generation module is used to generate relationships between unlabeled entity pairs.

3. The method for generating unbiased scene graphs to alleviate long-tail distribution according to claim 1, characterized in that: The image data obtained in step S2 includes object and relation annotations. The image dataset includes a training set and a test set, with the training set accounting for 80% and the test set accounting for 20%. Data augmentation is performed on the dataset, including rotation and cropping. In step S3, the Faster R-CNN model is used to perform object detection on the image, extract the features of each object, and improve the resolution of the target features through ROIAlign. The object features are then input into the GLS and RMPN modules for further processing.

4. The method for generating unbiased scene graphs to alleviate long-tail distribution according to claim 1, characterized in that: In the S4 step of GLS, to capture the correlations between entities and improve the prediction of long-tailed distribution relationships, the main goal of GLS is to construct a graph structure by enhancing the representation of nodes and edges in the graph. Nodes represent objects, and edges represent relationships. The correlations between nodes are learned through a Graph Attention Network (GAT), as shown in the following formula: in, N represents the updated feature vector of node i. i Indicates the neighboring nodes of node i. Let be the degree of node i. Let W be the degree of node j. (l) Let σ be the weights of the graph convolutional layer, and σ be the activation function. Let j be the original feature vector of the neighbor node j at layer l; An edge self-supervised mechanism is adopted to optimize the edge representation, remove unnecessary edges, and supplement missing edges. The formula is as follows: Among them, L GLS Let o be the edge self-supervised loss function; i and r j w represents objects and relations respectively. i This represents the weights obtained through learning.

5. The method for generating unbiased scene graphs to alleviate long-tail distribution according to claim 1, characterized in that: In step S5, the semantic representation of each node is further enhanced through an information propagation mechanism, and the reasoning ability of relationships between nodes is improved. Step S5 enhances the model's ability to predict complex relationships through multiple information transmissions and cross-scale feature fusion. Each node generates a more accurate relationship representation after multiple message passing steps. The representation formula of the message passing process is as follows: in, Let be the feature representation of node i after being updated at layer k+1. This represents the message passing matrix at step k. Let be the representation of neighbor node j at step k.

6. The method for generating unbiased scene graphs to alleviate long-tail distribution according to claim 1, characterized in that: In step S6, pseudo-labels are generated and optimized by predicting unlabeled relation triples to further expand the training dataset. The generation of pseudo-labels mainly involves inferring missing relations in the model. These relations are tail-category relations, which increases the amount of data for tail-category relations. High-confidence pseudo-labels are introduced to avoid the influence of noise. The pseudo-label generation process is as follows: The model infers unlabeled relationships based on object pairs and learned relationship graphs; Use a relation classifier to predict these unlabeled triples; And generate pseudo-labels based on the prediction results; The pseudo-labels are the predicted relationships assigned by the model to unlabeled entity pairs. The formula for generating the pseudo-labels is as follows: in, This represents the relation label predicted by the model, and Conf represents the model's confidence level. This represents the predicted probability.

7. The method for generating unbiased scene graphs to alleviate long-tail distribution according to claim 1, characterized in that: The loss function in step S7 includes basic supervision loss, pseudo-label loss, and graph structure constraint loss. These three losses work together to ensure the model can balance the learning of head and tail categories. The supervision for learning the head category uses cross-entropy loss, enabling the model to predict labeled relation triples as accurately as possible. The formula for the basic supervision loss is as follows: Among them, L sup Based on the monitoring loss, y i For real labels, p i This represents the predicted probability distribution. For relation triples generated by pseudo-labels, the quality of pseudo-labels is constrained by calculating the difference between pseudo-labels and real labels, using KL divergence to help the model learn effectively on tail categories. The pseudo-label loss formula is as follows: L pseudp =KL(p pseudo ||p true )#Formula 6 Among them, L pseudp For pseudo-label loss, KL() is used for divergence calculation, p pseudo p is the predicted distribution of the pseudo-labels. true The distribution of the true labels; Graph structure regularization ensures the connectivity and consistency of the graph structure. Similar entities and relationships should have similar representations, and there should be no misleading connections between unrelated entities. The graph structure constraint loss formula is as follows: Among them, L graph For graph structure constraint loss, w i,j r represents the weight between entity i and entity j. i and r j These represent the relationship between entity i and entity j, respectively. Combining the basic supervision loss, pseudo-label loss, and graph structure constraint loss, the final loss function is obtained as follows: L total =L sup +α1L pseudo +α2L graph #Formula8 Here, α1 and α2 are parameters that balance different loss terms, with α1 being the weight that controls the pseudo-label loss and α2 being the weight that controls the graph structure loss.

8. The method for generating unbiased scene graphs to alleviate long-tail distribution according to claim 1, characterized in that: In step S9, priority is given to stabilizing classification accuracy, weighted cross-entropy loss is used to suppress overfitting, pseudo-label contrast loss is used to enhance feature discrimination, and dynamic weight adjustment is combined to ensure that the model can balance the accuracy of high-frequency relationships and the recall of low-frequency relationships under complex long-tail distribution.

9. The method for generating unbiased scene graphs to alleviate long-tail distribution according to claim 1, characterized in that: After one iteration of the iterative training in step S10, the model based on multiple unbiased scene graphs is validated using the validation set in the VG dataset to prevent the model from overfitting.

Images