Scene graph generation method based on dynamic query distribution and decoupled attention mechanism

Abstract

The application discloses a scene graph generation method based on dynamic query distribution and decoupled attention mechanism, comprising the following steps: inputting a training image into a backbone network and a Transformer encoder; generating a relationship query through an adaptive semantic relationship query generation module; inputting the relationship query into a relationship query attention decoupling decoder; in a one-to-many relationship coarse positioning module, calculating a one-to-many stage loss; in a one-to-one relationship fine positioning and deduplication module, calculating a one-to-one stage loss; weighting and combining the one-to-many stage loss and the one-to-one stage loss to obtain a total loss, and training the scene graph generation; if the set batch processing size is reached, returning the image training; if all the training pictures are read, outputting the trained model. The application can select high-quality tokens from the encoder tokens to construct a relationship query through the adaptive semantic relationship query generation module.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and computer vision technology, and more specifically, relates to a method for generating scene graphs based on dynamic query allocation and decoupled attention mechanism. Background Technology

[0002] Scene Graph Generation (SGG) is an important task in computer vision, aiming to identify entities and predict visual relationships between them from images. Each relationship is represented by a triple, such as "person-rider-bicycle," where "person" is the subject instance, "bicycle" is the object instance, and "rider" is the relationship between them. This structured scene representation can provide accurate semantic understanding support for downstream tasks such as image captioning, visual question answering, and image retrieval. In recent years, one-stage methods based on DETR (Detection Transformer) have received widespread attention due to their simplicity. The DETR framework transforms scene graph generation into an ensemble prediction problem: the input image is processed by a CNN backbone network to extract feature maps, which are then processed by a Transformer encoder to generate encoder tokens containing contextual information. A predefined relation query interacts with the encoder tokens, and the Transformer decoder simultaneously predicts objects and relationships. Finally, the Hungarian algorithm is used for one-to-one ensemble matching to assign the true label to the prediction result. This end-to-end framework eliminates the need for manually designed modules such as Non-Maximum Suppression (NMS), greatly simplifying the training process.

[0003] However, one-stage scene graph generation methods based on DETR face a severe problem of supervision sparsity. Taking the widely used VG150 dataset as an example, each image contains only about 5.5 relation annotations on average, while DETR-based models typically use more than 100 relation queries. Furthermore, in the traditional one-to-one matching strategy, each real relation is assigned to only one query during training, resulting in only about 5.5 positive samples per image for optimizing the loss function. A large number of queries (over 95%) lack effective supervision signals, leading to slow model training convergence. To address this issue, the recent Hydra-SGG method (2025 ICLR) proposed a hybrid relation assignment strategy, combining one-to-one and one-to-many relation assignment methods. One-to-one assignment uses the Hungarian algorithm to assign the optimal matching query to each real relation; one-to-many assignment, based on an IoU threshold, additionally assigns multiple (e.g., 6) high-quality matching queries as positive samples to each real relation. The assignment score is defined as: ,when When the value exceeds a threshold T (e.g., T=0.4), the query is selected as a positive sample. Simultaneously, Hydra-SGG introduces an auxiliary decoder, the Hydra branch, which shares parameters with the main decoder but removes the self-attention layer. This makes it easier for different queries to generate repeated predictions for the same relation, combined with a one-to-many assignment strategy. In this way, Hydra-SGG increases the number of positive samples by more than 50% in each training iteration, achieving a performance of 18.2 mR@50 on the VG150 dataset with only 10 training epochs, approximately 10 times faster than the previous stage method in terms of convergence speed.

[0004] Although Hydra-SGG significantly improves training efficiency through hybrid relation assignment and Hydra branching, this method still has key drawbacks. First, Hydra-SGG uses randomly initialized relation queries, which lack semantic information and spatial priors. Encoder tokens correspond to specific locations in the feature map, containing contextual and spatial layout information for that region, but the query generation process of Hydra-SGG is completely decoupled from this valuable information. In the early stages of training, a large number of queries are irrelevant to the image content; even with the increase in the number of positive samples through one-to-many assignment, a large number of queries (approximately 70-80%) still consistently act as negative samples. Although each real relation can be assigned 6 queries, the actual number of high-quality matches remains limited due to the inconsistent query quality, and the large number of invalid queries slows down model convergence.

[0005] Secondly, Hydra-SGG's hybrid use of one-to-one and one-to-many assignment strategies within the same decoder leads to a "query ambiguity" problem: the same decoder weights need to simultaneously satisfy two conflicting optimization objectives. One-to-one assignment requires each query to focus on predicting a unique relation, suppressing duplicate predictions through self-attention; while one-to-many assignment allows multiple queries to predict the same relation, even encouraging duplicate predictions by removing self-attention. These two objectives interact antagonistically in the feature space: one-to-one assignment pushes queries to optimize in different directions to avoid duplication, while one-to-many assignment pulls queries towards the same true relation. This conflict leads to unstable training, frequent gradient direction conflicts, loss function oscillations, and difficulty in finding the optimal solution. Some queries are uncertain between the two optimization objectives, unable to achieve precise one-to-one matching or effectively participate in one-to-many supervision.

[0006] Furthermore, while Hydra-SGG incorporates the Hydra branch to facilitate one-to-many assignment, it is essentially just an auxiliary branch for parameter sharing and does not functionally decouple the one-to-one and one-to-many assignment strategies at the attention mechanism level. Functionally, cross-attention handles the interaction between the query and encoder tokens, extracting visual features and is suitable for coarsely locating multiple candidate relations; while self-attention handles mutual inhibition and deduplication between queries, suitable for fine-grained matching and disambiguation. However, while both branches of Hydra-SGG are optimized simultaneously during training, there is no clear division of labor: which stage focuses on utilizing one-to-many supervision? Which stage focuses on precise one-to-one matching? This results in both branches participating in relation prediction, but with unclear responsibilities, leading to an unbalanced loss function. and Simultaneous interaction with shared parameters leads to mixed gradient signals, affecting convergence speed and final performance. In summary, while existing one-stage scene graph generation methods based on DETR alleviate the supervised sparsity problem to some extent through hybrid relation assignment strategies, they still face three core challenges: First, low relation query quality; randomly initialized queries do not fully utilize the semantic and spatial information in the encoder, resulting in a large number of invalid queries. Second, query ambiguity; the use of hybrid assignment strategies in the same decoder leads to conflicting optimization objectives and training instability. Third, a lack of functional decoupling; the responsibilities of one-to-many coarse localization and one-to-one fine-tuning are not clearly defined at the attention mechanism level. These problems limit the learning efficiency and relation prediction accuracy of the model in sparsely labeled scenarios. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a scene graph generation method based on dynamic query allocation and a decoupled attention mechanism. By designing an adaptive semantic relation query generation module, it selects high-quality tokens carrying rich spatial and contextual information from the encoder tokens to construct relation queries, improving query quality from the source, reducing invalid queries, and increasing the positive sample hit rate. Simultaneously, by designing a relation query attention decoupled decoder, the attention mechanism is functionally divided into two independent stages: a coarse localization stage using cross-attention for one-to-many relations maximizes the use of supervision signals, while a fine deduplication stage using self-attention for one-to-one relations eliminates prediction ambiguity. A gradient blocking mechanism ensures independent optimization between the two stages, effectively avoiding conflicts in optimization objectives. This systematically solves the problems of supervision sparsity, query ambiguity, and lack of functional decoupling, significantly improving the training efficiency and relation prediction accuracy of the model in sparsely labeled scenarios.

[0008] To address at least one of the aforementioned technical problems, according to one aspect of the present invention, a method for generating scene graphs based on dynamic query allocation and decoupled attention mechanism is provided, comprising the following steps:

[0009] Step 1: Input the training images into the backbone network and Transformer encoder to generate encoder token features containing rich semantic information, then proceed to Step 2;

[0010] Step 2: Using the adaptive semantic relation query generation module, select the M most valuable tokens from the encoder tokens and generate relation queries based on the bidirectional cross-attention mechanism, then proceed to Step 3;

[0011] Step 3: Input the relation query into the relation query attention decoupling decoder. This decoder consists of two stages: a one-to-many relation coarse localization module and a one-to-one relation fine deduplication module. Proceed to Step 4.

[0012] Step 4: In the one-to-many relationship coarse localization module, the relationship query is updated using the cross-attention mechanism, the bounding boxes and classification results of the subject and object and the relationship category are predicted, the one-to-many label allocation strategy is adopted to increase the number of positive samples, the loss of the one-to-many stage is calculated, and then proceed to step 5.

[0013] Step 5: In the one-to-one relationship refinement and deduplication module, the multi-head self-attention mechanism is used to deduplicate the relationship query, eliminate duplicate predictions, use the Hungarian algorithm for one-to-one matching, calculate the loss of the one-to-one stage, and proceed to step 6.

[0014] Step 6: Weight the one-to-many stage loss and the one-to-one stage loss to obtain the total loss, and use it to train scene graph generation. Proceed to Step 7.

[0015] Step 7: If the set batch processing size is reached, return to Step 1; if all training images have been read, proceed to Step 8.

[0016] Step 8: Output the trained model and end.

[0017] Preferably, in step 1, the backbone network of this invention employs a Convolutional Neural Network (CNN). Each layer of the Transformer encoder includes a multi-head self-attention mechanism and a feedforward neural network, connected through residual connections and layer normalization, enabling each encoder token to capture global contextual information and local spatial information. The training images are all from the standard dataset VG150 for scene graph generation. This dataset consists of 108,077 images, containing the 150 most common instance classes and 50 predicate classes. This invention divides 70% of the images in the dataset into a training set and the remaining 30% into a test set.

[0018] Preferably, the detailed steps of the adaptive semantic relationship query generation module in step 2 of the present invention are as follows:

[0019] Step 2-1: Input all encoder token features F into the classification head MLP, which contains multiple fully connected layers and activation functions, and outputs the classification score of each token; calculate the threshold τ based on the classification score, and select the M tokens with the highest classification scores as the most valuable object representation tokens. These tokens have richer semantic information and more accurate spatial positioning capabilities compared to randomly initialized queries. The dimension of each token is 1×256.

[0020] Step 2-2: Copy the M selected tokens into two copies, using the first copy as the main query. The second document was used as the subject of the inquiry. Both have a dimension of M×256, which are used to represent the subject entity and the object entity in the scene, respectively;

[0021] Steps 2-3: To capture the bidirectional dependency between the subject and the object, calculate the cross-attention weights from the subject to the object. Cross-attention weights from object to subject :

[0022]

[0023]

[0024] Wherein, CA represents cross-attention calculation, which includes linear transformations of queries, keys, and values, as well as the calculation of attention weights and weighted summation operations; the subject-to-object attention weight dimension is M×256, reflecting the degree of attention each subject query pays to all object queries; the object-to-subject attention weight dimension is also M×256, reflecting the degree of attention each object query pays to all subject queries;

[0025] Steps 2-4: Based on the calculated cross-attention weights, enhance the original subject query and object query to include each other's contextual information:

[0026]

[0027]

[0028] The enhanced subject query not only retains the original subject information, but also incorporates the contextual information of the object; the enhanced object query also incorporates the contextual information of the subject.

[0029] Steps 2-5: Enhance the subject query and object query Perform a concatenation operation along dimension 1 to generate the final relation query Qrel:

[0030]

[0031] The relation query has a dimension of M×512, which includes subject information, object information, and interaction information between the two. Compared with traditional random initialization queries, it can more effectively represent the relations in the scenario, reduce invalid and redundant queries from the source, and improve the positive sample hit rate and training efficiency.

[0032] Preferably, the detailed steps of the one-to-many relationship coarse positioning module in step 4 of the present invention are as follows:

[0033] Step 4-1: Decompose the relation query Qrel with dimension M×512 into subject query and object query along the channel dimension:

[0034]

[0035] The decomposed subject query and object query dimensions are both M×256;

[0036] Step 4-2: Update the subject query and object query separately using the cross-attention mechanism and multilayer perceptron, so that they can fully interact with the encoder features F:

[0037]

[0038]

[0039] Where LN represents layer normalization, MLP represents multilayer perceptron, and CA represents cross-attention; the cross-attention operation allows the query to extract relevant visual information from the encoder features; layer normalization and residual connections ensure the stability of training; the multilayer perceptron further performs nonlinear transformations on the features; the updated subject query and object query are denoted as... and ;

[0040] Step 4-3: Query the updated subject and object query The data is input into separate prediction heads to generate bounding box predictions for subjects and objects, classification predictions, and relationship predictions.

[0041]

[0042]

[0043]

[0044] Among them, BoxHead is the bounding box prediction header, outputting 4D coordinate information; ClsHead is the classification prediction header, outputting a |C|-dimensional class probability distribution, where |C| is the number of object classes; RelHead is the relation prediction header, taking the concatenation of the subject query and object query as input, and outputting a |R|-dimensional relation class probability distribution, where |R| is the number of relation classes; bounding box prediction... and All dimensions are M×4, classification prediction and All dimensions are M×|C|, relation prediction The dimension is M×|R|;

[0045] Step 4-4: For each real relation triplet and each predicted triple Calculate the one-to-many matching score:

[0046]

[0047] in, This represents the probability value corresponding to the true subject category in the predicted subject classification probability. The probability value corresponding to the real object category in the predicted object classification probability is represented by IoU, which is the intersection-union ratio function used to measure the degree of overlap between the predicted bounding box and the real bounding box. This product score comprehensively considers the accuracy of subject classification, object classification, subject localization, and object localization, ensuring that the matched predictions are of high quality in both category and localization, thereby reducing the false match rate and improving the quality of positive samples.

[0048] Steps 4-5: Given a threshold T=0.4 and a maximum number of matching queries K=6 for each real target, select queries that meet the following conditions as positive samples for each real relation triplet: First, filter by matching score. All queries; among the queries that meet the criteria, select the top K queries with the highest scores; these selected queries form the positive sample index set. The remaining queries constitute the negative sample index set. Through this one-to-many allocation strategy, each real relation can be matched with up to K high-quality queries. Compared with the traditional one-to-one allocation strategy, this significantly increases the number of positive samples, effectively alleviates the sparse supervision problem, and enables the model to obtain sufficient supervision signals even when each image has only a few annotations.

[0049] Steps 4-6: Based on the positive sample index set and negative sample index set The three loss components of the one-to-many stage are calculated, including relation classification loss, subject-object bounding box regression loss, and generalized IoU loss. The three loss components are then weighted and summed according to the set weights to obtain the total loss of the one-to-many stage.

[0050] As a preferred embodiment, the detailed steps of the one-to-one relationship refinement and deduplication module in step 5 of the present invention are as follows:

[0051] Step 5-1: Use the relationship query updated after T1 iterations in the one-to-many coarse positioning module as the initial query for the one-to-one fine deduplication module. The dimension is M×512; before entering the multi-head self-attention deduplication process, the initial query is first refined using the encoder feature F; the initial query is decomposed into the main query. and object query :

[0052]

[0053] Refine subject and object queries through cross-attention and layer normalization:

[0054]

[0055]

[0056] The refined subject and object queries further integrate visual information from the encoder features, providing a more accurate query representation for subsequent deduplication processing;

[0057] Step 5-2: Through The multi-head self-attention module in each iteration performs deduplication. Each iteration includes self-attention operations, layer normalization, and a multilayer perceptron.

[0058]

[0059] Among them, multi-head self-attention block The definition of is:

[0060]

[0061] Let represent the self-attention operation in the t-th iteration. Through the self-attention mechanism, each query can perceive information from all other queries. During the self-attention computation, multiple queries predicting the same or similar relationships will receive higher attention weights. This mechanism allows these queries to perceive each other and compete. In each iteration, the model gradually adjusts the feature representation of the query, so that the query that best matches the real relationship gets a higher classification confidence, while the confidence of other redundant queries is gradually suppressed.

[0062] Step 5-3: After passing through The main query obtained after the second iteration and object query By concatenating along dimension 1, a refined relational query is generated. :

[0063]

[0064] The refined relation query dimension is M×512. After multiple self-attention iterations, the semantic boundaries between different queries are clearer, and redundant information is effectively suppressed.

[0065] Step 5-4: Decompose the refined relationship query into subject query and object query, and generate the final bounding box prediction, classification prediction, and relationship prediction through independent prediction heads:

[0066]

[0067]

[0068]

[0069] These final predictions will be used for one-to-one matching with the ground truth annotations;

[0070] Step 5-5: The Hungarian algorithm is used for one-to-one optimal bipartite graph matching. This algorithm assigns a unique optimal query to each real relation by minimizing the overall matching cost. First, the matching cost between each predicted triple and each real triple is calculated, taking into account classification loss, bounding box regression loss, and IoU loss. Then, the Hungarian algorithm is applied to solve for the optimal allocation scheme σ*, which minimizes the total matching cost. This one-to-one matching strategy ensures that each real relation corresponds to only one prediction, effectively eliminating duplicate predictions generated in the one-to-many stage and guaranteeing the uniqueness and accuracy of the final prediction result.

[0071] Steps 5-6: Based on the optimal allocation scheme σ* obtained by the Hungarian algorithm, calculate the three loss components of the one-to-one stage, including relation classification loss, subject-object bounding box regression loss, and generalized IoU loss, and then sum these three loss components according to the set weights to obtain the total loss of the one-to-one stage.

[0072] Preferably, the loss function for a one-to-many stage in step 4 of this invention includes the following three components:

[0073] Relationship classification loss The classification loss is calculated separately for positive and negative samples using the cross-entropy loss function. For positive samples, the cross-entropy between the predicted relation probability and the true relation class is calculated; for negative samples, the cross-entropy between the predicted relation probability and the empty class (background class) is calculated.

[0074]

[0075] Where is the set of positive sample indices. Let CE be the set of negative sample indices, and let CE represent the cross-entropy loss function. This represents the probability distribution of relation prediction for the i-th query. The label represents the category of the i-th real relationship, and ∅ represents the background category;

[0076] Bounding box regression loss The L1 loss function is used to calculate the coordinate difference between the predicted bounding box and the ground truth bounding box, including the subject bounding box and the object bounding box.

[0077]

[0078] Where L1 represents the L1 norm loss function, and Let represent the predicted bounding box and the ground truth bounding box of the i-th query, respectively. and Let represent the predicted bounding box and the true bounding box of the object for the i-th query, respectively; L1 loss calculates the absolute difference of the four dimensions of the bounding box coordinates (top-left x-coordinate, top-left y-coordinate, width, and height) and sums them;

[0079] Generalized IoU loss The generalized IoU loss function is used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box. Generalized IoU considers not only the overlapping area of ​​the bounding boxes, but also the influence of non-overlapping regions.

[0080]

[0081] in, This represents the generalized IoU loss function, which is calculated by subtracting the generalized IoU value from 1. The generalized IoU is defined as IoU minus the proportion of the area not covered by the two bounding boxes in the smallest bounding rectangle containing the two bounding boxes.

[0082] Total loss in one-to-many phase The result is obtained by weighted summation of the three loss components mentioned above:

[0083]

[0084] in, , , These are the weighting coefficients for the three loss components. By adjusting these weights, the importance of different loss components can be balanced.

[0085] Preferably, the loss function in step 5 of this invention for the one-to-one stage includes the following three components:

[0086] Relationship classification loss Based on the optimal allocation scheme σ* obtained by the Hungarian algorithm, calculate the classification loss between each true relation and its matching query:

[0087]

[0088] Where N represents the number of real relation triples, and σ*(i) represents the query index matched by the Hungarian algorithm for the i-th real relation. This represents the probability distribution of relation predictions for the matching query. Represents the category label of the i-th real relation; since a one-to-one matching strategy is used, each real relation corresponds to only one query, so only the classification loss of positive samples is calculated, and negative samples are not involved.

[0089] Bounding box regression loss Calculate the L1 loss between the predicted bounding box and the ground truth bounding box for the matching query:

[0090]

[0091] in, and These represent the predicted bounding boxes for the subject and object of the matching query, respectively. and Let x represent the subject and object bounding boxes of the i-th real relation, respectively.

[0092] Generalized IoU loss Calculate the generalized IoU loss between the predicted bounding box and the ground truth bounding box of the matching query:

[0093]

[0094] Total loss in one-on-one phase The result is obtained by weighted summation of the three loss components mentioned above:

[0095]

[0096] in, , , These are the weighting coefficients for the three loss components; the one-to-one stage loss function focuses on accurate matching and deduplication to ensure the high quality and uniqueness of the final prediction.

[0097] Preferably, the detailed calculation method of the total loss function in step 6 of the present invention is as follows:

[0098] The total loss function uses a weighted combination to fuse the total loss for one-to-many phases, the total loss for one-to-one phases, and the auxiliary loss:

[0099]

[0100] Among them, the total loss in the one-to-many stage The following results were obtained using weighted combination relationship classification loss, bounding box regression loss, and generalized IoU loss:

[0101]

[0102] Total loss in one-on-one phase Similarly, by weighting and combining the three loss components, we obtain:

[0103]

[0104] Auxiliary loss Used to supervise the learning of the classification head MLP in the adaptive semantic relation query generation module, this loss is calculated as the cross-entropy loss between the encoder token's prediction after passing through the classification head and the real object category, helping the model learn to select the most valuable token;

[0105] Based on experimental verification and experience from references, the loss weights are configured as follows: weights for one-to-many stages. , , Weights in one-to-one phases , , The auxiliary loss weight is 0.5; the bounding box regression loss has a higher weight because accurate localization is crucial for scene graph generation.

[0106] A gradient blocking mechanism is employed during training. Specifically, the gradient is truncated before the output of the one-to-many stage is passed to the one-to-one stage, preventing the gradient of the one-to-one stage from being backpropagated to the one-to-many stage. This mechanism ensures that the training of the two stages is conducted independently, avoiding the recurrent oppositional interaction problem. Recurrent oppositional interaction refers to the situation where, when two modules use the same decoder weights but have conflicting optimization objectives, the gradient update of one module weakens the learning effect of the other module, leading to training instability and performance degradation. Through the gradient blocking mechanism, the one-to-many stage can focus on increasing positive samples and coarse localization, while the one-to-one stage can focus on deduplication and precise matching, with both working together without interfering with each other.

[0107] To test the output of the trained model in the scene graph generation method based on dynamic query allocation and decoupled attention mechanism, the testing process of this invention includes the following steps:

[0108] Step 1: Input the test image into the trained scene graph generation model. The test image is first processed by the backbone network and Transformer encoder to generate encoder token features, and then proceed to Step 2.

[0109] Step 2: The encoder token features are generated by the adaptive semantic relation query generation module. The trained classification head MLP adaptively selects the M most valuable tokens and generates relation queries through a bidirectional cross-attention mechanism. These relation queries contain rich semantic and spatial information. Proceed to Step 3.

[0110] Step 3: The relation query is input into the trained relation query attention decoupling decoder. First, it goes through the one-to-many relation coarse localization module to perform preliminary relation prediction using cross attention. Then, it goes through the one-to-one relation fine deduplication module to perform deduplication using multi-head self attention. Finally, it outputs the fine relation triple prediction results. Each prediction result includes the bounding box and category of the subject, the bounding box and category of the object, and the relationship category between them. Proceed to Step 4.

[0111] Step 4: Compare the predicted relation triples of the model with the ground truth annotations in the test set. For each predicted subject-relation-object triple, determine whether it matches a ground truth triple. The matching criteria include: the IoU between the predicted bounding box of the subject and the ground truth bounding box is greater than a set threshold (usually 0.5) and the category is correct; the IoU between the predicted bounding box of the object and the ground truth bounding box is greater than a set threshold and the category is correct; and the relation category is correct. Calculate the prediction results and matching status of all test images and proceed to Step 5.

[0112] Step 5: Evaluate the model performance using standard evaluation metrics for scene graph generation tasks. Key metrics include Recall@K and mean recall mR@K, where K is typically 50 or 100. Recall@K represents the proportion of correctly predicted true relations out of all true relations predicted in the first K predicted relations, reflecting the model's overall recall capability. mR@K represents the average of Recall@K calculated for each predicate category; this metric focuses more on the model's ability to predict rare relation categories and better reflects whether the model is biased towards common relation categories. Additionally, Recall@K can be calculated under different constraint levels, including three task settings: predicate classification (PredCls), scene graph classification (SGCls), and scene graph detection (SGDet), corresponding to different input conditions and difficulty levels. Proceed to Step 6.

[0113] Step 6: Output the test results, including the values ​​of each evaluation indicator, the distribution of prediction accuracy for different relationship categories, the inference speed of the model, and other information. End.

[0114] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the scene graph generation method based on dynamic query allocation and decoupled attention mechanism of the present invention.

[0115] According to another aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the scene graph generation method based on dynamic query allocation and decoupled attention mechanism of the present invention.

[0116] Compared with existing technologies, the beneficial effects of the above-described method of the present invention are as follows:

[0117] This invention selects high-quality tokens from the encoder tokens to construct relation queries through an adaptive semantic relation query generation module, effectively improving the semantic expressiveness and spatial positioning accuracy of the queries and reducing invalid queries at the source. By using a relation query attention decoupling decoder, the coarse positioning of one-to-many relations and the fine deduplication of one-to-one relations are functionally decoupled at the attention mechanism level. Cross-attention is used to maximize the use of supervision signals, self-attention is used to eliminate prediction ambiguity, and gradient blocking mechanism is used to ensure independent optimization in two stages, effectively avoiding optimization target conflict and cyclic adversarial interaction problems. This invention systematically solves the key problems of supervision sparsity, low query quality and query ambiguity in existing methods, and significantly improves the training efficiency, convergence speed and relation prediction accuracy of the model in sparse labeling scenarios. Attached Figure Description

[0118] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention.

[0119] Figure 1 This is a schematic diagram of the overall network framework of a preferred embodiment of the present invention;

[0120] Figure 2 This is a schematic diagram of an adaptive semantic query generation module according to a preferred embodiment of the present invention;

[0121] Figure 3 This is a flowchart illustrating the overall training process of a preferred embodiment of the present invention. Detailed Implementation

[0122] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention.

[0123] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0124] Example 1:

[0125] like Figure 1-3 As shown, this invention provides a scene graph generation method based on dynamic query allocation and decoupled attention mechanism, including the following steps:

[0126] This embodiment utilizes the Visual Genome 150 (VG150) dataset to complete a scene graph generation task based on dynamic query allocation and a decoupled attention mechanism. This dataset contains 108,077 images, covering the 150 most common instance categories and 50 predicate categories. To reasonably divide the dataset, 70% of the images are allocated to the training set, and the remaining 30% is used as the test set. For training and testing, Ubuntu 20.04 was chosen as the operating system, and the research was conducted in an experimental environment of PyTorch 1.10.0 + CUDA 11.3.

[0127] The model of this invention uses ResNet-50 as the backbone network to extract image features. The Transformer encoder has 6 layers, each containing 8 attention heads, and the hidden layer dimension is 256. In the adaptive semantic relation query generation module, the number of most valuable tokens M is set to 100, and the classification head MLP contains 3 fully connected layers. In the relation query attention decoupling decoder, the number of iterations T1 for the one-to-many relation coarse localization module is set to 6, and the number of iterations T2 for the one-to-one relation fine deduplication module is set to 6.

[0128] Regarding the weight configuration of the loss function, the weights in a one-to-many stage... , , Weights in one-to-one phases , , The auxiliary loss weight is set to 0.5. The threshold T in the one-to-many allocation strategy is set to 0.4, and the maximum number of matching queries K for each real relationship is set to 6.

[0129] This invention uses the AdamW optimizer for training, with an initial learning rate of 1×10⁻⁴, a weight decay factor of 1×10⁻⁴, and a batch size of 16. The model is trained for a total of 10 epochs, with the learning rate decaying to 0.1 times its original value in the 8th epoch. A gradient blocking mechanism is employed to ensure independent optimization in both the one-to-many and one-to-one phases, avoiding cyclical adversarial interactions. All experiments are performed on hardware equipped with an NVIDIA GeForce RTX 4090 GPU (24GB VRAM) to ensure high efficiency in both training and testing.

[0130] This invention is comprehensively evaluated on the Scene Graph Detection (SGDet) task on the VG150 dataset and compared with existing state-of-the-art methods. Evaluation metrics include Recall@K (R@K) and mean Recall@K (mR@K), where K takes values ​​of 20, 50, and 100, respectively. Experimental results are shown in Table 1.

[0131] Table 1. Comparison results of the method of the present invention and existing methods on the VG150 dataset.

[0132] Model Backbone Epoch R@20 R@50 R@100 mR@20 mR@50 mR@100 DSGG - 60 - 32.9 38.5 - 13.0 17.3 SpeaQ ResNet101 52 - 32.9 36.0 - 11.8 14.1 EGTR ResNet50 275+ 23.5 30.2 34.3 5.5 7.9 10.1 Hydra-SGG ResNet50 12 21.9 28.6 33.4 10.3 15.9 19.4 This invention ResNet50 10 25.6 33.2 39.8 15.3 18.2 21.6

[0133] As can be seen from the experimental results in Table 1, the method proposed in this invention achieves significant performance improvements across all evaluation metrics:

[0134] (1) Compared with the Hydra-SGG method (12 epochs) which has a similar number of training epochs, this invention only requires 10 epochs, achieving improvements of 3.7, 4.6, and 6.4 percentage points in R@20, R@50, and R@100, respectively, representing improvements of 16.9%, 16.1%, and 19.2%. It also achieves improvements of 5.0, 2.3, and 2.2 percentage points in mR@20, mR@50, and mR@100, respectively, representing improvements of 48.5%, 14.5%, and 11.3%. This fully demonstrates that the dynamic query allocation strategy and decoupled attention mechanism of this invention can significantly improve model performance.

[0135] (2) Compared with the DSGG method, which requires 60 epochs of training, this invention only requires 10 epochs to achieve an R@50 score of 33.2%, exceeding DSGG's 32.9%; and an R@100 score of 39.8%, exceeding DSGG's 38.5%, while improving training efficiency by 6 times. More importantly, this invention achieves mR@50 and mR@100 scores of 18.2% and 21.6%, respectively, which are 5.2 and 4.3 percentage points higher than DSGG's 13.0% and 17.3%, respectively, representing improvements of 40.0% and 24.9%. This indicates that the method of this invention has a significant advantage in predicting rare relation categories.

[0136] (3) Compared with the EGTR method, which requires 275+ epochs of training, the present invention has an overwhelming advantage in training efficiency, requiring only about 1 / 27 of the training epochs, and significantly outperforming in all metrics. In particular, the improvement of the present invention is particularly significant in the mR@K metric, with mR@100 reaching 21.6%, while EGTR is only 10.1%, representing an improvement of 113.9%, fully demonstrating the superior ability of the present invention in rare relation prediction.

[0137] (4) Compared with the SpeaQ method, which uses ResNet101 as the backbone network and requires 52 epochs of training, this invention uses a lighter ResNet50 backbone network and requires fewer training epochs (only 10 epochs), but still improves the R@100 index by 3.8 percentage points and significantly outperforms the mR@K index, fully demonstrating the efficiency and effectiveness of the method of this invention.

[0138] The experimental results fully validate the superiority of the method described in this invention. Through the adaptive semantic relation query generation module, this invention improves query quality from the source, reduces invalid queries, and increases the positive sample hit rate. Through the relation query attention decoupling decoder, this invention functionally decouples the coarse localization of one-to-many relations from the fine deduplication of one-to-one relations at the attention mechanism level, effectively avoiding optimization target conflicts and cyclical adversarial interactions. Through the gradient blocking mechanism, this invention ensures independent optimization in both stages, significantly improving training efficiency and model performance. In particular, the significant improvement in the mR@K metric demonstrates the effectiveness of this invention in solving long-tail distribution problems and predicting rare relation categories, which is of great significance for understanding complex relationships in practical applications.

[0139] The model proposed in this invention uses the following two metrics: Recall@K (R@K) and mean Recall@K (mR@K), where K is typically 50 or 100. R@K represents the proportion of correctly predicted true relations out of all true relations predicted in the first K predicted relations, reflecting the overall recall capability of the model; mR@K represents the average of R@K calculated for each predicate category, focusing more on the model's predictive ability for rare relation categories.

[0140] To verify the beneficial effects of this invention, experiments were conducted on the Visual Genome 150 (VG150) dataset, which contains the 150 most common instance classes and 50 predicate classes. In the scene graph detection task, the method of this invention achieved the following evaluation results: R@50 was 33.2%, R@100 was 39.8%, mR@50 was 17.2%, and mR@100 was 21.6%. Compared to the existing Hydra-SGG method, this invention improves the mR@50 metric by 2.3% and increases the training convergence speed by approximately 30%, fully demonstrating the significant advantages of this invention in improving relation prediction accuracy and training efficiency.

[0141] Example 2:

[0142] This embodiment will introduce one applicable scenario of the present invention.

[0143] In smart home systems, users want to interact with intelligent assistants using natural language, such as asking questions like "What are the people in the living room doing?" or "What's on the kitchen table?" These visual question-answering tasks require the system to accurately understand the scene structure and object relationships in an image. However, images in a home environment often contain complex backgrounds, diverse objects, and rich spatial relationships. Traditional scene graph generation methods, due to sparse supervision signals and low query quality, struggle to accurately predict these relationships, leading to the intelligent assistant's inability to provide accurate answers. This invention proposes a scene graph generation method based on dynamic query allocation and a decoupled attention mechanism, which effectively addresses this challenge.

[0144] First, the smart home system acquires real-time images through indoor cameras and performs preprocessing, including basic image processing steps such as image resizing and illumination correction, to optimize image quality. Then, the Faster R-CNN model is used to extract features from these preprocessed images, identifying and locating individual object instances within the images.

[0145] Next, the proposed method generates a detailed and accurate scene graph. This method uses an adaptive semantic relation query generation module to automatically select the most valuable, high-quality tokens from the encoder tokens to construct relation queries, resulting in richer semantic information and more accurate spatial localization compared to randomly initialized queries. Then, through a relation query attention decoupling decoder, a one-to-many relation coarse localization module is used to increase the number of positive samples and quickly learn relation features. Finally, a one-to-one relation fine deduplication module eliminates duplicate predictions and obtains accurate relation prediction results. Even in complex home environments with diverse object relationships, the model can accurately identify important relationships such as "person-sitting-sofa" and "book-placed-on-table."

[0146] The generated scene graph is then used in the semantic understanding module of the intelligent assistant. The system can quickly retrieve relevant information based on the accurate scene graph and generate accurate answers to the user's questions. For example, when a user asks "What are the people in the living room doing?", the system can answer "The people in the living room are sitting on the sofa and watching TV" based on the relationship triples "person-sitting-sofa" and "person-watching-TV" in the scene graph.

[0147] Finally, the system stores the generated scene graph and analysis results in a local database for continuous learning and optimization of the user experience. Through the application of this invention, smart home systems can more accurately understand the relationships between objects in a home environment, providing more intelligent and personalized services.

[0148] Example 3:

[0149] The computer-readable storage medium of this embodiment stores a computer program that, when executed by a processor, implements the steps in the scene graph generation method based on dynamic query allocation and decoupled attention mechanism of embodiment 1 or 2.

[0150] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0151] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0152] Example 4:

[0153] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the scene graph generation method based on dynamic query allocation and decoupled attention mechanism of embodiment 1 or 2.

[0154] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0155] Those skilled in the art will understand that the content disclosed in the embodiments can be provided as a method, system, or computer program product. Therefore, this solution can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this solution can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0156] This solution is described with reference to flowchart illustrations and / or block diagrams of methods and computer program products according to embodiments of this solution. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0160] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.

[0161] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. A method for generating scene graphs based on dynamic query allocation and decoupled attention mechanism, characterized in that, Includes the following steps: Step 1: Input the training images into the backbone network and Transformer encoder to generate encoder token features containing rich semantic information, then proceed to Step 2; Step 2: Using the adaptive semantic relation query generation module, select the M most valuable tokens from the encoder tokens and generate relation queries based on the bidirectional cross-attention mechanism, then proceed to Step 3; Step 3: Input the relation query into the relation query attention decoupling decoder. This decoder consists of two stages: a one-to-many relation coarse localization module and a one-to-one relation fine deduplication module. Proceed to Step 4. Step 4: In the one-to-many relationship coarse localization module, the relationship query is updated using the cross-attention mechanism, the bounding boxes and classification results of the subject and object and the relationship category are predicted, the one-to-many label allocation strategy is adopted to increase the number of positive samples, the loss of the one-to-many stage is calculated, and then proceed to step 5. Step 5: In the one-to-one relationship refinement and deduplication module, the multi-head self-attention mechanism is used to deduplicate the relationship query, eliminate duplicate predictions, use the Hungarian algorithm for one-to-one matching, calculate the loss of the one-to-one stage, and proceed to step 6. Step 6: Weight the one-to-many stage loss and the one-to-one stage loss to obtain the total loss, and use it to train scene graph generation. Proceed to Step 7. Step 7: If the set batch processing size is reached, return to Step 1; if all training images have been read, proceed to Step 8. Step 8: Output the trained model and end.

2. The method as described in claim 1, characterized in that, In step 1, the backbone network adopts a convolutional neural network. Each layer of the Transformer encoder contains a multi-head self-attention mechanism and a feedforward neural network, which are connected through residual connections and layer normalization, so that each encoder token can capture global context information and local spatial information.

3. The method as described in claim 2, characterized in that, The specific steps of the adaptive semantic relation query generation module in step 2 are as follows: Step 2-1: Input all encoder token features F into the classification head MLP. The MLP contains multiple fully connected layers and activation functions, and outputs the classification score of each token. Calculate the threshold τ based on the classification score, and select the M tokens with the highest classification scores as the most valuable object representation tokens. These tokens have richer semantic information and more accurate spatial positioning capabilities compared to randomly initialized queries. The dimension of each token is 1×256. Step 2-2: Copy the M selected tokens into two copies, using the first copy as the main query. The second document was used as the subject of the inquiry. Both have a dimension of M×256, which are used to represent the subject entity and the object entity in the scene, respectively; Steps 2-3: To capture the bidirectional dependency between the subject and the object, calculate the cross-attention weights from the subject to the object. Cross-attention weights from object to subject : ； ； Wherein, CA represents cross-attention calculation, which includes linear transformations of queries, keys, and values, as well as the calculation of attention weights and weighted summation operations; the subject-to-object attention weight dimension is M×256, reflecting the degree of attention each subject query pays to all object queries; the object-to-subject attention weight dimension is also M×256, reflecting the degree of attention each object query pays to all subject queries; Steps 2-4: Based on the calculated cross-attention weights, enhance the original subject query and object query to include each other's contextual information: ； ； The enhanced subject query not only retains the original subject information, but also incorporates the contextual information of the object; the enhanced object query also incorporates the contextual information of the subject. Steps 2-5: Enhance the subject query and object query Perform a concatenation operation along dimension 1 to generate the final relation query Qrel: ； The relation query has a dimension of M×512, which includes subject information, object information, and interaction information between the two.

4. The method as described in claim 3, characterized in that, The specific steps for coarse positioning of one-to-many relationships in step 4 are as follows: Step 4-1: Decompose the relation query Qrel with dimension M×512 into subject query and object query along the channel dimension: ； The decomposed subject query and object query dimensions are both M×256; Step 4-2: Update the subject query and object query separately using the cross-attention mechanism and multilayer perceptron, so that the subject query and object query can fully interact with the encoder features F: ； ； Where LN represents layer normalization, MLP represents multilayer perceptron, and CA represents cross-attention; the cross-attention operation allows the query to extract relevant visual information from the encoder features; layer normalization and residual connections ensure the stability of training; the multilayer perceptron further performs nonlinear transformations on the features; the updated subject query and object query are denoted as... and ; Step 4-3: Query the updated subject and object query The data is input into separate prediction heads to generate bounding box predictions for subjects and objects, classification predictions, and relationship predictions. ； ； ； Among them, BoxHead is the bounding box prediction header, outputting 4D coordinate information; ClsHead is the classification prediction header, outputting a |C|-dimensional class probability distribution, where |C| is the number of object classes; RelHead is the relation prediction header, taking the concatenation of the subject query and object query as input, and outputting a |R|-dimensional relation class probability distribution, where |R| is the number of relation classes; bounding box prediction... and All dimensions are M×4, classification prediction and All dimensions are M×|C|, relation prediction The dimension is M×|R|; Step 4-4: For each real relation triplet and each predicted triple Calculate the one-to-many matching score: ； in, This represents the probability value corresponding to the true subject category in the predicted subject classification probability. The probability value corresponding to the real object category in the predicted object classification probability is represented by IoU, which is the intersection-union ratio function used to measure the degree of overlap between the predicted bounding box and the real bounding box. This product score comprehensively considers the accuracy of subject classification, object classification, subject localization, and object localization, ensuring that the matched predictions are of high quality in both category and localization, thereby reducing the false match rate and improving the quality of positive samples. Steps 4-5: Given a threshold T=0.4 and a maximum number of matching queries K=6 for each real target, select queries that meet the following conditions as positive samples for each real relation triplet: First, filter by matching score. All queries; among the queries that meet the criteria, select the top K queries with the highest scores; these selected queries form the positive sample index set. The remaining queries constitute the negative sample index set. Through this one-to-many allocation strategy, each real relation can be matched with up to K high-quality queries. Compared with the traditional one-to-one allocation strategy, this significantly increases the number of positive samples, effectively alleviates the sparse supervision problem, and enables the model to obtain sufficient supervision signals even when each image has only a few annotations. Steps 4-6: Based on the positive sample index set and negative sample index set The three loss components of the one-to-many stage are calculated, including relation classification loss, subject-object bounding box regression loss, and generalized IoU loss. The three loss components are then weighted and summed according to the set weights to obtain the total loss of the one-to-many stage.

5. The method as described in claim 4, characterized in that, The loss function for the one-to-many stage in step 4 includes the following three components: Relationship classification loss The classification loss is calculated separately for positive and negative samples using the cross-entropy loss function. For positive samples, the cross-entropy between the predicted relation probability and the true relation category is calculated; for negative samples, the cross-entropy between the predicted relation probability and the empty category is calculated. ； Where is the set of positive sample indices. Let CE be the set of negative sample indices, and let CE represent the cross-entropy loss function. This represents the probability distribution of relation prediction for the i-th query. The label represents the category of the i-th real relationship, and ∅ represents the background category; Bounding box regression loss The L1 loss function is used to calculate the coordinate difference between the predicted bounding box and the ground truth bounding box, including the subject bounding box and the object bounding box. ； Where L1 represents the L1 norm loss function, and Let represent the predicted bounding box and the ground truth bounding box of the i-th query, respectively. and Let represent the predicted bounding box and the true bounding box of the object for the i-th query, respectively; L1 loss calculates the absolute difference of the bounding box coordinates in each of the four dimensions and sums them. Generalized IoU loss The generalized IoU loss function is used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box. Generalized IoU considers not only the overlapping area of ​​the bounding boxes, but also the influence of non-overlapping regions. ； in, This represents the generalized IoU loss function, which is calculated by subtracting the generalized IoU value from 1. The generalized IoU is defined as IoU minus the proportion of the area not covered by the two bounding boxes in the smallest bounding rectangle containing the two bounding boxes. Total loss in one-to-many phase The result is obtained by weighted summation of the three loss components mentioned above: ； in, , , These are the weighting coefficients for the three loss components. By adjusting these weights, the importance of different loss components can be balanced.

6. The method as described in claim 5, characterized in that, The specific steps of the one-to-one relationship fine-grained deduplication module in step 5 are as follows: Step 5-1: Use the relationship query updated after T1 iterations in the one-to-many coarse positioning module as the initial query for the one-to-one fine deduplication module. The dimension is M×512; before entering the multi-head self-attention deduplication process, the initial query is first refined using the encoder feature F; the initial query is decomposed into the main query. and object query : ； Refine subject and object queries through cross-attention and layer normalization: ； ； The refined subject and object queries further integrate visual information from the encoder features, providing a more accurate query representation for subsequent deduplication processing; Step 5-2: Through The multi-head self-attention module in each iteration performs deduplication. Each iteration includes self-attention operations, layer normalization, and a multilayer perceptron. ； Among them, multi-head self-attention block The definition of is: ； Let represent the self-attention operation in the t-th iteration. Through the self-attention mechanism, each query can perceive information from all other queries. During the self-attention computation, multiple queries predicting the same or similar relationships will receive higher attention weights. This mechanism allows these queries to perceive each other and compete. In each iteration, the model gradually adjusts the feature representation of the query, so that the query that best matches the real relationship gets a higher classification confidence, while the confidence of other redundant queries is gradually suppressed. Step 5-3: After passing through The main query obtained after the second iteration and object query By concatenating along dimension 1, a refined relational query is generated. : ； The refined relation query dimension is M×512. After multiple self-attention iterations, the semantic boundaries between different queries are clearer, and redundant information is effectively suppressed. Step 5-4: Decompose the refined relationship query into subject query and object query, and generate the final bounding box prediction, classification prediction, and relationship prediction through independent prediction heads: ； ； ； The final predictions will be used for one-to-one matching with the ground truth annotations; Step 5-5: The Hungarian algorithm is used for one-to-one optimal bipartite graph matching. By minimizing the overall matching cost, a unique optimal query is assigned to each real relation. First, the matching cost between each predicted triple and each real triple is calculated. The matching cost comprehensively considers classification loss, bounding box regression loss, and IoU loss. Then, the Hungarian algorithm is applied to solve for the optimal allocation scheme σ*, which minimizes the total matching cost. This one-to-one matching strategy ensures that each real relation corresponds to only one prediction. Steps 5-6: Based on the optimal allocation scheme σ* obtained by the Hungarian algorithm, calculate the three loss components of the one-to-one stage, including relation classification loss, subject-object bounding box regression loss, and generalized IoU loss, and then sum these three loss components according to the set weights to obtain the total loss of the one-to-one stage.

7. The method as described in claim 6, characterized in that, The total loss function in step 6 is calculated as follows: The total loss function uses a weighted combination to fuse the total loss for one-to-many phases, the total loss for one-to-one phases, and the auxiliary loss: ； Among them, the total loss in the one-to-many stage The following results were obtained using weighted combination relationship classification loss, bounding box regression loss, and generalized IoU loss: ； Total loss in one-on-one phase Similarly, by weighting and combining the three loss components, we obtain: ； Auxiliary loss Used to supervise the learning of the classification head MLP in the adaptive semantic relation query generation module, this loss is calculated as the cross-entropy loss between the encoder token's prediction after passing through the classification head and the real object category, helping the model learn to select the most valuable token; Based on experimental verification and experience from references, the loss weights are configured as follows: weights for one-to-many stages. , , Weights in one-to-one phases , , The auxiliary loss weight is 0.5; the bounding box regression loss has a higher weight because accurate localization is crucial for scene graph generation. The gradient blocking mechanism is used during training, which is specifically implemented by truncating the gradient before the output of the one-to-many stage is passed to the one-to-one stage.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the steps in the scene graph generation method based on dynamic query allocation and decoupled attention mechanism as described in any one of claims 1 to 7.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the scene graph generation method based on dynamic query allocation and decoupled attention mechanism as described in any one of claims 1 to 7.

Images