A Scene Graph Generation Method Based on a Diffusion Model

Abstract

This invention relates to the field of computer vision processing technology, and provides a scene graph generation method based on a diffusion model. The method includes: acquiring training data containing labeled information to obtain entity candidate boxes and relation candidate boxes; adding noise to obtain entity noise candidate boxes and relation noise candidate boxes; extracting features from the image to be processed using the entity noise candidate boxes and relation noise candidate boxes to obtain entity features and relation features; constructing a deep learning network based on the entity features and relation features and learning the anti-diffusion process of entity detection and relation detection to obtain a diffusion model; obtaining entity location boxes and relation location boxes of the image to be processed using the diffusion model, calculating the intersection-union ratio (IUR), and matching according to the highest IUR to obtain relation triples; and generating a scene graph based on the relation triples and graph structure constraints. This method can utilize a diffusion model to complete a flexible and scalable end-to-end scene graph generation task.

Description

Technical Field

[0001] This invention relates to the field of computer vision processing technology, and in particular to a method for generating scene graphs based on a diffusion model. Background Technology

[0002] With the advancement of the internet age, human production and daily life have generated and recorded massive amounts of data, including a vast amount of image data. Under the same data volume, images can convey far more information than simple text. A single image may contain dozens of target entities and numerous relationships between them. In visual understanding tasks, these relationships can be modeled as several "subject-verb-object" triples. Based on these modeled relational triples, the image scene can be further organized into a graph structure, namely a scene graph. In the scene graph, nodes and edges represent target instances and the relationships between pairs of objects in the image, respectively. Scene graph generation tasks use images as input to generate scene graphs, providing understanding of the structured semantics of images. Therefore, they exhibit stronger capabilities in visual reasoning compared to other visual understanding tasks, making scene graphs widely applicable in image retrieval, visual question answering, image generation, and editing.

[0003] In addition to its existing applications in image semantic understanding, diffusion models have achieved remarkable results in image segmentation and object detection. However, research and application of diffusion models in scene graph generation are still lacking. The biggest challenge lies in the difficulty of adapting traditional relation modeling architectures to the probabilistic sampling and progressive denoising optimization mechanisms of diffusion models. The flexibility, scalability, and ease of training of diffusion models, however, hold great promise for applications in scene graph generation.

[0004] Identifying relationships between target entities in an image is a crucial task in scene graph generation, as this is central to deep semantic understanding of the image. Most existing scene graph generation work categorizes relationship modeling into two main types: graph structure-based modeling and pairwise query-based modeling. Graph structure-based relationship modeling primarily involves two stages. First, a pre-trained object detection model is used to detect target entities in the input image, obtaining a candidate set of entities and their corresponding features. Then, this candidate set is used as the point set of a directed acyclic graph (DAG), with the edges between pairs of entities forming the initial graph structure for modeling image semantics. Finally, based on this graph structure modeling, the entity and relationship context is modeled, and entity and relationship categories are further classified to predict relationship triples.

[0005] However, this graph-based relation modeling heavily relies on the performance of the object detector, and graph-based contextual feature learning introduces contextual noise and significant time complexity. Recent research has proposed a single-stage approach to scene graph generation. These methods treat relation modeling as a sparse triple query task and train in an end-to-end manner, thus reducing time costs and dependence on the object detector. However, existing single-stage methods suffer from poor performance due to the lack of explicit modeling of target entities and lack flexibility due to their highly coupled model structure. Summary of the Invention

[0006] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a scene graph generation method based on a diffusion model.

[0007] This invention provides a scene graph generation method based on a diffusion model, comprising:

[0008] S100: Obtain training data containing annotation information, obtain entity candidate boxes and relationship candidate boxes based on the training data, add noise to the entity candidate boxes and relationship candidate boxes, and obtain entity noise candidate boxes and relationship noise candidate boxes;

[0009] S200: Extract features from the image to be processed using the entity noise candidate box and the relation noise candidate box to obtain entity features and relation features;

[0010] S300: Construct a deep learning network based on the entity features and the relationship features, and learn the anti-diffusion process of entity detection and relationship detection through the deep learning network to obtain a diffusion model;

[0011] S400: Obtain the entity location bounding box and relation location bounding box of the image to be processed through the diffusion model, calculate the intersection-union ratio of the entity location bounding box and the relation location bounding box, and match according to the highest intersection-union ratio to obtain relation triples;

[0012] S500: Generate a scene graph based on the constraints of the relation triplet combined graph structure.

[0013] According to the present invention, a scene graph generation method based on a diffusion model is provided, wherein the noise is random noise that conforms to a Gaussian distribution.

[0014] According to the scene graph generation method based on a diffusion model provided by the present invention, step S100 includes:

[0015] S110: Introduce a scene graph generation dataset, and select training data from the scene graph generation dataset;

[0016] S120: Extract entity candidate boxes and relation candidate boxes from the training data;

[0017] S130: Transform the entity candidate box and the relationship candidate box from the representation space of the upper left and lower right coordinates to the representation space of the center coordinate size to obtain the entity candidate box and the relationship candidate box in the center coordinate size space;

[0018] S140: Add noise to the entity candidate boxes and relation candidate boxes in the center coordinate size space to obtain pre-entity noise candidate boxes and pre-relation noise candidate boxes;

[0019] S150: Transform the pre-entity noise candidate box and the pre-relationship noise candidate box from the representation space of the center coordinate to the representation space of the upper left and lower right coordinates to obtain entity noise candidate boxes and relation noise candidate boxes.

[0020] According to the scene graph generation method based on a diffusion model provided by the present invention, the noise addition process in step S140 is represented as follows:

[0021]

[0022] in, For entity noise candidate boxes, Candidate boxes for relational noise. Label the truth values ​​for entity candidate boxes. Label the truth values ​​for the relation candidate boxes. For diffusion process, The noise added at time t.

[0023] According to the scene graph generation method based on a diffusion model provided by the present invention, step S300 includes:

[0024] S310: Embed sampling time into the entity features and the relationship features, and correct the entity noise candidate box and the relationship noise candidate box according to the entity features and relationship features after embedding the sampling time;

[0025] S320: Combine the entity features and relation features embedded after the sampling time with a classification network to predict the predicted entity category and the predicted relation category;

[0026] S330: Construct the deep learning network based on the entity features, the relationship features, the predicted entity category, and the predicted relationship category;

[0027] S340: Optimize the deep learning network by predicting the ground truth values ​​of entity categories and relation categories through optimal transmission allocation;

[0028] S350: The optimized deep learning network learns the anti-diffusion process and is trained to obtain the diffusion model.

[0029] According to the scene graph generation method based on a diffusion model provided by the present invention, step S340 includes:

[0030] S341: Deep learning network predicts training images and obtains prediction results;

[0031] S342: Calculate the cross-entropy loss function, mean absolute value error loss function, and generalized cross-union ratio loss function of the prediction results using the optimal transmission allocation method;

[0032] S343: Based on the cross-entropy loss function, the mean absolute value error loss function, and the generalized intersection-union loss function, iterate the deep learning network multiple times to obtain an optimized deep learning network.

[0033] According to the scene graph generation method based on a diffusion model provided by the present invention, step S500 further includes:

[0034] S510: Filter out valid relation triples from the relation triples to generate a scene graph in combination with graph structure constraints.

[0035] According to the scene graph generation method based on a diffusion model provided by the present invention, step S510 includes:

[0036] S511: A relation triplet with a relation predicate between two entities is designated as a valid relation triplet.

[0037] S512: For a relation triplet between two entities with multiple relation predicates, calculate the prediction probability of any relation predicate, and label the two entities and the relation predicate with the highest prediction probability as a valid relation triplet;

[0038] S513: Construct a directed acyclic graph structure based on the effective relation triples to generate the scene graph.

[0039] The present invention provides a scene graph generation method based on a diffusion model, which can effectively perform explicit modeling of entity features to obtain better scene graph generation results. Furthermore, it can effectively utilize the characteristics of progressive sampling and iterative optimization of the diffusion model to achieve flexible and scalable scene graph generation, thereby better completing the end-to-end scene graph generation task.

[0040] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a scene graph generation method based on a diffusion model provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0044] In the description of the embodiments of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0045] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0046] In embodiments of the present invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0047] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0048] The following is combined Figure 1 The embodiments provided by the present invention are described below.

[0049] This invention provides a scene graph generation method based on a diffusion model, comprising:

[0050] S100: Obtain training data containing annotation information, obtain entity candidate boxes and relationship candidate boxes based on the training data, add noise to the entity candidate boxes and relationship candidate boxes, and obtain entity noise candidate boxes and relationship noise candidate boxes;

[0051] The noise is random noise that conforms to a Gaussian distribution.

[0052] Step S100 includes:

[0053] S110: Introduce a scene graph generation dataset, and select training data from the scene graph generation dataset;

[0054] S120: Extract entity candidate boxes and relation candidate boxes from the training data;

[0055] S130: Transform the entity candidate box and the relationship candidate box from the representation space of the upper left and lower right coordinates to the representation space of the center coordinate size to obtain the entity candidate box and the relationship candidate box in the center coordinate size space;

[0056] S140: Add noise to the entity candidate boxes and relation candidate boxes in the center coordinate size space to obtain pre-entity noise candidate boxes and pre-relation noise candidate boxes;

[0057] S150: Transform the pre-entity noise candidate box and the pre-relationship noise candidate box from the representation space of the center coordinate to the representation space of the upper left and lower right coordinates to obtain entity noise candidate boxes and relation noise candidate boxes.

[0058] The noise addition process in step S140 is represented as follows:

[0059]

[0060] in, For entity noise candidate boxes, Candidate boxes for relational noise. Label the truth values ​​for entity candidate boxes. Label the truth values ​​for the relation candidate boxes. For diffusion process, The noise added at time t.

[0061] In some embodiments, training data is first obtained from the publicly available scene graph generation dataset Visual Genome, and real scene images I and corresponding scene icon annotations from this dataset are used as dataset samples. The given scene icon annotations include the location and category information of entities in the image, as well as the relationship category information between entities in the image; that is, the scene graph can be represented as a set of several relation triples.

[0062] Furthermore, ground truth entity candidate boxes and ground truth relation candidate box pairs are obtained from the location bounding box data of the labeled data. The location bounding box representation data in two-point coordinates is transformed into a center coordinate-size representation space, which incorporates the center point coordinates, width, and height of the location bounding box. The value space of the center coordinates and size is scaled to the image size. Between these steps, entity candidate boxes and relation candidate boxes in the center coordinate-size space can be obtained. Then, Gaussian noise is added to them. Then, through the reverse transformation process, the position box representation is transformed from the center coordinate-size representation space to the top-left-bottom-right coordinate representation space to obtain entity noise candidate boxes and relation noise candidate boxes.

[0063] S200: Extract features from the image to be processed using the entity noise candidate box and the relation noise candidate box to obtain entity features and relation features;

[0064] In some embodiments, an entity feature and relation feature extraction module is first constructed. For any entity candidate box and the subject and object candidate boxes of a relation candidate box pair, the corresponding entity features and relation subject and object features are extracted from the feature map generated by the pre-trained backbone model using ROI Pooling. The calculation method is as follows:

[0065]

[0066]

[0067] in, For entity features, To extract features and unify their representation, The backbone network for feature extraction. This refers to the input image. The visual features corresponding to the subject in the relation features, For visual features corresponding to objects in relational features, The visual features are the union regions in the relational features.

[0068] S300: Construct a deep learning network based on the entity features and the relationship features, and learn the anti-diffusion process of entity detection and relationship detection through the deep learning network to obtain a diffusion model;

[0069] Step S300 includes:

[0070] S310: Embed sampling time into the entity features and the relationship features, and correct the entity noise candidate box and the relationship noise candidate box according to the entity features and relationship features after embedding the sampling time;

[0071] In some embodiments, based on the entity features and relationship features obtained in step S200, the position box correction offset of entity and relationship noise candidate boxes can be predicted, and the calculation method is as follows:

[0072]

[0073]

[0074]

[0075]

[0076] in, Correct the offset of the candidate bounding box for entity noise. This is the correction offset for the subject or object candidate box of the relation within the relation noise candidate box. Correcting the offset prediction model for entities. To correct the offset prediction model for the relationship Sampling time, For relational features, These are candidate bounding boxes for entity noise after position correction. These are relation noise candidate boxes after the positions of the subject or object candidate boxes have been corrected. This is a correction operation for candidate boxes with noise in entities. This is a correction operation for candidate boxes with relational noise.

[0077] S320: Combine the entity features and relation features embedded after the sampling time with a classification network to predict the predicted entity category and the predicted relation category;

[0078] In some embodiments, based on the entity features obtained in step S200, the corresponding entity category is predicted. The prediction of the entity category is mainly based on the input entity visual feature representation and the temporal embedding of the current sampling step, and the calculation method is as follows:

[0079]

[0080] in, To predict entity categories, A network for classifying entity categories.

[0081] In some embodiments, based on the relation features obtained in step S200, particularly the subject-object relation features, it is possible to predict the possible predicate categories between the subject and object positions of the corresponding relation. Specifically, for the construction of the relation feature representation, this invention uses the visual feature splicing features of the subject, object, and union region as the relation feature representation, and uses a predicate classification network to classify the relation predicates. The calculation method is as follows:

[0082]

[0083] in, To predict the relationship category, This is a relational predicate classification network that predicts the predicate category of the corresponding relation based on the concatenated relational feature representation and the temporal embedding of the current sampling step.

[0084] S330: Construct the deep learning network based on the entity features, the relationship features, the predicted entity category, and the predicted relationship category;

[0085] S340: Optimize the deep learning network by predicting the ground truth values ​​of entity categories and relation categories through optimal transmission allocation;

[0086] S350: The optimized deep learning network learns the anti-diffusion process and is trained to obtain the diffusion model.

[0087] In some embodiments, step S340 further includes assigning matching location box ground values ​​to the corresponding entities and relations. In addition, for the entity noise candidate boxes, relation noise candidate boxes, and relation predicate categories in the predicted entity category and predicted relation category obtained in the above steps, the present invention assigns the most matching ground value label to the prediction results based on the optimal transfer assignment method, thereby calculating the loss function for the entity and relation detection results.

[0088] Furthermore, by utilizing a loss function based on cross-entropy of multiple categories and consistency constraints of location bounding box prediction, and in conjunction with the AdamW optimizer, the scene graph generation is gradually optimized through multiple iterations. The training model learns entity detection and relationship detection based on the anti-diffusion process of step S100, enabling it to predict the corresponding entities and location-based relationships from randomly sampled noisy candidate boxes.

[0089] Step S340 includes:

[0090] S341: Deep learning network predicts training images and obtains prediction results;

[0091] S342: Calculate the cross-entropy loss function, mean absolute value error loss function, and generalized cross-union ratio loss function of the prediction results using the optimal transmission allocation method;

[0092] S343: Based on the cross-entropy loss function, the mean absolute value error loss function, and the generalized intersection-union loss function, iterate the deep learning network multiple times to obtain an optimized deep learning network.

[0093] S400: Obtain the entity location bounding box and relation location bounding box of the image to be processed through the diffusion model, calculate the intersection-union ratio of the entity location bounding box and the relation location bounding box, and match according to the highest intersection-union ratio to obtain relation triples;

[0094] In some embodiments, the goal of this stage is to obtain relation triples. Unlike traditional graph-based relation modeling and pairwise query-based relation modeling, this invention detects entity-based and location-based relations respectively, and uses an entity and location box matching strategy to match the subject and object location boxes of each relation with at most Q entities with the highest intersection-union ratio, thereby obtaining relation triples.

[0095] Furthermore, entity noise candidate boxes and relation noise candidate box pairs are first randomly sampled from Gaussian noise. Based on a deep learning model, candidate entities and position-based relations existing in the input image are predicted. The intersection-union ratio of the subject and object location boxes of each relation and all candidate entity location boxes is calculated.

[0096] Furthermore, based on the intersection-union ratio (IUU) between the relation subject / object location boxes and the entity location boxes calculated above, the Q entity prediction results with the maximum IUU ratio are selected and detected using the maximum Q-value selection method as the matching results for the relation subject or object location boxes. The calculation method is as follows:

[0097]

[0098] in, To the maximum Value selection operation, For the number of entity-relationship matches, This is the intersection-union ratio (IUU) of the obtained relational subject or object with the detected entity location bounding boxes. A mapping matrix representing the subject or object bounding boxes of a relationship to the matching entities.

[0099] S500: Generate a scene graph based on the constraints of the relation triplet combined graph structure.

[0100] According to the present invention, a scene graph generation method based on a diffusion model is provided, wherein the noise is random noise that conforms to a Gaussian distribution.

[0101] Step S500 further includes:

[0102] S510: Filter out valid relation triples from the relation triples to generate a scene graph in combination with graph structure constraints.

[0103] Step S510 includes:

[0104] S511: A relation triplet with a relation predicate between two entities is designated as a valid relation triplet.

[0105] S512: For a relation triplet between two entities with multiple relation predicates, calculate the prediction probability of any relation predicate, and label the two entities and the relation predicate with the highest prediction probability as a valid relation triplet;

[0106] S513: Construct a directed acyclic graph structure based on the effective relation triples to generate the scene graph.

[0107] The effectiveness of the diffusion model-based scene graph generation method of the present invention is verified below. Experimental results show that the present invention outperforms other methods in terms of the quality and scalability of end-to-end scene graph generation.

[0108] This invention conducted scene graph generation experiments on the widely used public scene graph generation dataset Visual Genome. Specifically, this invention selected the VG150 dataset partitioning method, which contains 108,000 images and includes 150 classes of target entity annotations and 50 classes of predicate annotations. In addition, this invention uses 70% of the data in the dataset for training and 30% for testing.

[0109] The experiment comprises three subtasks: Predicate Classification (PredCls, classifying relation predicates between paired target entity classes and bounding boxes), Scene Graph Classification (SGCls, classifying target entities and their relationships given their locations), and Scene Graph Generation (SGDet, generating scene graphs from given images). For all three subtasks, six commonly used evaluation metrics are employed: R@K (Recall of top K triplets), ng-R@K (No-graph constraint Recall of top K triplets), mR@K (Mean Recall of top K triplets), ng-mR@K (No-graph constraint mean Recall of top K triplets), zR@K (Zero-shot Recall of top K triplets), and ng-zR@K (No-graph constraint mean Recall of top K triplets). Zero-shot recall of top K triplets (non-graph-constrained zero-shot recall of the top K triplets).

[0110] Experimental results show that the method proposed in this invention improves performance to varying degrees under different experimental settings compared to other end-to-end scene graph generation methods, especially significantly outperforming existing end-to-end scene graph generation methods (FCSGG and CoRF) in the scene graph generation subtask. Furthermore, the method demonstrates significant performance improvements under different inference settings, indicating that it possesses flexibility and scalability not found in existing scene graph generation methods. In particular, the scene graphs generated by this method can detect more semantic relationships in the images more accurately. These comparative results fully demonstrate that the method proposed in this invention achieves excellent results in end-to-end scene graph generation tasks.

[0111] This invention provides a scene graph generation method based on a diffusion model for end-to-end image scene graph generation. It explicitly extracts visual features of entities and relationships using random noise boxes, achieving more accurate feature representations and thus stronger relationship detection capabilities. Based on the diffusion model architecture, it predicts the possible existence of relational predicates between any two positions in the image from randomly sampled noise boxes, eliminating the dependence on object detection models in traditional scene graph generation architectures. It also enables the detection of relationships between target entities at multiple scales, demonstrating stronger performance in detecting complex relationships and zero-shot relational predicates. To enhance the flexibility and scalability of the scene graph generation model, the diffusion model-based training and inference architecture allows the model to handle multiple inference settings with a single training iteration, achieving a trade-off between model performance and time cost. In other words, this invention can explicitly and more accurately model image entity and relationship features, significantly improving scene graph generation performance. Simultaneously, the relation detection mechanism based on random noise boxes eliminates the dependence on object detection models, achieving stronger zero-shot and complex relationship detection capabilities. Furthermore, the diffusion model-based training and inference architecture significantly enhances the flexibility and scalability of the model during the inference phase.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A scene graph generation method based on a diffusion model, characterized in that, include: S100: Obtain training data containing annotation information, obtain entity candidate boxes and relationship candidate boxes based on the training data, add noise to the entity candidate boxes and relationship candidate boxes, and obtain entity noise candidate boxes and relationship noise candidate boxes; S200: Extract features from the image to be processed using the entity noise candidate box and the relation noise candidate box to obtain entity features and relation features; S300: Construct a deep learning network based on the entity features and the relationship features, and learn the anti-diffusion process of entity detection and relationship detection through the deep learning network to obtain a diffusion model; step S300 includes: S310: Embed sampling time into the entity features and the relationship features, and correct the entity noise candidate box and the relationship noise candidate box according to the entity features and relationship features after embedding the sampling time; S320: Combine the entity features and relation features embedded after the sampling time with a classification network to predict the predicted entity category and the predicted relation category; S330: Construct the deep learning network based on the entity features, the relationship features, the predicted entity category, and the predicted relationship category; S340: Optimize the deep learning network by predicting the ground truth values ​​of entity categories and relation categories through optimal transmission allocation; S350: The optimized deep learning network learns the anti-diffusion process and is trained to obtain the diffusion model; S400: Obtain the entity location bounding box and relation location bounding box of the image to be processed through the diffusion model, calculate the intersection-union ratio of the entity location bounding box and the relation location bounding box, and match according to the highest intersection-union ratio to obtain relation triples; S500: Generate a scene graph based on the constraints of the relation triplet combined graph structure.

2. The scene graph generation method based on a diffusion model according to claim 1, characterized in that, The noise is random noise that follows a Gaussian distribution.

3. The scene graph generation method based on a diffusion model according to claim 1, characterized in that, Step S100 includes: S110: Introduce a scene graph generation dataset, and select training data from the scene graph generation dataset; S120: Extract entity candidate boxes and relation candidate boxes from the training data; S130: Transform the entity candidate box and the relationship candidate box from the representation space of the upper left and lower right coordinates to the representation space of the center coordinate size to obtain the entity candidate box and the relationship candidate box in the center coordinate size space; S140: Add noise to the entity candidate boxes and relation candidate boxes in the center coordinate size space to obtain pre-entity noise candidate boxes and pre-relation noise candidate boxes; S150: Transform the pre-entity noise candidate box and the pre-relationship noise candidate box from the representation space of the center coordinate to the representation space of the upper left and lower right coordinates to obtain entity noise candidate boxes and relation noise candidate boxes.

4. The scene graph generation method based on a diffusion model according to claim 3, characterized in that, The noise addition process in step S140 is represented as follows: in, For entity noise candidate boxes, Candidate boxes for relational noise. Label the truth values ​​for entity candidate boxes. Label the truth values ​​for the relation candidate boxes. For diffusion process, The noise added at time t.

5. The scene graph generation method based on a diffusion model according to claim 1, characterized in that, Step S340 includes: S341: Deep learning network predicts training images and obtains prediction results; S342: Calculate the cross-entropy loss function, mean absolute value error loss function, and generalized cross-union ratio loss function of the prediction results using the optimal transmission allocation method; S343: Based on the cross-entropy loss function, the mean absolute value error loss function, and the generalized intersection-union loss function, iterate the deep learning network multiple times to obtain an optimized deep learning network.

6. The scene graph generation method based on a diffusion model according to claim 1, characterized in that, Step S500 also includes: S510: Filter out valid relation triples from the relation triples to generate a scene graph in combination with graph structure constraints.

7. The scene graph generation method based on a diffusion model according to claim 6, characterized in that, Step S510 includes: S511: A relation triplet with a relation predicate between two entities is designated as a valid relation triplet. S512: For a relation triplet between two entities with multiple relation predicates, calculate the prediction probability of any relation predicate, and label the two entities and the relation predicate with the highest prediction probability as a valid relation triplet; S513: Construct a directed acyclic graph structure based on the effective relation triples to generate the scene graph.

Images