A predicate and triple-based multi-granularity scene graph generation method

Abstract

The application belongs to the technical field of image processing, and proposes a multi-granularity scene graph generation method based on predicates and triplets. The method comprises: establishing an image dataset; constructing a scene graph generation model based on predicates and triplets; setting model training constraints, using joint loss to train the model training constraints in the first stage, and the joint loss comprises cross-entropy loss and contrastive learning loss, using cross-entropy loss to train the model training constraints in the second stage; training the scene graph generation model, using the back propagation algorithm to iteratively update and optimize the model parameters until the model loss area converges; and applying the trained scene graph generation model to scene graph generation. The application supplements the coarse-granularity predicate features and the fine-granularity triplet features between entities respectively, makes up for the defects that the existing algorithms are insufficient in modeling the predicates with large visual appearance changes, and expands the scene graph generation ideas based on the deep learning model.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method for generating multi-granularity scene graphs based on predicates and triples. Background Technology

[0002] Entities and the relationships (predicates) between them are the cornerstone of understanding visual content. Scene graph generation is a fundamental task in visual content understanding, aiming to detect entities in visual images and predict the predicates between them. This objective can be represented as a <subject, predicate, object> triple. Thus, the triple represents a fine-grained category of predicates. The subject and object are entities. Since entities can be identified using object detection techniques, the focus of scene graph generation is on recognizing the predicates between entities. The generated scene graphs can be used in various domains, such as embodied intelligence, image retrieval, and visual question answering tasks.

[0003] Existing scene graph generation methods mainly rely on entity information from object detection to extract predicate features that distinguish between entities. However, the visual appearance of predicates varies greatly under different subject-object combinations, which poses a challenge to scene graph generation models in capturing robust predicate features. To address this issue, Chaofan Zheng et al. (Zheng C, Lyu X, Gao L, et al. Prototype-based embedding network for scene graph generation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:22783-22792.) proposed in 2023 that text features of predicate labels are used as predicate category prototypes to maximize the distance between features of different predicate categories and minimize the distance between predicate features within a category. Although this improves the ability of scene graph generation models to identify predicate categories, this method still does not consider the objective fact that the visual appearance of the same predicate category varies greatly, making it difficult to directly extract predicate cues.

[0004] Furthermore, many recent methods have begun to address the long-tail problem in scene graph generation tasks. The limited number of predicate categories and insufficient triplet types in the tail of the predicate make it difficult for models to adapt to the dataset distribution. Existing scene graph generation methods enhance attention to tail predicate categories through reweighting and resampling schemes, but they neglect the essence of the long-tail problem: the patterns of tail predicate categories are insufficient to generalize to more test samples. Summary of the Invention

[0005] Although samples belonging to the same predicate category may have significantly different visual appearances, these differences diminish when the focus is narrowed to a specific triplet. Therefore, fine-grained triplet features aid in the identification of coarse-grained predicate categories, increasing the generalizable predicate patterns. For the long-tail problem in scene graph generation, it is natural to use coarse-grained predicate features and fine-grained triplet features at the tail end to amplify patterns and alleviate the long-tail problem. Based on this, this invention proposes a multi-granularity scene graph generation method based on predicates and triples, the technical solution of which is as follows:

[0006] A multi-granularity scene graph generation method based on predicates and triples includes the following steps:

[0007] Step S1: Create an image dataset;

[0008] Step S2: Construct a scene graph generation model, which is a two-stream scene graph generation network. It models both predicate context and triplet context, and integrates the extracted information at the end stage to form mutual guidance. Then, it integrates the predicate context and triplet context to achieve scene graph generation.

[0009] Step S3: Set model training constraints. The first stage of training uses joint loss to constrain the model training. The joint loss includes cross-entropy loss and contrastive learning loss. The second stage of training uses cross-entropy loss to constrain the model training.

[0010] Step S4: Train the scene graph generation model. The training uses the backpropagation algorithm to iteratively update and optimize the model parameters until the model loss region converges.

[0011] Step S5: Apply the trained scene graph generation model to scene graph generation.

[0012] Preferably, the scene graph generation model includes an object detector, a predicate context modeler, a triplet context modeler, and a feature fusion module; wherein,

[0013] The object detector is used to extract initialized entity features, initialized predicate features, and initialized triplet features from image data;

[0014] The predicate context modeler takes initialized entity features and initialized predicate features as inputs, models the context within predicates, within entities, and between predicates and entities, and obtains refined predicate features;

[0015] The triplet context modeler takes the initial triplet features as input, models the context within the triplet, and obtains refined triplet features.

[0016] The feature fusion module is used to fuse the refined predicate features and refined triplet features and classify them.

[0017] Preferably, the predicate context modeler is divided into two layers, each of which includes a predicate context module, an entity context module, and a context modeling module between predicates and entities.

[0018] Preferably, the triple context modeler is divided into two layers, each of which includes a triple visual context module, a triple semantic context module, and a triple visual and semantic context module.

[0019] Preferably, the joint loss of the first stage of training is calculated as follows:

[0020] 1) The cross-entropy loss The cross-entropy between predicate probabilities and true labels:

[0021]

[0022] Where y i,j Indicates the true label, prob i,j Let p represent the predicate probability, and M be the predicate feature. i,j The number of;

[0023] 2) Contrastive learning loss is used to constrain the features of predicates and triples of the same category:

[0024]

[0025] in This represents the contrastive learning loss of triples. τ represents the predicate contrastive learning loss. t τ represents the temperature coefficient of the contrastive learning loss of triplet pairs. p The temperature coefficient represents the predicate contrast loss, and <, ·> represents the dot product. This represents the triplet representation of subject i and object j. Triple features obtained through spatial mapping Indicates and Triples belonging to the same triplet category are represented as follows: This represents the predicate representation of subject i and object j. Predicate features obtained through spatial mapping Indicates and Predicates belonging to the same predicate category;

[0026]

[0027] Among them W t and W p All of these are weight matrices that can be optimized through backpropagation;

[0028] 3) The combined loss is:

[0029]

[0030] Where, λ cls , and All of these represent parameters for the joint loss trade-off.

[0031] Preferably, the cross-entropy loss in the second stage is calculated as follows:

[0032]

[0033] Where y i,j Indicates the true label, prob i,j Let p represent the predicate probability, and M be the predicate feature. i,j The number of.

[0034] Preferably, the training of the scene graph generation model in step S4 is divided into two stages. The first stage trains the scene graph generation model with image data from the image dataset as input, and the second stage trains the feature fusion module of the scene graph generation model by constructing predicate features and triple features through knowledge transfer methods as input.

[0035] Preferably, the second stage of training constructs predicate features and triplet features based on multivariate Gaussian distribution estimation and sampling through knowledge transfer methods, including:

[0036] 1) Assumption and It follows a multivariate Gaussian distribution, where the predicate representations belonging to predicate category c are... Record Belongs to the triplet category<s,c,o> The triplet representation is denoted as Let s be the category of subject i and o be the category of object j, with their mean and covariance matrices as follows:

[0037]

[0038] Among them, M c This represents the predicate features in the data that belong to predicate category c. Number, M <s,c,o> This indicates that the data belongs to the category of triples.<s,c,o> Triple characteristics number, and The mean and covariance of the predicate features representing predicate category c are given. and Indicates the category of triples<s,c,o> The mean and covariance of the triplet characteristics;

[0039] 2) Define knowledge transfer as the transfer covariance matrix from predicate categories or triple categories with a large number of samples to predicate categories or triple categories with a small number of samples;

[0040] The knowledge transfer process in predicates is expressed by the following formula:

[0041]

[0042] Where β c,k The mean of the c-th predicate category and the k-th predicate category and Euclidean similarity between them, Q c To construct the target number of predicate features of type c, The predicate covariance matrix after knowledge transfer;

[0043] The knowledge transfer process in triplet is expressed by the following formula:

[0044]

[0045] Where β <s,c,o>,k For the first<s,c,o> The mean of the k-th triplet category and the k-th triplet category and Euclidean similarity between them, Q <s,c,o> To construct the first<s,c,o> The number of targets for triple-like features The triplet covariance matrix after knowledge transfer;

[0046] 3) Triple feature samples are obtained by sampling from the multivariate Gaussian distribution after knowledge transfer:

[0047]

[0048] in Representative with As the mean, with The covariance is a multivariate Gaussian distribution. Features of the triplet obtained from sampling;

[0049] 4) Predicate feature samples are obtained by sampling from the multivariate Gaussian distribution after knowledge transfer:

[0050]

[0051] in Representative with As the mean, with The covariance is a multivariate Gaussian distribution. These are the predicate features obtained from sampling.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] This invention proposes a multi-granularity scene graph generation method based on predicates and triples. By effectively modeling coarse-grained predicate features and fine-grained triple features, it overcomes the deficiency of insufficient predicate modeling in existing algorithms when dealing with large variations in visual appearance. By transferring predicate knowledge from the head to the tail, it effectively alleviates the long-tail problem and expands the ideas for scene graph generation based on deep learning models. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of the multi-granularity scene graph generation method based on predicates and triples of the present invention.

[0056] Figure 2 This is a diagram of the scene graph generation model architecture constructed for this invention.

[0057] Figure 3 This is the scene graph result output by the scene graph generation model of the present invention. Detailed Implementation

[0058] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0059] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0060] A multi-granularity scene graph generation method based on predicates and triples includes the following steps:

[0061] Step 1: Prepare the image dataset. Select the image dataset and divide the image data.

[0062] Step 2: Construct a scene graph generation model. The scene graph generation model is a two-stream scene graph generation network that models both predicate context and triplet context. At the end stage, the extracted information is fused to form mutual guidance, and then the predicate context and triplet context are fused to realize scene graph generation.

[0063] Specifically, the scene graph generation model includes an object detector, a predicate context modeler, a triplet context modeler, and a feature fusion module; wherein,

[0064] The object detector is used to extract initialized entity features, initialized predicate features, and initialized triplet features from image data;

[0065] The predicate context modeler takes initialized entity features and initialized predicate features as inputs, models the context within predicates, within entities, and between predicates and entities, and obtains refined predicate features;

[0066] The triplet context modeler takes the initial triplet features as input, models the context within the triplet, and obtains refined triplet features.

[0067] The feature fusion module is used to fuse the refined predicate features and refined triplet features and classify them.

[0068] 1) The predicate context modeler is divided into two layers, each containing a predicate context module, an entity context module, and a predicate and entity context modeling module, used to refine predicate features. For the input initial predicate features {p i,j} M and initializing entity features {e i} N The predicate context modeler will calculate refined predicate features. The specific calculations are as follows:

[0069]

[0070] PCM1 and PCM2 represent the first and second layers of the predicate context modeler, respectively, and both have the same model structure and computation method. M represents p i,j The number of elements, N is e i The number of elements, where M = N × (N-1). and The results of predicate features and entity features computed in the first layer of the predicate context modeler. Predicate features p i,j This indicates that its main body is entity e. i and the object is entity e j Taking PCM1 as an example, for the input initial predicate features {p i,j} Mand initializing entity features {e i} N Calculate the features {p′} of each feature with its own internal context. i,j} M and {e′ i} N Then, calculate the features {p″} of the predicate with entity context and the entity with predicate context. i,j} M and {e″ i} N Adding them together, we get and The specific calculations are as follows:

[0071] {p′ i,j} M =SelfAttention({p i,j} M )

[0072] {p″ i,j} M =CrossAttention({p i,j} M ,{e i} N )

[0073]

[0074] {e′ i} N =SelfAttention({e i} N )

[0075] {e″ i} N =CrossAttention({e i} N ,{p i,j} M )

[0076]

[0077] Here, SelfAttention represents the self-attention operator, whose goal is to obtain contextual information between input feature sets, and CrossAttention represents the cross-attention operator, whose goal is to obtain contextual information between two input feature sets. Taking the calculation of predicate features {p′ in PCM1 as an example... i,j} M Taking the self-attention operator as an example, the self-attention operator first initializes the input predicate feature set {p}. i,j}M Mapped to A total of 24 types of spaces were obtained. and Let q1…q8 represent the "query features" mapped to the 8 feature spaces, k1…k8 represent the "key features" mapped to the 8 feature spaces, and v1…v8 represent the "value features" mapped to the 8 feature spaces. Then calculate… and and Attention matrix between feature sets Finally, the attention matrix and... Multiplying and concatenating the spatial feature sets yields the output {p′ i,j} M The specific calculations are as follows:

[0078]

[0079] Where ⊙ represents matrix multiplication, () T Represents matrix transpose; Concat represents concatenation. W SA This represents the weight matrix that can be optimized through backpropagation. The predicate features {p″} are calculated using PCM1. i,j} N×(N-1) Taking the cross-attention operator as an example, the cross-attention operator first combines the input feature set {p} i,j} M and {e i} N Mapped to A total of 24 types of spaces were obtained. and Then calculate and and Attention matrix between feature sets Finally, the attention matrix and... Multiplying and concatenating spatial feature sets yields the output {p″ i,j} M The specific calculations are as follows:

[0080]

[0081] 2) The triplet context modeler consists of two layers, each containing a triplet visual context module, a triplet semantic context module, and a triplet visual and semantic context module, used to refine triplet features. The triplet visual features are initialized as {Concat(e i ,p i,j ,e j )} M The semantic features of the triple are initialized to {s} i,j}M , where s i,j Entity e extracted for the target detector i and e j The corresponding semantic information. For the input triplet visual features and triplet semantic features, the triplet context modeler will obtain refined triplet features. The specific calculations are as follows:

[0082]

[0083] TCM1 and TCM2 represent the first and second layers of the triplet context modeler, respectively, and both have the same model structure and operation method. and The results of calculating the visual and semantic features of triples in the first layer of the triple context modeler. Triple feature t i,j This indicates that it is the subject or entity e. i and the object is entity e j Taking TCM1 as an example, for the initialization triplet visual features of the input {Concat(e i ,p i,j ,e j )} M and the semantic features of the initialized triples {s i,j} M Calculate the features {t′} of each feature with its own internal context. i,j} M and {s′ i,j} M Then, calculate the features {t″} of the triple visual context with semantic context and the triple semantic context with visual context. i,j} M and {s″ i,j} M Adding them together, we get and The specific calculations are as follows:

[0084] {t′ i,j} M =SelfAttention({Concat(e i ,p i,j ,e j )} M )

[0085] {t″ i,j} M =CrossAttention({Concat(e i ,p i,j ,e j )} M,{s i,j} M )

[0086]

[0087] {s′ i,j} M =SelfAttention({s i,j} M )

[0088] {s″ i,j} M =CrossAttention({s i,j} M ,{p i,j} M )

[0089]

[0090] 3) The feature fusion module refines predicate features. and refined triplet features The data is then assembled and categorized. The specific calculations are as follows:

[0091]

[0092]

[0093] Among them W f This represents the weights that can be optimized through backpropagation. Here, exp refers to the exponent with the base of the natural logarithm. logit i,j Indicates the weight W f The output score, prob i,j That is, the model output is entity e i As the main body and with entity e j The predicate probability of the object.

[0094] Step 3: Set model training constraints. The first stage of training uses joint loss to constrain the model training. The joint loss includes cross-entropy loss and contrastive learning loss. The second stage of training uses cross-entropy loss to constrain the model training.

[0095] Specifically, the joint loss for the first stage of training is calculated as follows:

[0096] 1) The cross-entropy loss The cross-entropy between predicate probabilities and true labels:

[0097]

[0098] Where y i,jIndicates the true label, prob i,j Let p represent the predicate probability, and M be the predicate feature. i,j The number of;

[0099] 2) Contrastive learning loss is used to constrain the features of predicates and triples of the same category:

[0100]

[0101] in This represents the contrastive learning loss of triples. τ represents the predicate contrastive learning loss. t τ represents the temperature coefficient of the contrastive learning loss of triplet pairs. p The temperature coefficient represents the predicate contrast loss, and <, ·> represents the dot product. This represents the triplet representation of subject i and object j. Triple features obtained through spatial mapping Indicates and Triples belonging to the same triplet category are represented as follows: This represents the predicate representation of subject i and object j. Predicate features obtained through spatial mapping Indicates and Predicates belonging to the same predicate category;

[0102]

[0103] Among them W t and W p All of these are weight matrices that can be optimized through backpropagation;

[0104] 3) The combined loss is:

[0105]

[0106] Where, λ cls , and All of these represent parameters for the joint loss trade-off.

[0107] The cross-entropy loss in the second stage is calculated as follows:

[0108]

[0109] Where y i,j Indicates the true label, prob i,j Let p represent the predicate probability, and M be the predicate feature. i,j The number of.

[0110] Step 4: Train the scene graph generation model. The training uses the backpropagation algorithm to iteratively update and optimize the model parameters until the model loss region converges.

[0111] Specifically, the training of the scene graph generation model is divided into two stages. The first stage trains the scene graph generation model with image data from the image dataset as input, and the second stage trains the feature fusion module of the scene graph generation model by constructing predicate features and triple features through knowledge transfer methods.

[0112] The second stage of training constructs predicate features and triplet features based on multivariate Gaussian distribution estimation and sampling through knowledge transfer methods, including:

[0113] 1) Assumption and It follows a multivariate Gaussian distribution, where the predicate representations belonging to predicate category c are... Record Belongs to the triplet category<s,c,o> The triplet representation is denoted as Let s be the category of subject i and o be the category of object j, with their mean and covariance matrices as follows:

[0114]

[0115] Among them, M c This represents the predicate features in the data that belong to predicate category c. Number, M <s,c,o> This indicates that the data belongs to the category of triples.<s,c,o> Triple characteristics number, and The mean and covariance of the predicate features representing predicate category c are given. and Indicates the category of triples<s,c,o> The mean and covariance of the triplet characteristics;

[0116] 2) Define knowledge transfer as the transfer covariance matrix from predicate categories or triple categories with a large number of samples to predicate categories or triple categories with a small number of samples;

[0117] The knowledge transfer process in predicates is expressed by the following formula:

[0118]

[0119] Where β c,k The mean of the c-th predicate category and the k-th predicate category and Euclidean similarity between them, Q c To construct the target number of predicate features of type c, The predicate covariance matrix after knowledge transfer;

[0120] The knowledge transfer process in triplet is expressed by the following formula:

[0121]

[0122] Where β <s,c,o>,k For the first<s,c,o> The mean of the k-th triplet category and the k-th triplet category and Euclidean similarity between them, Q <s,c,o> To construct the first<s,c,o> The number of targets for triple-like features The triplet covariance matrix after knowledge transfer;

[0123] 3) Triple feature samples are obtained by sampling from the multivariate Gaussian distribution after knowledge transfer:

[0124]

[0125] in Representative with As the mean, with The covariance is a multivariate Gaussian distribution. Features of the triplet obtained from sampling;

[0126] 4) Predicate feature samples are obtained by sampling from the multivariate Gaussian distribution after knowledge transfer:

[0127]

[0128] in Representative with As the mean, with The covariance is a multivariate Gaussian distribution. These are the predicate features obtained from sampling.

[0129] Step 5: Apply the trained scene graph generation model to scene graph generation.

[0130] Example 1

[0131] A multi-granularity scene graph generation method based on predicates and triples, such as Figure 1 As shown, the steps are as follows:

[0132] 1. Image dataset preparation: Select and divide the image dataset.

[0133] 1.1 This example selects the commonly used Visual Genome dataset, Open Image V6 dataset, and GQA dataset.

[0134] 1.2 For the Visual Genome dataset, 57,723 images were used for training, 5,000 for validation, and 26,446 for testing. For the Open Image V6 dataset, 126,368 images were used for training, 1,183 for validation, and 5,322 for testing. For the GQA dataset, 57,613 images were used for training, 5,000 for validation, and 8,208 for testing.

[0135] 2. Construct a scene graph generation model, such as Figure 2 As shown, the scene graph generation model includes an object detector, a predicate context modeler, a triplet context modeler, and a feature fusion module.

[0136] 3. Design model training constraints. Model training is divided into two stages. The first stage uses joint loss to constrain model training, which includes cross-entropy loss and contrastive learning loss. The second stage uses cross-entropy loss. Specific constraint designs have been discussed in the invention description and will not be repeated here. The trade-off parameters for each sub-constraint of the joint loss are set as follows:

[0137] λ cls =3,

[0138] 4. The scene graph generation model is trained. In the first stage, backpropagation is used to update and optimize the weights of all network parameters until the model loss converges. In the second stage, backpropagation is used to update and optimize only the weights of the feature fusion module network parameters. In this embodiment, both scene graph generation training and evaluation are performed on the PyTorch platform. The model is trained on a single NVIDIA A800 GPU (80GB) with a batch size of 16. The SGD optimizer is used in both the first and second stages, with an initial learning rate of 1×10⁻⁶. -4 Train the model. For all image datasets, observe the model's performance on the validation dataset after every 2000 iterations. Stop training the model when the validation set results no longer improve.

[0139] 5. After model training is complete, scene graph generation inference is performed. The image is input into the scene graph generation model, and the model output is the scene graph result. For example... Figure 3 As shown, the present invention can effectively complete the task of scene graph generation.

[0140] The nonlinear electromagnetic field analysis method described in this embodiment achieves efficient numerical analysis of nonlinear electromagnetic fields with stochastic effects, minimizing computational scale and improving computational efficiency. Furthermore, it does not require solving ultra-large-scale algebraic equations at each time step and ensures the convergence of the numerical analysis method.

[0141] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0142] In this invention, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating multi-granularity scene graphs based on predicates and triples, characterized in that, Includes the following steps: Step S1: Create an image dataset; Step S2: Construct a scene graph generation model, which is a two-stream scene graph generation network. It models both predicate context and triplet context, and integrates the extracted information at the end stage to form mutual guidance. Then, it integrates the predicate context and triplet context to achieve scene graph generation. Step S3: Set model training constraints. The first stage of training uses joint loss to constrain the model training. The joint loss includes cross-entropy loss and contrastive learning loss. The second stage of training uses cross-entropy loss to constrain the model training. Step S4: Train the scene graph generation model. The training uses the backpropagation algorithm to iteratively update and optimize the model parameters until the model loss region converges. Step S5: Apply the trained scene graph generation model to scene graph generation; The training of the scene graph generation model in step S4 is divided into two stages. The first stage trains the scene graph generation model with image data from the image dataset as input. The second stage trains the feature fusion module of the scene graph generation model by constructing predicate features and triplet features through knowledge transfer methods as input. The second stage of training constructs predicate features and triplet features based on multivariate Gaussian distribution estimation and sampling through knowledge transfer methods, including: 1) Assumption and It follows a multivariate Gaussian distribution, where the values ​​belong to the predicate category. Predicate representation Record It belongs to the triplet category. The triplet representation is denoted as , s as the main body i Category o as an object j The categories, with their means and covariance matrices as follows: in, This indicates that the data belongs to the predicate category. c Predicate features number, This indicates that the data belongs to the category of triples. Triple characteristics number, and Indicates the predicate category The mean and covariance of the predicate features, and Indicates the category of triples The mean and covariance of the triplet characteristics; 2) Define knowledge transfer as the transfer covariance matrix from predicate categories or triple categories with a large number of samples to predicate categories or triple categories with a small number of samples; The knowledge transfer process in predicates is expressed by the following formula: in For the first The predicate category and the first Mean of each predicate category and The European similarity between them To construct the first The number of target predicate features The predicate covariance matrix after knowledge transfer; The knowledge transfer process in triplet is expressed by the following formula: in For the first The three-pair category and the first Triple category mean and The European similarity between them To construct the first The number of targets for triple-like features The triplet covariance matrix after knowledge transfer; 3) Triple feature samples are obtained by sampling from the multivariate Gaussian distribution after knowledge transfer: in Representative with As the mean, with The covariance is a multivariate Gaussian distribution. Features of the triplet obtained from sampling; 4) Predicate feature samples are obtained by sampling from the multivariate Gaussian distribution after knowledge transfer: in Representative with As the mean, with The covariance is a multivariate Gaussian distribution. These are the predicate features obtained from sampling.

2. The multi-granularity scene graph generation method according to claim 1, characterized in that, The scene graph generation model includes an object detector, a predicate context modeler, a triplet context modeler, and a feature fusion module; wherein... The object detector is used to extract initialized entity features, initialized predicate features, and initialized triplet features from image data; The predicate context modeler takes initial entity features and initial predicate features as inputs, models the context within predicates, within entities, and between predicates and entities, and obtains refined predicate features; The triplet context modeler takes the initial triplet features as input, models the context within the triplet, and obtains refined triplet features. The feature fusion module is used to fuse the refined predicate features and refined triplet features and classify them.

3. The multi-granularity scene graph generation method according to claim 2, characterized in that, The predicate context modeler is divided into two layers, each containing a predicate context module, an entity context module, and a context modeling module between predicates and entities.

4. The multi-granularity scene graph generation method according to claim 2, characterized in that, The triple context modeler is divided into two layers, each containing a triple visual context module, a triple semantic context module, and a triple visual and semantic context module.

5. The multi-granularity scene graph generation method according to claim 1, characterized in that, The joint loss for the first stage of training is calculated as follows: 1) The cross-entropy loss The cross-entropy between predicate probabilities and true labels: in Indicates the true label, Represents the probability of the predicate. Features of predicates The number of; 2) Contrastive learning loss is used to constrain the features of predicates and triples of the same category: in This represents the contrastive learning loss of triples. This indicates the learning loss due to predicate contrast. The temperature coefficient representing the contrastive learning loss of triplet pairs. The temperature coefficient representing the predicate contrastive learning loss. Represents the dot product. Representing the subject and object Corresponding triplet representation Triple features obtained through spatial mapping Indicates and Triples belonging to the same triplet category are represented as follows: Representing the subject and object Corresponding predicate representation Predicate features obtained through spatial mapping Indicates and Predicates belonging to the same predicate category; ， in and All of these are weight matrices that can be optimized through backpropagation; 3) The combined loss is: in, , and All of these represent parameters for the joint loss trade-off.

6. The multi-granularity scene graph generation method according to claim 1, characterized in that, The cross-entropy loss in the second stage is calculated as follows: in Indicates the true label, Represents the probability of the predicate. Features of predicates The number of.

Images