A bias-free scene graph generation method based on a double-branch hybrid learning network

Abstract

The application discloses a kind of methods for generating unbiased scene graph based on double-branch hybrid learning network, it is related to image processing technical field, solve recent unbiased scene graph generation method cannot simultaneously pay attention to head predicate and tail predicate learning technical problem, including the following steps: image is input into target detector, the visual feature, semantic feature and spatial feature of instance in image are obtained;The features of instance in image are sent into coarse-grained learning branch and fine-grained learning branch, wherein coarse-grained learning branch is used to learn the expert knowledge of head predicate and obtain the robustness feature of head predicate, fine-grained learning branch is used to predict informative tail predicate;Knowledge distillation, using semantic context module, unreasonable prediction in the prediction result of fine-grained learning branch is corrected, the gap between the semantics of predicted scene graph and the semantics of labeled scene graph is calculated simultaneously, and is optimized by mean square error loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, more particularly to a bias-free scene graph generation method based on a double-branch hybrid learning network. BACKGROUND

[0002] In recent years, deep neural networks have been widely used in various fields due to their superior performance. In addition to basic cognitive tasks such as image classification and object detection, some application tasks that require higher-level semantic understanding have also received great attention with the development of visual cognition and natural language processing, such as image description generation and image question answering. However, basic cognitive tasks treat instances in images in isolation, ignoring the visual relationship information between different instances, and thus cannot fully understand the visual content in the image.

[0003] Image scene graph generation is a task that specifically detects the visual relationship between different instances, which uses a graph structure to represent the visual relationship, where nodes represent instances detected in the image, and edges represent the relationship between instances, containing different relationship predicate information. The scene graph generation task can provide a medium-level image abstract feature for downstream tasks, facilitating further processing.

[0004] Because the widely used scene graph dataset Visual Genome has a natural long-tail distribution, if a common model is used for training, most of the predicted results will be some common predicates, such as "on", "has", and "near", which cannot provide rich semantics, resulting in the generated picture scene graph only providing a small amount of semantic information, which is less helpful for downstream tasks. Therefore, most current methods are solving the long-tail distribution problem to generate bias-free scene graphs in order to predict more predicates with rich information, such as "standing on", "parked on", etc. However, many bias-free scene graph generation methods overemphasize the tail predicates and underestimate the importance of the head predicates during the entire training process, thereby damaging the representation ability of the head predicate features. In addition, because the inference of the tail predicates is highly dependent on the basic features provided by the head predicates, such as "standing on" and "parked on" being highly dependent on "on", the damaged head predicate features also affect the learning of the tail predicates. Therefore, these bias-free scene graph generation methods neither achieve very good tail predicate recognition performance nor achieve satisfactory head predicate recognition performance. SUMMARY

[0005] The application aims at solving the technical problem that the recent unbiased scene graph generation method cannot simultaneously consider head predicate and tail predicate learning, and provides an unbiased scene graph generation method based on a double-branch mixed learning network.

[0006] In order to achieve the above-mentioned purpose, the application specifically adopts the following technical scheme:

[0007] The application provides an unbiased scene graph generation method based on a double-branch mixed learning network, comprising the following steps:

[0008] S1, inputting an image into a target detector to obtain visual features, semantic features and spatial features of instances in the image;

[0009] S2, inputting the features of the instances in the image into a coarse-grained learning branch and a fine-grained learning branch, wherein the coarse-grained learning branch is used to learn expert knowledge of head predicates and obtain robust features of the head predicates, and the fine-grained learning branch is used to predict informative tail predicates;

[0010] S3, knowledge distillation, distilling the expert knowledge of the coarse-grained learning branch on the head predicates to the fine-grained learning branch;

[0011] S4, adopting a semantic context module to correct unreasonable predictions in the prediction results of the fine-grained learning branch, simultaneously calculating the gap between the semantics of the predicted scene graph and the semantics of the labeled scene graph, and optimizing by using a mean square error loss;

[0012] S5, using a standard cross-entropy loss function to optimize the prediction results of the coarse-grained learning branch, and using a curriculum reweighting mechanism to optimize the prediction results of the fine-grained learning branch;

[0013] S6, under the guidance of the branch curriculum mechanism, the model first focuses on the learning of the coarse-grained branch and then gradually shifts the focus to the fine-grained learning branch, and the overall loss of the model is calculated.

[0014] As an optional technical scheme, in S1, the input of the target detector is an image, and the output is visual features V, spatial features B and semantic features L of instances in the image, and then the features are input into the coarse-grained learning branch and the fine-grained learning branch.

[0015] As an optional technical scheme, in S2, the coarse-grained learning branch uses a general scene graph model to encode context information for the three visual features, to enhance the features of each instance feature, and the obtained instance features with context are used for relationship predicate classification, and the formula is as follows:

[0016] C c =Extractor c(B, V, L)

[0017] Z c = Rel_Dec c (C c , U, W bias )

[0018] where Extractor c (·, ·, ·) is the coarse-grained relation feature extractor, C c is the coarse-grained instance feature with context information, Rel_Dec c (·, ·, ·) is the coarse-grained relation decoder for relation predicate classification, U is the joint region feature, W bias is the statistical prior bias of the training dataset, Z c is the predicted coarse-grained predicate logic;

[0019] The fine-grained learning branch uses the same scene graph model as the coarse-grained learning branch to encode the context information, while also using a semantic context module to correct unreasonable predictions, which is expressed as follows:

[0020] C f = Extractor f (B, V, L)

[0021] Z f = Rel_Dec f (C f , U, W bias )

[0022] Z' = SCM(Z f , L)

[0023] Z o = Z' + Z f

[0024] where Extractor f (·, ·, ·) is the fine-grained relation feature extractor, C f is the fine-grained instance feature with context information, Rel_Dec f (·, ·, ·) is the fine-grained relation decoder for relation predicate classification, Z f is the predicted coarse-grained predicate logic, SCM(·, ·) is our semantic context module, Z' is the predicate correction logic output by the semantic context module; Zo is the corrected predicate logic used for final output, it should be noted that Extractor f (·, ·, ·) and Extractor c(·,·,·) are parameters that are shared, so the fine-grained learning branch can share the robust predicate features learned by the coarse-grained learning branch.

[0025] As an optional technical solution, the knowledge distillation in S3 has the following formula:

[0026]

[0027] Among them, L KD It is the distillation loss function, N R It is the total number of categories of relational predicates, p T and q T It's about predicate logic Z. c and Z o The soft probability distribution, log() is the logarithmic function. and They are p T and q T The i-th element in the array is calculated as follows:

[0028]

[0029]

[0030] Where τ is a temperature hyperparameter. and It is predicate logic Z c and Z o The i-th element, exp() is an exponential function. It should be noted that knowledge distillation is only performed between the head predicates.

[0031] As an optional technical solution, the semantic context module in S4 specifically includes:

[0032] First, the predicate logic predicted by the model is processed to obtain the predicate probability distribution. Then, a pre-trained word embedding model (GloVe) is used to map each predicate probability distribution and the probability distribution of the instance into a 200-dimensional vector, thereby obtaining the predicate semantic representation s. p and instance semantic representations o Then, the semantic representation of the predicate is concatenated with the semantic representations of its corresponding subject and object to obtain the semantic representation of the relation triple:

[0033]

[0034] in, and It is the semantic representation of the i-th predicate predicted in the current image. The relevant semantic representations of the subject and the semantic representation of the object, is the semantic representation of the i-th relation triple predicted in the current image, and W is a learnable linear mapping that maps the concatenated triple semantic representation to D dimensions. In addition, a global representation s is added. global As a semantic representation of the entire scene graph, the calculation method is as follows:

[0035]

[0036] Where N is the total number of triples in an image. The same operation is performed on the ground truth labeled scene graph of the current image to obtain the semantic features of the relation triples in the ground truth labeled image. and scene graph semantic representation t global .

[0037] Based on the above preparations, a regular Transformer Encoder is used to construct the contextual semantic representation, whose input is a set of all relation triples in an image and the semantic representation of the entire scene graph. The calculation is as follows:

[0038] S r ′=Trans_Enc(S r )

[0039] Where Trans_Ens(·) is the Transformer Encoder

[0040] It is a set of relation triples with contextual semantics and the semantic representation of the entire scene graph. The same operation is used to obtain the semantic contextual representation T of the ground truth labeled scene graph of the current image. r ′.

[0041] Using s′ global and t′ global Calculate the semantic gap between the generated scene graph and the ground truth labeled scene graph, and minimize it using mean squared error loss:

[0042]

[0043] Among them, L SC It is the scene graph semantic loss function, ||·|| is the vector L2 norm, and the triplet semantic context representation. Predicate classification will be performed to obtain the corrected predicate logic Z′. Then, the corrected predicate logic Z′ will be compared with the predicate logic Z predicted by the fine-grained learning branch. f Addition is used to correct some unreasonable predictions:

[0044] Z o =Z′+Z f

[0045] Z o This is the final predicate logic, used for output.

[0046] As an optional technical solution, the course reweighting mechanism in S5 specifically includes:

[0047] Optimize fine-grained learning branches using a course reweighting mechanism:

[0048]

[0049] Among them, L CRM This is the loss function for the course reweighting mechanism, where exp() is the exponential function, log() is the logarithmic function, Z is the predicted predicate logic, y is its corresponding label, and λ is the loss function. i It is the trade-off factor for predicate category i, y i z is the value of the i-th category in the label. i w is the value of the i-th element in predicate logic. i The weights of predicate category i are calculated using a reweighting method, where the weighting factor λ i The definition is as follows:

[0050]

[0051] Here, max() is the maximum value function, H is the head predicate index set, and those with more than 10,000 samples are considered head predicates. It is a scheduling function that decreases from 1 to 0, representing the learning "attention" weight assigned to the head predicate. To avoid forgetting the head predicate, a threshold hyperparameter β1∈[0,1] is used to avoid assigning a weight of 0 to the head predicate. The definition is as follows:

[0052]

[0053] Where k is the current number of training iterations, K is the total number of iterations, and K1 is the intermediate iteration number.

[0054] As an optional technical solution, the branching course mechanism in S6 specifically includes:

[0055] The branching course mechanism uses a tradeoff factor α to adjust the learning weights of coarse-grained and fine-grained learning branches, defined as follows:

[0056]

[0057] Where K1 and K2 are intermediate iteration numbers, k is the current training iteration number, and β2∈[0,1] is a threshold hyperparameter to prevent assigning 0 weights to coarse-grained learning branches. It is another scheduling function, defined as follows:

[0058]

[0059] Under the adjusted branch course mechanism, the joint loss function is calculated as follows:

[0060] L hybrid =α·L CE +(1-α)·L CRM

[0061] Among them, L CE It is the cross-entropy loss function used in coarse-grained branching, L CRM It is the loss function of the course reweighting mechanism used in fine-grained branching.

[0062] The final overall loss function of the model is as follows:

[0063] L total =L hybrid +L SC +μ·L KD

[0064] Here, μ is a balance factor.

[0065] The beneficial effects of this invention are as follows:

[0066] 1. A novel dual-branch hybrid learning network structure is proposed to generate unbiased scene graphs. It can simultaneously take into account the learning of head coarse-grained predicates and tail fine-grained predicates, achieving advanced tail predicate recognition performance while maintaining good head predicate recognition performance.

[0067] 2. A course reweighting mechanism is proposed, which first learns simple head predicates that can provide general recognition patterns, and then gradually focuses on learning fine-grained tail predicates; a semantic context module is proposed, which can explore and utilize the correlation between relation triples in the image to correct some unreasonable predictions. Attached Figure Description

[0068] Figure 1 This is a flowchart of an unbiased scene graph generation method based on a dual-branch hybrid learning network;

[0069] Figure 2 This is a general framework diagram of an unbiased scene graph generation method based on a dual-branch hybrid learning network. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0071] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0072] refer to Figure 1 , 2 The flowchart and overall framework diagram of an unbiased scene graph generation method based on a dual-branch hybrid learning network are shown respectively. This method includes steps S1 to S6. Figure 2 As shown, the main functional modules are divided into coarse-grained learning branch, fine-grained learning branch, semantic context module, and course reweighting mechanism.

[0073] In one embodiment of the present invention, steps S1 and S2, specifically include the execution steps of the coarse-grained learning branch and the fine-grained learning branch:

[0074] A1. The object detector takes an image as input and outputs visual features V, spatial features B, and semantic features L of the instances in the image. These features are then fed into coarse-grained learning branches and fine-grained learning branches.

[0075] A2. The coarse-grained learning branch uses a general scene graph model to encode contextual information for these three visual features, thereby enhancing the features of each instance. The resulting context-rich instance features are used for relational predicate classification, as shown in the following formula:

[0076] C c =Extractor c (B, V, L)

[0077] Z c =Rel_Dec c (C c U, W bias )

[0078] Among them, Extractor c (·,·,·) is a coarse-grained relation feature extractor, C cIt is a coarse-grained instance feature with contextual information, Rel_Dec c (·,·,·) is a coarse-grained relation decoder used for relational predicate classification, U is the joint region feature, and W is the joint region feature. bias Z is the statistical prior bias of the training dataset. c It is a coarse-grained predicate logic for prediction.

[0079] A3. The fine-grained learning branch uses the same scene graph model as the coarse-grained learning branch to encode contextual information. It also uses a semantic context module to correct unreasonable predictions, as shown in the following formula:

[0080] C f =Extractor f (B, V, L)

[0081] Z f =Rel_Dec f (C f U, W bias )

[0082] Z′=SCM(Z f L)

[0083] Z o =Z′+Z f

[0084] Among them, Extractor f (·,·,·) is a fine-grained relation feature extractor, C f It is a fine-grained instance feature with contextual information, Rel_Dec f (·,·,·) is a fine-grained relation decoder used for relational predicate classification, Z f This refers to the coarse-grained predicate logic for prediction. SCM(·,·) is our semantic context module, Z′ is the predicate correction logic output by the semantic context module, and Zo is the corrected predicate logic used for the final output. It's important to note that Extractor... f (·,·,·) and Extractor c (·,·,·) are parameters that are shared, so the fine-grained learning branch can share the robust predicate features learned by the coarse-grained learning branch.

[0085] In one embodiment of the present invention, step S3 describes the use of knowledge distillation to distill the expert knowledge of the head predicate of the coarse-grained learning branch to the fine-grained learning branch, as shown in the following formula:

[0086]

[0087] Among them, L KDIt is the distillation loss function, N R It is the total number of categories of relational predicates, p T and q T It's about predicate logic Z. c and Z o The soft probability distribution, log() is the logarithmic function. and They are p T and q T The i-th element in the array is calculated as follows:

[0088]

[0089]

[0090] Where τ is a temperature hyperparameter. and It is predicate logic Z c and Z o The i-th element, exp() is an exponential function that performs knowledge distillation only between the head predicates.

[0091] In one embodiment of the present invention, step S4, in which the semantic context module performs the following steps:

[0092] A1. First, the predicate logic predicted by the model is processed to obtain the predicate probability distribution. Then, a pre-trained word embedding model (GloVe) is used to map each predicate probability distribution and the probability distribution of the instance into a 200-dimensional vector, thereby obtaining the predicate semantic representation s. p and instance semantic representations o Then, the semantic representation of the predicate is concatenated with the semantic representations of its corresponding subject and object to obtain the semantic representation of the relation triple:

[0093]

[0094] in, and It is the semantic representation of the i-th predicate predicted in the current image. The relevant semantic representations of the subject and the semantic representation of the object, is the semantic representation of the i-th relation triple predicted in the current image, and W is a learnable linear mapping that maps the concatenated triple semantic representation to D dimensions. In addition, a global representation s is added. global As a semantic representation of the entire scene graph, the calculation method is as follows:

[0095]

[0096] Where N is the total number of triples in an image. The same operation is performed on the ground truth labeled scene graph of the current image to obtain the semantic features of the relation triples in the ground truth labeled image. and scene graph semantic representation t global .

[0097] A2. Based on the above preparations, a regular Transformer Encoder is used to construct the contextual semantic representation. Its input is a set of all relation triples in an image and the semantic representation of the entire scene graph. The calculation is as follows:

[0098] S r ′=Trans_Enc(S r )

[0099] Where Trans_Enc(·) is the Transformer Encoder,

[0100] It is a set of relation triples with contextual semantics and the semantic representation of the entire scene graph. The same operation is used to obtain the semantic contextual representation T of the ground truth labeled scene graph for the current image. r ′.

[0101] A3. Using s′ global and t′ global Calculate the semantic gap between the generated scene graph and the ground truth labeled scene graph, and minimize it using mean squared error loss:

[0102]

[0103] Among them, L SC It is the scene graph semantic loss function, ||·|| is the vector L2 norm, and the triplet semantic context representation. Predicate classification will be performed to obtain the corrected predicate logic Z′. Then, the corrected predicate logic Z′ will be compared with the predicate logic Z predicted by the fine-grained learning branch. f Addition is used to correct some unreasonable predictions:

[0104] Z o =Z′+Z f

[0105] Z o This is the final predicate logic, used for output.

[0106] In one embodiment of the present invention, the specific description of the course reweighting mechanism in step S5 is as follows:

[0107]

[0108] Among them, L CRM This is the loss function for the course reweighting mechanism, where exp() is the exponential function, log() is the logarithmic function, Z is the predicted predicate logic, y is its corresponding label, and λ is the loss function. i It is the trade-off factor for predicate category i, y i z is the value of the i-th category in the label. i w is the value of the i-th element in predicate logic. i The weights of predicate category i are calculated using a reweighting method, where the weighting factor λ i The definition is as follows:

[0109]

[0110] Here, max() is the maximum value function, and H is the head predicate index set. Predicates with more than 10,000 samples in the training set are considered head predicates. It is a scheduling function that decreases from 1 to 0, representing the learning "attention" weight assigned to the head predicate. To avoid forgetting the head predicate, a threshold hyperparameter β1∈[0,1] is used to avoid assigning a weight of 0 to the head predicate. The definition is as follows:

[0111]

[0112] Where k is the current number of training iterations, K is the total number of iterations, and K1 is the intermediate iteration number.

[0113] In one embodiment of the present invention, step S6, the execution of the branch course mechanism and the calculation of the total loss, specifically includes:

[0114] A1. The branching course mechanism uses a tradeoff factor α to adjust the learning weights of coarse-grained learning branches and fine-grained learning branches, defined as follows:

[0115]

[0116] Where K1 and K2 are intermediate iteration numbers, k is the current training iteration number, and β2∈[0,1] is a threshold hyperparameter to prevent assigning 0 weights to coarse-grained learning branches. It is another scheduling function, defined as follows:

[0117]

[0118] A2. Under the adjustment of the branch course mechanism, the joint loss function is calculated as follows:

[0119] L hybrid =α·L CE+(1-α)·L CRM

[0120] Among them, L CE It is the cross-entropy loss function used in coarse-grained branching, L CRM It is the loss function of the course reweighting mechanism used in fine-grained branching.

[0121] The final overall loss function of the model is as follows:

[0122] L total =L hybrid +L SC +μ·L KD

[0123] Here, μ is a balance factor.

[0124] The following section compares the effectiveness of this solution with existing scene graph generation methods through simulation experiments:

[0125] The experimental conditions were as follows: System: Ubuntu 20.04, Software: Python 3.7, Processor: Intel(R) Xeon(R) CPU E5-2678 v3@2.50GHz×2, Memory: 256GB.

[0126] Test Description: Each algorithm is used sequentially to train the training set in the dataset. After training, each algorithm is used to predict the images on the test set of the dataset to obtain the predicted scene map. The specific test of the image scene map generation task includes three sub-tasks:

[0127] 1) Predicate classification: Given bounding boxes and instance labels of all instances in an image, then detect and classify the relationships between instances;

[0128] 2) Scene Graph Classification: Given the bounding boxes of all instances in an image, the instances need to be classified and the relationships between them need to be detected.

[0129] 3) Scene Graph Detection: Given an image, predict a scene graph from scratch.

[0130] The evaluation metrics used include Recall@K (R@K) and mean Recall@K (mR@K). Recall@K refers to the proportion of correct predicates among the first K predicted relation predicates; mean Recall@K is the average Recall@K for each relation predicate. For both metrics, higher values ​​indicate better performance. The test results are shown in the table below:

[0131] Table 1

[0132]

[0133] Table 2

[0134]

[0135] By comparing the experimental results in Tables 1 and 2, it can be seen that the proposed method performs better when using the same general scene graph generation method. These results verify the effectiveness of our proposed method.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating unbiased scene graphs based on a dual-branch hybrid learning network, characterized in that, Includes the following steps: S1. Input the image into the target detector to obtain the visual features, semantic features, and spatial features of the instances in the image; S2. The features of instances in the image are fed into the coarse-grained learning branch and the fine-grained learning branch. The coarse-grained learning branch is used to learn expert knowledge of the head predicate and obtain robust features of the head predicate. The fine-grained learning branch is used to predict informative tail predicates. S3. Knowledge distillation: Distill the expert knowledge in the head predicate of the coarse-grained learning branch into the fine-grained learning branch. S4. The semantic context module is used to correct unreasonable predictions in the prediction results of the fine-grained learning branch. At the same time, the difference between the semantics of the predicted scene graph and the semantics of the labeled scene graph is calculated and optimized using mean squared error loss. S5. Use the standard cross-entropy loss function to optimize the prediction results of the coarse-grained learning branch, and use the course reweighting mechanism to optimize the prediction results of the fine-grained learning branch. S6. Guided by the branching course mechanism, the model will first focus on learning coarse-grained branches and then gradually shift its learning focus to fine-grained branches, and calculate the overall loss of the model.

2. The method for generating unbiased scene graphs based on a dual-branch hybrid learning network according to claim 1, characterized in that, In S1, the input to the target detector is an image, and the output is the visual features V, spatial features B, and semantic features L of the instances in the image. These features are then input into the coarse-grained learning branch and the fine-grained learning branch.

3. The method for generating unbiased scene graphs based on a dual-branch hybrid learning network according to claim 2, characterized in that, In S2, The coarse-grained learning branch uses a general scene graph model to encode contextual information for these three visual features, thereby enhancing the features of each instance. The resulting context-rich instance features are used for relational predicate classification, as shown in the following formula: C c =Extractor c (B,V,L) Z c =Rel_Dec c (C c ,U,W bias ) Among them, Extractor c (,…,) is a coarse-grained relation feature extractor, C c It is a coarse-grained instance feature with contextual information, Rel_Dec c (·,·,·) is a coarse-grained relation decoder used for relational predicate classification, U is the joint region feature, and W is the joint region feature. bias Z is the statistical prior bias of the training dataset. c It is a coarse-grained predicate logic for prediction; The fine-grained learning branch uses the same scene graph model as the coarse-grained learning branch to encode contextual information, and also uses a semantic context module to correct unreasonable predictions, as shown in the following formula: C f =Extractor f (B,V,L) Z f =Rel_Dec f (C f ,U,W bias ) Z′=SCM(Z f ,L) WITH o =Z′+Z f Among them, Extractor f (,…,) is a fine-grained relation feature extractor, C f It is a fine-grained instance feature with contextual information, Rel_Dec f (·,·,·) is a fine-grained relation decoder used for relational predicate classification, Z f This is the coarse-grained predicate logic for prediction. SCM(·,·) is our semantic context module, Z′ is the predicate correction logic output by the semantic context module, and Zo is the corrected predicate logic used for the final output. Note that Extractor... f (·,·,·) and Extractor c (·,·,·) are parameters that are shared, so the fine-grained learning branch can share the robust predicate features learned by the coarse-grained learning branch.

4. The method for generating unbiased scene graphs based on a dual-branch hybrid learning network according to claim 1, characterized in that, The knowledge distillation in S3 is represented by the following formula: Among them, L KD It is the distillation loss function, N R It is the total number of categories of relational predicates, p T and q T It's about predicate logic Z. c and Z o The soft probability distribution, log() is the logarithmic function. and They are p T and q T The i-th element in the array is calculated as follows: Where τ is a temperature hyperparameter. and It is predicate logic Z c and Z o The i-th element, exp() is an exponential function. It should be noted that knowledge distillation is only performed between the head predicates.

5. The method for generating unbiased scene graphs based on a dual-branch hybrid learning network according to claim 1, characterized in that, The semantic context module in S4 specifically includes: First, the predicate logic predicted by the model is processed to obtain the predicate probability distribution. Then, a pre-trained word embedding model, GloVe, is used to map each predicate probability distribution and the probability distribution of the instance into a 200-dimensional vector, thereby obtaining the predicate semantic representation s. p and instance semantic representations o Then, the semantic representation of the predicate is concatenated with the semantic representations of its corresponding subject and object to obtain the semantic representation of the relation triple: in, and It is the semantic representation of the i-th predicate predicted in the current image. The relevant semantic representations of the subject and the semantic representation of the object, Let W be the semantic representation of the i-th relation triple predicted in the current image. W is a learnable linear mapping that maps the concatenated triple semantic representation to D dimensions. In addition, a global representation s is added. global As a semantic representation of the entire scene graph, the calculation method is as follows: Where N is the total number of triples in an image. The same operation is performed on the ground truth labeled scene graph of the current image to obtain the semantic features of the relation triples in the ground truth labeled image. and scene graph semantic representation t global , Based on the above preparations, a regular Transformer Encoder is used to construct the contextual semantic representation, whose input is a set of all relation triples in an image and the semantic representation of the entire scene graph. The calculation is as follows: S r′ =Trans_Enc(S′) Where Trans_Enc(·) is the Transformer Encoder, It is a set of relation triples with contextual semantics and the semantic representation of the entire scene graph. The same operation is used to obtain the semantic contextual representation T of the ground truth labeled scene graph of the current image. r′ , Using s′ global and t′ global Calculate the semantic gap between the generated scene graph and the ground truth labeled scene graph, and minimize it using mean squared error loss: Among them, L SC It is the scene graph semantic loss function, ||·|| is the vector L2 norm, and the triplet semantic context representation. Predicate classification will be performed to obtain the corrected predicate logic Z′. Then, the corrected predicate logic Z′ will be compared with the predicate logic Z predicted by the fine-grained learning branch. f Addition is used to correct some unreasonable predictions: WITH o =Z′+Z f Z o This is the final predicate logic, used for output.

6. The method for generating unbiased scene graphs based on a dual-branch hybrid learning network according to claim 1, characterized in that, The course reweighting mechanism in S5 specifically includes: Optimize fine-grained learning branches using a course reweighting mechanism: Among them, L CRM This is the loss function for the course reweighting mechanism, where exp() is the exponential function, log() is the logarithmic function, Z is the predicted predicate logic, y is its corresponding label, and λ is the loss function. i It is the trade-off factor for predicate category i, y i z is the value of the i-th category in the label. i w is the value of the i-th element in predicate logic. i The weights of predicate category i are calculated using a reweighting method, where the weighting factor λ i The definition is as follows: Here, max() is the maximum value function, and H is the set of head predicate indices, where predicates with more than 10,000 samples are considered head predicates. It is a scheduling function that decreases from 1 to 0, representing the learning "attention" weights assigned to the head predicate. To avoid forgetting the head predicate, a threshold hyperparameter β1∈[0,1] is used to prevent assigning a weight of 0 to the head predicate. The definition is as follows: Where k is the current number of training iterations, K is the total number of iterations, and K1 is the intermediate iteration number.

7. The method for generating unbiased scene graphs based on a dual-branch hybrid learning network according to claim 1, characterized in that, The branched course mechanism in S6 specifically includes: The branching course mechanism uses a tradeoff factor α to adjust the learning weights of coarse-grained and fine-grained learning branches, defined as follows: Where K1 and K2 are intermediate iteration numbers, k is the current training iteration number, and β2∈[0,1] is a threshold hyperparameter to prevent assigning 0 weights to coarse-grained learning branches. It is another scheduling function, defined as follows: Under the adjusted branch course mechanism, the joint loss function is calculated as follows: L hybrid =α·L CE +(1-a)·L CRM Among them, L CE It is the cross-entropy loss function used in coarse-grained branching, L CRM It is the loss function of the course reweighting mechanism used in fine-grained branching; The final overall loss function of the model is as follows: L total =L hybrid +L SC +μ·L KD Here, μ is a balance factor.

Images