A method for generating an unbiased scene graph based on candidate predicate relationship bias

Abstract

This invention, a method for generating unbiased scene graphs based on candidate predicate relation bias, belongs to the field of computer vision understanding. This invention utilizes the similarity relationship between candidate predicates to adjust the model's balance between head and tail predicates, enabling the differentiation of semantically similar predicates on top of previous methods for generating biased scene graphs. Significant results have been achieved, controlling the recall rate from a large drop while effectively improving the average recall rate. This results in a scene graph generation model that obtains richer and more granular semantic information than previous methods, surpassing some current mainstream unbiased scene graph generation models. It has wide applications in practical applications and can fully demonstrate its value in a series of downstream tasks such as visual question answering, image segmentation, and image retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision understanding, and particularly discloses an unbiased scene graph generation method based on candidate predicate relation bias. BACKGROUND

[0002] The ultimate goal of computer vision (CV) is to build an intelligent system that can extract valuable information from digital images, videos or other modalities as humans do. In the past few decades, machine learning (ML) has made significant contributions to the progress of CV. Inspired by the fact that humans can effortlessly interpret and understand visual scenes, visual scene understanding has been advocated as the holy grail of CV and has attracted extensive attention from the research community.

[0003] Scene graph generation (SGG) is a computational task of scene understanding that aims to generate a graphical structure representing a scene by analyzing the content of an image and the relationships between objects in the scene. Commonly used scene graph generation methods include using deep learning models, image segmentation techniques, ensemble models and knowledge-based methods. Deep learning models are a type of machine learning method based on neural networks that can capture detailed semantics by explicitly modeling objects, object attributes and relationships between pairs of objects. These models usually include an object detection module and a relationship classification module to identify objects from the input image and determine their relationships. Image segmentation is an image processing technique that can divide an image into multiple parts or regions. These regions can represent different objects in the image, which can then be used to generate a scene graph using the objects and their relationships. Ensemble models combine multiple scene graph generation methods to improve the accuracy and robustness of the generated scene graphs. For example, a deep learning model can be combined with image segmentation techniques to improve the accuracy of object detection and use a relationship classification module to identify relationships between objects. Knowledge-based methods use manually defined knowledge bases to generate scene graphs. These knowledge bases include objects and their attributes in the image, as well as relationships between objects. These knowledge can then be used to generate a scene graph. These methods can segment the input image into objects and identify relationships between them, and then use a scene graph generation model to convert these objects and relationships into a graphical structure. For example, an image content is a person wearing a helmet holding a mobile phone surfing on a skateboard, and a scene graph generation model can get "person wearing helmet", "person on skateboard", "person holding mobile phone", etc.

[0004] Unbiased SGG is also a method of scene graph generation, aiming to predict more informative scene graphs composed of more "tail predicates" by handling skewed, long-tail predicate class distribution. In scene graph generation, bias often occurs, which may reflect the social bias and stereotypes present in its training data, such as labeling women as chefs and men as engineers. Therefore, unbiased scene graph generation can help scene graph generation to better apply to reality by reducing biased information.

[0005] Unbiased scene graph generation also has a wide range of application scenarios in many practical applications, such as image description, image question answering, visual reasoning, etc. It can help computers better understand image content, so as to more accurately answer questions, improve visual reasoning ability, and provide better image search and identification services for people. In addition, it also helps the construction of new smart cities, which can improve the retrieval efficiency and computing power of infrastructure, such as in smart transportation, unbiased scene graph generation can generate real-time scene graphs of the city through data sources such as city traffic monitoring cameras and sensors, helping traffic management departments better understand the city's traffic situation, so as to better manage and dispatch traffic, improve traffic efficiency and safety. For example, scene graph generation technology can be used to simulate city traffic flow, congestion, accident information, etc. to help traffic management departments better understand the city's situation and problems. So that people, cars and roads can work closely together to improve the transportation environment and improve resource utilization. In smart security, unbiased scene graph generation can generate real-time scene graphs of the city through data sources such as city security monitoring cameras and sensors, helping security departments better understand the city's security situation, so as to better manage and prevent security, improve the city's security. For example, scene graph generation technology can be used to simulate city security monitoring, alarm response, etc. information to help security departments better understand the city's situation and problems. In smart education, it helps children understand more useful and learnable information in images. Moreover, compared to scene graph generation, unbiased scene graph generation can get more rich semantic information to make users' experience better in practice, not just limited to a few common predicates.

[0006] In summary, the unbiased scene graph generation is very meaningful and promising, the candidate relation predicate is the remaining possible relation category except the highest scoring relation category proposed by the patent, the patent solves the existing problems by using the candidate relation predicate bias to improve the effect of unbiased scene graph generation in various important indicators, and finally completes an unbiased scene graph generation system with higher efficiency and richer semantics. The problems existing in the process of the unbiased scene graph generation at present include that some models only fit the tail predicate by re-weighting and re-sampling, so that many over-fitting phenomena occur; the model cannot effectively distinguish the predicates that are not mutually exclusive in semantics, so that more false and coarse-grained predicates appear, and the effect of unbiased scene graph generation is low. SUMMARY

[0007] In order to overcome the problems existing in the above-mentioned technology, the model can effectively distinguish the predicates with similar semantics, and then generate the relation predicates containing more rich semantic information. The purpose of the present application is to propose a method for generating unbiased scene graph based on candidate predicate relation bias.

[0008] The technical scheme adopted by the present application is: a method for generating unbiased scene graph based on candidate predicate relation bias, comprising the following steps:

[0009] Step 1, selecting pictures in the data set for preprocessing and dividing the data set;

[0010] Step 2, using the target detection network (Faster RCNN) that has been trained to obtain the target candidate box (Object proposals) and the related features of the picture from the data set picture after preprocessing. For any candidate box in each picture, the target detector is used to calculate its visual features g v , the bounding box coordinates b v , the classification score z v of the target entity corresponding to the candidate box, in addition, the joint features of the entities s and o corresponding to the two target candidate boxes are generated by using the backbone network C v representing the number of all possible categories of target entities;

[0011] Step 3, constructing the spatial features of the target candidate box, for each candidate box, the spatial features pos(b v ) of the candidate box are calculated by using the bounding box coordinates b v obtained in step 2;

[0012] Step 4, constructing the semantic features of the target candidate box, the entity category corresponding to the highest score in the classification score z v of the candidate box obtained in step 2 is selected as the label c of the entityv , will c v The result edb(c) is obtained by adding it to the word vector model. v As a semantic feature, edb is an unsupervised learning model, GloVe, used to obtain the vector representation of words;

[0013] Step 5: Construct a target entity encoder to obtain the contextual feature representation and entity category of the target entity. Based on the features obtained from the target detector, convert the visual features g obtained in Step 2 into... v And the spatial features pos(b) obtained in step 3 v ) and the semantic features edb(c) obtained in step 4 v Perform a concatenation operation and input the obtained target features into n. o The final target feature representation is obtained from the target encoder. Will The input to the target classifier and activation function is the predicted score p of the object category to which the target candidate box belongs. v ;

[0014] Step 6: Construct an entity relation encoder to obtain the feature representation of the relation context and the relation category. For each directed pair (s,o) in an image, first, the final feature representation of entity s obtained in Step 5 is... The final feature representation of entity o and the joint feature representation of entities s and o obtained in step 2. The initial relational feature representation is obtained by concatenation, and then the dimension of the vector is transformed through a linear transformation layer. The transformed relational feature representation is then input into n in step 5. o The target encoder is constructed with the same n r In the relation encoder, the output yields the final relation representation features of entities s and o. Finally, the final relation representation features are input into the relation classifier to obtain the relation classification score z. s,o Classify the relation by score z s,o The input is fed into the activation function to obtain the prediction score p of the relationship between the two entity objects. s,o ;

[0015] Step 7: Construct candidate predicate relation biases, use these biases to adjust the model strength under different entity relations, and calculate the relation classification score z obtained in Step 6. s,o The top m pairs of entities with the highest relationship scores r Construct a candidate predicate matrix m based on the occurrence counts of each candidate predicate. s,o Using the candidate predicate matrix m s,o Calculate the deviation b of the generated candidate predicate relation;

[0016] Step 8, construct the loss function, use the relationship classification score z in step 6 s,o Subtract the predicate relationship bias b in step 7 and input the activation function to get the new entity-object relationship prediction score And the prediction score p of the object category obtained in step 5 v And the entity-object relationship prediction score Respectively and the corresponding label vector are input into the cross-entropy loss function to calculate the respective loss values;

[0017] Step 9, train the model, according to the loss values of the entity and relationship corresponding to step 8, use the back propagation algorithm to perform gradient back propagation on the parameters of the target entity encoder and relationship encoder constructed in steps 5 and 6, and then continuously optimize until the entire network model converges;

[0018] Step 10, entity and relationship prediction value calculation, sort the classification score vector of each candidate box, and select the category corresponding to the highest score as the predicted category; sort the classification score vector of the relationship between the candidate boxes, and select the relationship category corresponding to the highest score.

[0019] Further, the specific implementation of step 1 is as follows: select VG (Visual Genome) as the data set for model training, a total of 108K pictures, each picture has an average of 35 target entities and 21 relationships, and the patent uses the most common 150 target categories and 50 relationship categories. First, remove some low-quality, weak-reference data set pictures, and apply 70% of the pictures in the data set to training, and the remaining 30% of the pictures are used for testing.

[0020] Further, the extraction of target candidate boxes and related features from the picture described in step 2 is as follows: use the output results of the RPN part of the already trained target detection network as the target candidate box and the visual features g corresponding to the candidate box v , the coordinates b of the candidate box v , the target entity classification score z of the candidate box v , and use the results obtained by the backbone network in the target detection network as the joint features of the entity pair

[0021] Further, the construction of the spatial features of the target candidate box described in step 3 is as follows: use the obtained boundary box position coordinates b v (x min , y min , x max , y max ) to calculate, where (x min , ymin ) as the coordinates of the top-left corner of the target bounding box, (x max , y max ) as the coordinates of the bottom-right corner of the target bounding box, the spatial features of the target bounding box are obtained as follows:

[0022]

[0023] where W and H represent the width and height of the corresponding picture, and w and h represent the width and height of the corresponding target bounding box.

[0024] Further, the construction of the semantic features of the candidate box in step 4 is as follows: from the classification scores of the target entity corresponding to the obtained candidate box , the entity category c v corresponding to the highest score is selected, and c v is input into the trained GloVe word vector model to obtain the semantic features of the candidate box, and the specific formula is as follows:

[0025] edb(c v ) = GloVe(c v ) (Formula 2)

[0026] Further, the construction of the target entity encoder in step 5 obtains the context feature representation of the target entity and the entity category as follows:

[0027] 5-1. Concatenate the visual features, spatial features, and semantic features and linearly change the dimensions of the results

[0028] After linear transformation, the visual, spatial, and semantic features obtained by the target detector have the same dimensions, because the three features can be directly operated. The obtained visual feature g v , spatial feature pos(b v ), and semantic feature edb(c v ) are directly concatenated, and then the concatenated features are input into the linear transformation layer W o to obtain the initial feature e v of the target entity. The specific formula is as follows:

[0029] e v = W o [pos(b v ), g v , edb(c v )] (Formula 3)

[0030] 5-2. Input the target entity encoder to obtain the final feature representation of the entity

[0031] The initial target entity feature e vInput to n o In the target entity encoder, n o is set to 4, representing e v The final feature representation of the entity is obtained by iterating the 4-layer target entity encoder Wherein, the target entity encoder mainly uses self-attention mechanism to represent the features, mainly including attention network and feedforward network, and the specific formula of attention matrix calculation is as follows:

[0032]

[0033] Wherein, Q, K, V represent the input features e v The values obtained by three different linear transformation layers, is the scale factor of Q, K dot product, and σ is the activation function;

[0034] Wherein, the attention network further includes using multi-head attention to divide Q, K, V into n h Parts, calculate the attention of each part, and then use the feedforward network to fuse the attention of each part to obtain new features, and iterate n o Target entity encoders to obtain the final target entity feature representation

[0035] 5-3. Through the classifier and the activation function to obtain the prediction probability score p of the belonging object class v

[0036] The specific formula is as follows:

[0037]

[0038] Wherein Represents the target entity classifier, and σ represents the activation function.

[0039] Further, the construction of the inter-entity relationship encoder in step 6 to obtain the feature representation of the relationship context and the relationship class is as follows:

[0040] 6-1. Splice the final features of entities s and o and the joint features of the two, and linearly change the results

[0041] The final features of entities s and o And the joint features The dimensions after linear transformation are the same, so they can be spliced directly, and then the results are input into the linear transformation layer W r To obtain the initial feature e s,o Of the inter-entity relationship, and the calculation process is as follows:

[0042]

[0043] 6-2. Input relation encoder gets the final feature representation of inter-entity relation

[0044] The obtained inter-entity relation initialization feature e s,o is input into n r relation encoders, n r is set to 2, representing e s,o The final inter-entity relation feature is obtained by iterating 2 relation encoders

[0045] 6-3. The classification score z of inter-entity relation is obtained through the classifier and the activation function s,o and the category prediction score p s,o , the specific formula is as follows:

[0046]

[0047] p s,o = σ(z s,o ) (formula 8)

[0048] wherein represents the relation classifier, and σ represents the activation function.

[0049] Further, the construction of the candidate predicate relation bias in step 7 is specifically as follows:

[0050] 7-1. Calculate the predicate similarity matrix

[0051] First, construct m s,o , which is initially a 51*51 unit matrix, corresponding to the similarity between each pair of the 50 relation categories, z s,o is an n*51 score matrix, n represents all possible entity relation pairs in a batch. The scores corresponding to the candidate predicates of each pair of entities in z s,o are sorted, so that the highest score corresponds to the predicate and the predicate in the top m r establishes a relationship, and the number of times this relationship appears in each pair of candidate predicates is added to the predicate similarity matrix m s,o , m r is set to 5. For example, the Top5 of the relation score of a pair of entities in the z s,o matrix correspond to the relation categories A, B, C, D, and E, respectively. In the predicate similarity matrix m s,o , (A, B), (A, C), (A, D), and (A, E) are added by 1, respectively. The Top5 of the relation score of n pairs of entities in the z s,o matrix are all executed in the above manner, thereby obtaining the matrix m s,o used to represent the predicate similarity.

[0052] 7-2. Calculate candidate predicate relation bias

[0053] Using the obtained predicate similarity matrix m s,o The candidate predicate relation bias between each pair of entities is constructed, and the specific formula is as follows:

[0054]

[0055] Wherein r, e represents the defined hyperparameter, C r Indicates all entity relation categories.

[0056] The candidate relation predicate bias between all entities is spliced to obtain the predicate relation bias matrix b, and the specific formula is as follows:

[0057] b=(b1,...,b i ) (formula 10)

[0058] Further, the loss function constructed in step 8 is specifically as follows:

[0059] Subtract b from z s,o To obtain the score matrix after adding the relation bias, the score matrix is input into the activation function to obtain the final relation category prediction score The specific formula is as follows:

[0060]

[0061] Therefore, the specific formula of the relation loss function calculation is as follows:

[0062]

[0063] The specific formula of the entity loss function calculation is as follows:

[0064]

[0065] The specific formula of the total loss function calculation is as follows:

[0066]

[0067] The beneficial effects of the present application are as follows:

[0068] The application proposes a new unbiased scene graph generation method, which uses the relationship between the similarity of candidate predicates to adjust the balance of head predicates and tail predicates in the relationship regulation model, so that it can distinguish predicates with similar semantics on the basis of generating unbiased scene graphs in the past, and has achieved remarkable results. On the one hand, the recall rate does not decrease greatly, on the other hand, the average recall rate is effectively improved, so that the scene graph generation model obtains more rich and more fine-grained semantic information than before, surpasses some mainstream unbiased scene graph generation models at present, and has a wide range of uses in practical applications. It can fully play a value in a series of downstream tasks such as visual question answering, picture segmentation and picture retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 The network architecture diagram of the unbiased scene graph generation method based on the relationship deviation of candidate predicates is shown in the figure.

[0070] Figure 2 The flowchart of the unbiased scene graph generation method based on the relationship deviation of candidate predicates is shown in the figure.

[0071] Figure 3 The experimental result diagram of the unbiased scene graph generation method based on the relationship deviation of candidate predicates is shown in the figure. DETAILED DESCRIPTION

[0072] As shown in the figure, the application proposes an unbiased scene graph generation method based on the relationship deviation of candidate predicates, which includes the following steps: Figure 1

[0073] The specific implementation of step 1 is as follows:

[0074] VG (Visual Genome) is selected as the data set for model training, a total of 108K pictures, each picture has an average of 35 target entities and 21 pairs of relationships. This patent uses the most common 150 target categories and 50 relationship categories. First, remove some low-quality, weak reference data set pictures, apply 70% of the pictures in the data set to training, and use the remaining 30% of the pictures for testing.

[0075] The target candidate box and related features extracted from the picture in step 2 are as follows:

[0076] The output results of the RPN part of the trained target detection network are used as target candidate boxes (Objectproposals) and visual features corresponding to the candidate boxes Boundary box coordinates b v Classification score of the target entity corresponding to the candidate box And use the results obtained by the backbone network in the target detection network as the joint features of the entity pair ​

[0077] The spatial feature of the target candidate box is constructed according to the following steps:

[0078] The spatial feature of the target candidate box is calculated using the obtained boundary box position coordinates b v (x min , y min , x max , y max ), wherein (x min , y min ) is the corresponding coordinate of the upper left corner of the target candidate box, and (x max , y max ) is the corresponding coordinate of the lower right corner of the target candidate box, so that the spatial feature of the target candidate box is wherein W and H represent the width and height of the corresponding picture, and w and h represent the width and height of the corresponding target candidate box.

[0079] The semantic feature of the target candidate box is constructed according to the following steps:

[0080] The semantic feature of the target candidate box is calculated using the obtained classification score of the target entity corresponding to the candidate box , wherein the entity category c v corresponding to the highest score is selected, and the semantic feature of the candidate box is obtained by inputting c v into the trained GloVe word vector model.

[0081] The context feature representation and the entity category of the target entity are constructed according to the following steps:

[0082] 5-1. Initial feature representation of the entity

[0083] The visual, spatial and semantic features obtained by the target detector have the same dimension after linear transformation, so the obtained visual feature g v , spatial feature pos(b v ) and semantic feature e v are directly spliced, and then the spliced features are input into the linear transformation layer W o to obtain the initial feature of the target entity.

[0084] 5-2. Final feature representation of the entity

[0085] The initial target entity feature e v is sequentially input into n o target entity encoders to obtain the final target entity feature. n o is set to 4.

[0086] 5-3. The prediction probability score of the object class to which the object belongs is obtained through the classifier and the activation function

[0087] The construction of the inter-entity relationship encoder in step 6 obtains the feature representation of the relationship context and the relationship class, which is as follows:

[0088] 6-1. Initial feature representation of the relationship

[0089] The final features of entities s and o And the joint feature The dimensions after linear transformation are the same, so they can be directly spliced, and the result is input into the linear transformation layer W r Obtain the initial feature of the inter-entity relationship

[0090] 6-2. Final feature representation of the relationship

[0091] The obtained initial feature of the relationship e s,o is sequentially input into n r relationship encoders to obtain the final inter-entity relationship feature n r is set to 2.

[0092] 6-3. Classification score z of the inter-entity relationship obtained through the classifier and the activation function s,o And the prediction score of the class to which it belongs

[0093] The construction of the candidate predicate relationship bias described in step 7 is as follows:

[0094] 7-1. Calculate the predicate similarity matrix

[0095] First, construct as an identity matrix, which corresponds to the similarity between each pair of 50 relationship classes, is a score matrix, and n represents all possible entity relationship pairs in a batch. Sort the scores corresponding to the candidate predicate of each pair of entities in z s,o , so that the predicate corresponding to the highest score and the predicates in the top m r establish a relationship, and the number of times this relationship appears in all entity pairs is added to the predicate similarity matrix m s,o , and m r is set to 5. For example, the Top5 relationship classes corresponding to the relationship score of a pair of entities in the z s,o matrix are A, B, C, D, and E, respectively. Then, in the predicate similarity matrix ms,o z s,o The Top5 of the relationship score of each pair of entities in the matrix is operated as described above, thereby obtaining a matrix m s,o .

[0096] 7-2. Calculate the candidate predicate relationship bias

[0097] In z s,o , find the label l i ∈|C r | of the highest score of the relationship between each pair of entities, and find the corresponding predicate similarity information in m s,o , as the candidate predicate relationship bias i Finally, splice the candidate predicate relationship bias b s,o between each pair of entities in z i to obtain the predicate relationship bias matrix

[0098] The construction of the loss function described in step 8 is as follows:

[0099] The score matrix with candidate predicate bias information obtained by subtracting b from z s,o is input into the activation function to obtain the final relationship class prediction score Finally, the loss is calculated using cross-entropy, so the specific formula of the total loss function calculation is as follows:

[0100]

[0101] The training model described in step 9 is as follows:

[0102] According to the loss value of the entity and the relationship calculated in step 8, the parameters of the target encoder and the relationship encoder constructed in steps 5 and 6 are gradient backpropagated using the backpropagation algorithm, and are continuously optimized until the entire network model converges.

[0103] The calculation of the entity and relationship prediction value described in step 10 is as follows:

[0104] Sort the class score vector of each candidate box, and select the class corresponding to the highest score as the target prediction class; sort the relationship class score vector between each pair of candidate boxes, and select the class corresponding to the highest score as its relationship class.

[0105] Comparative Example 1

[0106] The specific implementation of step 1 is as follows:

[0107] ​The VG (Visual Genome) is selected as the data set for model training, and there are 108K pictures, each picture has an average of 35 target entities and 21 pairs of relations. The patent uses the most common 150 target categories and 50 relationship categories. First, remove some low-quality, weak reference data set pictures, and apply 70% of the pictures in the data set to training, and the remaining 30% of the pictures are used for testing.

[0108] The target candidate box and related features extracted from the picture in step 2 are as follows:

[0109] The output results of the RPN part of the target detection network are used as target candidate boxes (Object proposals) and visual features corresponding to the candidate boxes Boundary box coordinates b v , the classification score of the target entity corresponding to the candidate box And use the results obtained by the backbone network in the target detection network as the joint features of the entity pair

[0110] The spatial features of the target candidate box constructed in step 3 are as follows:

[0111] The obtained boundary box position coordinates b v (x min , y min , x max , y max ) are calculated, where (x min , y min ) is the coordinate corresponding to the upper left of the target candidate box, (x max , y max ) is the coordinate corresponding to the lower right of the target candidate box, and the spatial features of the target candidate box obtained are where W and H represent the width and height of the corresponding picture, and w and h represent the width and height of the corresponding target candidate box.

[0112] The semantic features of the candidate box constructed in step 4 are as follows:

[0113] The classification score of the target entity corresponding to the candidate box Select the entity category c corresponding to the highest score item v , input c v into the trained GloVe word vector model to obtain the semantic features of the candidate box

[0114] The target entity encoder in step 5 obtains the context feature representation of the target entity and the entity category as follows:

[0115] 5-1. Initial feature representation of entity

[0116] The visual, spatial, semantic features obtained by the target detector have the same dimension after linear transformation, so the obtained visual feature g v , spatial feature pos(b v ) and semantic feature e v are directly spliced, and then the spliced features are input into the linear transformation layer W o to obtain the initial feature of the target entity

[0117] 5-2. Final feature representation of entity

[0118] The initial feature e v of the target entity is sequentially input into n o target entity encoders to obtain the final feature of the target entity n o is set to 4.

[0119] 5-3. The classifier and the activation function are used to obtain the prediction probability score of the object class to which the target entity belongs

[0120] The construction of the inter-entity relationship encoder described in step 6 obtains the feature representation of the relationship context and the relationship class, which is as follows:

[0121] 6-1. Initial feature representation of relationship

[0122] The final features of the entities s and o and the joint feature have the same dimension after linear transformation, so they can be directly spliced, and then the result is input into the linear transformation layer W r to obtain the initial feature of the inter-entity relationship

[0123] 6-2. Final feature representation of relationship

[0124] The obtained initial feature e s,o of the relationship is sequentially input into n r relationship encoders to obtain the final feature of the inter-entity relationship n r is set to 2.

[0125] 6-3. The classifier and the activation function are used to obtain the classification score z s,o and the prediction score of the class to which the inter-entity relationship belongs

[0126] The construction of the loss function in step 7 is as follows:

[0127] z s,o The input activation function obtains the final relationship category prediction score p, and finally the cross entropy is used to calculate the loss, so the total loss function calculation formula is as follows:

[0128]

[0129] The training model in step 8 is as follows:

[0130] According to the loss value of the entity and the relationship calculated in step 7, the parameters of the target encoder and the relationship encoder constructed in steps 5 and 6 are back propagated using the back propagation algorithm, and are continuously optimized until the entire network model converges.

[0131] The entity and relationship prediction value calculation in step 9 is as follows:

[0132] Sort the class score vector of each candidate box, and select the class corresponding to the highest score as the target prediction class; sort the relationship class score vector between each pair of candidate boxes, and select the class corresponding to the highest score as the relationship class.

[0133] From Figure 3 The experimental results fully prove that without the candidate predicate bias proposed in the present patent, the unbiased scene graph generation has a significant reduction in the effect of each indicator, and cannot accurately distinguish predicates with similar semantics in the predicate generation result between entity pairs, thereby making the model lack practicality.

[0134] The embodiments of the present application scheme are described in detail above in combination with the drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge possessed by those skilled in the art without departing from the purpose of the present application.

Claims

1. A method for generating an unbiased scene graph based on candidate predicate relation bias, characterized in that, According to the following steps: Step 1), select the pictures in the data set for preprocessing and divide the data set; Step 2), using the trained target detection network FasterRCNN to obtain the target candidate box Objectproposals and the related features of the picture from the picture of the pre-processed data set, and using the target detector to calculate the visual features g of any candidate box in each picture v , the bounding box coordinates b v , the classification score z of the target entity corresponding to the candidate box v , in addition to this, the joint features gU of the two target candidate boxes corresponding to the entities s and o are generated using the backbone network (s,o) ; Step 3), constructing the spatial feature of the target candidate box, for each candidate box, using the bounding box coordinates b obtained in step 2) v Calculate the spatial feature pos(b v ) of the candidate box; Step 4), constructing semantic features of the target bounding box, the bounding box corresponding to the entity classification score z obtained in step 2) v The entity category corresponding to the highest score is selected as the label c of the entity v , c v is added to the word vector model as the result edb(c v ) obtained by calculation, and edb is an unsupervised learning model GloVe for obtaining vector representation of words; Step 5), constructing the target entity encoder to obtain the context feature representation of the target entity and the entity category, and splicing the visual feature g v obtained in step 2), the spatial feature pos(b v obtained in step 3), and the semantic feature edb(c v obtained in step 4) according to the feature obtained by the target detector, and inputting the obtained target feature into the n o target encoder to obtain the final target feature representation The input target classifier and the activation function, and the output result is the prediction score p v of the object category to which the target candidate frame belongs.​ Step 6), constructing the inter-entity relationship encoder to obtain the feature representation of the relationship context and the relationship category, for each directed pair (s, o) in an image, first concatenate the final feature representation of the entity s obtained in step 5) and the final feature representation of the entity o and the joint feature representation of the entities s and o obtained in step 2) to obtain an initial relationship feature representation, and then perform linear transformation on the dimension of the vector to obtain a transformed relationship feature representation, and input the transformed relationship feature representation into the relationship classifier in step 5) n o The target encoder is constructed in the same way as n r In the relationship encoder, the output result obtains the final relationship representation feature of the entity s and the entity o Finally, the final relationship representation feature is input into the relationship classifier to obtain a relationship classification score z s,o , and the relationship classification score z s,o is input into an activation function to finally obtain a relationship prediction score p s,o between the two entity objects; Step 7), constructing candidate predicate relation bias, using candidate predicate relation bias to adjust the intensity of the model under different entity relations, and the relation classification score z obtained in step 6) s,o The top m of the highest relation score between each pair of entities in the middle r The number of candidate predicate names appearing to construct a candidate predicate matrix m s,o Using the candidate predicate matrix m s,o Calculate the generated candidate predicate relation bias b; Step 8), construct the loss function, use the relationship classification score z in step 6) s,o Subtract the predicate relationship bias b in step 7) and input the activation function to get a new entity-object relationship prediction score Then, the prediction score p of the category to which the object belongs obtained in step 5) v And the entity-object relationship prediction score Respectively, and the corresponding label vector is input into the cross-entropy loss function to calculate the respective loss value; Step 9), train the model, according to the loss value corresponding to the entity and relationship calculated in step 8), use the back propagation algorithm to perform gradient back propagation on the parameters of the target entity encoder and relationship encoder constructed in steps 5) and 6), and then continuously optimize until the entire network model converges; Step 10), entity and relationship prediction value calculation, sort the classification score vector of each candidate box, and select the category corresponding to the highest score as its predicted category; sort the classification score vector of the relationship between the candidate boxes, and select the relationship corresponding to the highest score as its relationship category. 2.The method of claim 1, wherein: The specific implementation of step 1) is as follows: Visual Genome is selected as the data set for model training, a total of 108K pictures, each picture has an average of 35 target entities, 21 relationships, using the most common 150 target categories and 50 relationship categories, first remove some low-quality, weak reference data set pictures, 70% of the pictures in the data set are used for training, and the remaining 30% of the pictures are used for testing. 3.The method of claim 1, wherein: Step 2) The target candidate box and related features extracted from the picture are as follows: the output result of the RPN part of the already trained target detection network is used as the target candidate box and the visual features g corresponding to the candidate box v , the coordinates b corresponding to the candidate box v , the target entity classification score z corresponding to the candidate box v , and the result obtained by using the backbone network in the target detection network as the joint features of the entity pair 4.The method of claim 1, wherein: Step 3) the construction of the spatial features of the target candidate frame is specifically as follows: using the obtained boundary frame position coordinates b v for (x min , y min , x max , y max ) to calculate, wherein (x min , y min ) is the corresponding coordinate of the upper left of the target candidate frame, (x max , y max ) is the corresponding coordinate of the lower right of the target candidate frame, using the parameters to obtain the spatial features pos(b v ) of the target candidate frame, wherein W, H represent the width and height of the corresponding picture, and w, h represent the width and height of the corresponding target candidate frame.

5. The method of claim 1, wherein: Step 4) The semantic features of the constructed candidate box are specifically as follows: the classification score of the target entity corresponding to the obtained candidate box The entity category c corresponding to the item with the highest score is selected v , c v is input into the trained GloVe word vector model to obtain the semantic features of the candidate box, and the specific formula is as follows: edb(c v ) = GloVe(c v ) (Equation 2). 6.The method of claim 1, wherein: The specific implementation of step 5) is as follows: 5-1. Concatenate visual features, spatial features, and semantic features and perform linear transformation on the result The visual, spatial, and semantic features obtained by the target detector have the same dimension after linear transformation, because the three features can be directly operated, and the obtained visual feature g v , spatial feature pos(b v ), and semantic feature e v are directly spliced, and then the spliced features are input into a linear transformation layer W o to obtain the initial feature e v of the target entity; the specific formula is as follows: e v = W o [pos(b v ), g v , edb(c v )] (Equation 3)5-2. Input target entity encoder to get the final feature representation of the entity The initialized target entity features e v are input to the n o target entity encoder, n o is set to 4, representing e v The final entity feature representation is obtained by iterating the 4-layer target encoder The target entity encoder mainly uses self-attention mechanism to represent the features, mainly including attention network and feedforward network, and the specific formula for calculating the attention matrix is as follows: where Q, K, V represent input features e v The values obtained by three different linear change layers, is the proportion factor of Q, K point multiplication, and σ is the activation function. wherein the attention network further comprises dividing Q, K, V into n h parts using multi-head attention, computing an attention for each part, and next fusing the attention for each part using a feed-forward network to obtain a new feature, iterating n o times to obtain a final target entity feature representation 5-3. The prediction probability score p of the object class to which the object belongs is obtained through the classifier and the activation function v The specific formula implementation is as follows: wherein denotes the target entity classifier, and σ denotes the activation function.

7. The method of claim 1, wherein: The specific implementation of step 6) is as follows: 6-1. Concatenate the final features of entities s and o and their joint features and perform linear transformation on the result Final features of entities s and o And joint features The dimensions after linear changes are the same, so they can be directly spliced, and the result is input into the linear change layer W r Get the initial features e of the relationship between entities s,o The calculation process is specifically as follows: 6-2. Input the relationship encoder to get the final feature representation of the relationship between entities The obtained inter-entity relationship initialization feature e s,o Input to n r In the relationship encoder, n r Is set to 2, representing e s,o Iterate 2 relationship encoders to obtain the final inter-entity relationship feature 6-3. The classification score z of the inter-entity relationship is obtained through the classifier and the activation function s,o and the category prediction score p s,o The specific formula is as follows: p s,o = σ(z s,o ) (Equation 8) wherein represents a relationship classifier, and σ represents an activation function. 8.The method of claim 1, wherein: The specific implementation of step 7) is as follows: 7-1. Calculate the predicate similarity matrix Firstly, m s,o is constructed as a 51*51 unit matrix, corresponding to the similarity between each pair of 50 relationship categories, z s,o is an n*51 score matrix, n represents all possible entity relationship pairs in a batch, and z s,o The scores corresponding to each pair of entity candidate predicates are sorted, and the highest score corresponds to the predicate and the predicate in the top m r The relationship between the names is established, and the number of times this relationship appears in each pair of entity candidate predicates is added to the predicate similarity matrix m s,o , m r is set to 5, and the Top5 relationship scores of a pair of entities in the z s,o matrix correspond to the relationship categories A, B, C, D, and E in turn. In the predicate similarity matrix m s,o , (A, B), (A, C), (A, D), and (A, E) are added by one respectively, and the Top5 relationship scores of n pairs of entities in the z s,o matrix are all executed as described above, thereby obtaining the matrix m s,o used to represent the predicate similarity. 7-2. Calculate the candidate predicate relationship bias The resulting predicate similarity matrix m is used s,s The candidate predicate relation bias between each pair of entities is constructed, and the specific formula is as follows: where r, e denote defined hyperparameters, C r denotes all entity relation classes, Concatenate all the candidate relationship predicate biases between entities to get the predicate relationship bias matrix b, and the specific formula is as follows: b = (b1,..., b i ) (Equation 10).

9. The method of claim 1, wherein: The specific implementation of step 8) is as follows: z s,o Subtracting b gives the score matrix after increasing the relation bias, and the score matrix is input into the activation function to obtain the final relation category prediction score The specific formula is as follows: Therefore, the specific formula for calculating the relationship loss function is as follows: The specific formula for calculating the entity loss function is as follows: The specific formula for calculating the total loss function is as follows:

Images