Scene graph generation method based on basic model, computer device and storage medium
By employing a scene graph generation method based on the CLIP model, leveraging relational hints and large-scale language models to enhance textual prompts, and combining triplet-perceptual logarithmic adjustment and adaptive evidence smoothing techniques, the method addresses the recognition difficulties of scene graph generation models under long-tailed distributions and sparse annotations, thereby improving the model's accuracy and efficiency.
Patent Information
- Application Number
- CN202410943323.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-26
AI Technical Summary
Existing scene graph generation models have difficulty identifying zero-shot triples caused by long-tailed distributions and sparse annotations, and also suffer from low training and inference efficiency, difficulty in distinguishing similar relationships, and double bias.
We employ a scene graph generation method based on the CLIP model, extracting relational embeddings of object pairs through relational cues and visual localization, enhancing textual cues by combining a large language model, and introducing triple perceptual logarithmic adjustment and adaptive evidence smoothing techniques for the base model to optimize model performance.
It improves the accuracy of scene graph generation, especially in the detection of tail relationships and zero-sample relationships, enhances the model's running efficiency, and achieves balanced detection of head and tail relationships.
Smart Images

Figure CN119027944A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computer vision and natural language processing technology, and in particular to a scene graph generation method based on a basic model, a computer device and a storage medium. Background Art
[0002] Scene Graph Generation (SGG) is a key task that connects computer vision and natural language processing and aims to generate structured semantic representations of scenes. It contains descriptions of objects in the scene (e.g., people, chair) and their pairwise relations (e.g., sitting on), forming triplets (e.g.,<people,standingon,chair> ). By converting images into structured and detailed semantic information, the understanding and reasoning ability of visual scenes are enhanced. Given its ability to understand complex scenes, SGG has received increasing attention in recent years and has been widely used in various downstream tasks, including image text retrieval, visual question answering, visual description, etc. At present, the SGG framework mainly includes Transformer-based, CNN-based, RNN / LSTM-based, and GNN-based SGG. Although each framework has its own advantages in processing complex object relationships in images, there are common problems caused by the SGG dataset itself: 1) Model bias caused by long-tail distribution. Due to the long-tail distribution in the training data, the model easily classifies fine-grained tail relations as coarse-grained head relations; 2) Zero-shot triple challenges caused by sparse annotations. Sparse annotations lead to a large number of zero-shot triplets, that is, triplets that are missing in the training set but exist in the test set, which poses a huge challenge to the recognition of existing models.
[0003] For model bias, current debiasing methods usually emphasize enhancing the influence of tail relations in training to improve their prediction performance, which inevitably degrades the performance of head relations. For the challenging task of zero-shot triple recognition, even if some zero-shot learning techniques can be adopted, the severe sparsity of the SGG dataset still hinders its effectiveness. To address the above problems, an SGG framework based on the CLIP model can be designed, as shown in Figure 1(a). First, the region containing the object pair is cropped according to the position information of the object pair, and then input into the CLIP image encoder to generate the relation embedding. At the same time, the CLIP text encoder is used to generate the relation description embedding. The similarity score between the relation embedding and the relation description embedding is calculated, and the input text with the highest score is selected as the final prediction. However, the SGG framework still has three problems that need to be solved: 1) Inefficient training and inference. The SGG framework usually requires feature extraction for each object pair in the image, which generates a lot of computational overhead during training and inference, resulting in significant resource consumption. Even if the CLIP image encoder can be used to extract complete image features, a large number of errors are prone to occur when separating the corresponding features according to the position information of the object pair; 2) Similar relations are difficult to distinguish. The CLIP model is trained on a large-scale dataset and is not optimized for fine-grained relations, which may make it difficult to effectively distinguish similar relations (as shown in Figure 1(b)). In addition, the noise in the CLIP model training dataset may further reduce its effectiveness in distinguishing these subtle differences, thereby affecting recognition accuracy in actual operations. 3) There are dual biases, namely, long-tail bias and base model bias. Although the SGG framework has improved in the recognition of tail relations and zero-shot triples, it still faces inevitable long-tail bias when dealing with extremely skewed distribution data. In addition, the large-scale training datasets of the CLIP base model come from the Internet, which inherently contain various biases and stereotypes and may perpetuate or even amplify these biases. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a scene graph generation method, a computer device and a storage medium based on a basic model to improve the accuracy of scene graph generation in view of the shortcomings of the existing technology.
[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a scene graph generation method based on a basic model, comprising the following steps:
[0006] S1. Take image I as the input of the target detector to get the relationship prompts; take image I as the input of CLIP to get the global features of the relationship
[0007] S2. Merge the relationship prompts and the relationship global features Get the merge relation sequence;
[0008] S3. Use the merged relationship sequence as the input of the attention structure to obtain the relationship feature i and j represent the serial numbers of the objects in image I respectively; As input to CLIP, get text features Indicates the text representation after SemAugm() enhancement; SemAugm() represents the enhanced kernel, TP (0) represents the original text representation, f gpt Represents the GPT model;
[0009] S4. Calculate the relationship feature Distance to text features Using the distance Construct the loss function l clip : in, N represents the number of image-text pairs, τ is the temperature coefficient;
[0010] S5. Use the loss function to update the parameters of the attention structure to obtain a scene graph generation model.
[0011] The relationship prompts include Rel ij 、Loc ij , Cla ij ,Rel ij 、Loc ij , Cla ij Respectively represent the relationship hint, location hint, and classification hint of the i-th and j-th objects:
[0012]
[0013] Among them, RoIPooling() represents RoI pooling, UnionBox() represents the embedding of two object boxes, and ClaEmbed() represents the class information embedding obtained from the language model. represents the features obtained by inputting image I into the object detector, Indicates that B is included i and B j The maximum bounding box, B i ,B j Represent the i-th object frame and the j-th object frame respectively, <o i ,o j > represents an object pair consisting of the i-th and j-th objects.
[0014] The text features The specific calculation formula is: in, Represents the output of the CLIP text editor, where Z represents the number of enhancements.
[0015] In the present invention, Z=3.
[0016] in, For triples <o i ,r k ,o j >Adjustment factor; | <o i ,r k ,o j >| train Represents the triplet in the training set <o i ,r k ,o j > count, r k Indicates o i With o j The relationship between, K and C represent the number of relationship classification and object classification respectively; Acc i and Acc j They represent the accuracy of the i-th object category and the j-th object category respectively, x i,j represents the jth instance of the i-th object category, f clip (x i,j ) is the x determined by CLIP i,j The object category, N evi The number of instances to sample for each class.
[0017] As an inventive concept, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the steps of the above method.
[0018] As an inventive concept, the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon; characterized in that the computer program / instruction implements the steps of the above method when executed by a processor.
[0019] A computer program product comprises a computer program / instruction; when the computer program / instruction is executed by a processor, the steps of the above method are implemented.
[0020] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes to improve the existing scene graph generation model based on a large model, the large model can provide more robust relationship features, and the detection capability of tail relations and zero-shot relations in the scene graph generation task is greatly improved; the relationship prompts can obtain all relationship embeddings from the entire image at one time, greatly improving the operation efficiency of the model; the model of the present invention can simultaneously ensure the detection capability of head relations and tail relations in the scene graph, so as to achieve performance optimization and balance of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1(a) shows a simple scene graph generation framework consisting of the CLIP model, and Figure 1(b) shows the comparison of similarity relations;
[0022] Figure 2 This is a schematic diagram of the method of an embodiment of the present invention;
[0023] Figure 3 A method for embedding position information of an object in an embodiment of the present invention;
[0024] Figure 4 The semantic content of the text prompt is enhanced for the embodiment of the present invention;
[0025] Figure 5 The effect diagrams of relationship adjustment at different levels in the embodiments of the present invention are as follows: (a) bias prediction, (b) relationship level adjustment, and (c) triple level adjustment;
[0026] Figure 6 Generate a result graph for the scene graph of the embodiment of the present invention;
[0027] Figure 7 This is a graph showing average recall rates of different triple relationships according to an embodiment of the present invention;
[0028] FIG8( a ) is a graph comparing ZsT-Guiding and No Guiding results of an embodiment of the present invention, and FIG8( b ) is a graph comparing FMAE-Smoothing and No Smoothing results of an embodiment of the present invention. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] Example 1
[0031] The embodiment of the present invention proposes a SGG framework (FMSGG) based on the basic model CLIP, such as Figure 2 As shown in the figure, the basic model is pre-trained on a large-scale dataset to obtain comprehensive features containing a large amount of tail relationship information, so that even when facing a skewed SGG dataset, the ability to distinguish fine-grained tail relationships can be maintained. CLIP can use text information to assist in inferring relationships in images, improving the ability of the SGG model to reason about implicit relationships. Its outstanding zero-shot learning ability is expected to improve SGG's ability to recognize zero-shot triplets.
[0032] The CLIP model consists of two main components: an image encoder and a text encoder. Object pairs are cropped from images and fed into the image encoder to obtain relational embeddings. Input text is constructed based on object category information and fed into the text encoder to obtain text embeddings. The two encoders share the same embedding space during training, enabling them to generate embeddings with similar semantics in the same vector space. Therefore, CLIP maximizes the similarity between image-text pairs while minimizing the similarity between images and unrelated texts. This can be represented by the similarity matrix S:
[0033]
[0034] where F I and F T Represent the embedding of image I and text T respectively. CLIP uses contrast loss, specifically, the InfoNCE loss function is:
[0035]
[0036] Among them l image-to-text represents the loss from image to text, l text-to-image represents the loss from text to image, defined as:
[0037]
[0038] Where N is the number of image-text pairs, T ′ Indicates other irrelevant text descriptions, I ′ Indicates other unrelated images.
[0039] In order to improve the efficiency of training and reasoning, the embodiment of the present invention proposes a relationship hint method based on visual localization, which uses relationship hints to locate and isolate the relationship embedding of any object pair from the complete image features obtained by the CLIP image encoder. Assume that image I contains M objects, and the i-th object o i The bounding box is B i =(x i1 ,yi1 ,x i2 ,y i2 ).pass and f detector The characteristics of I obtained are and
[0040]
[0041] Relationship cues include feature information Location Information and classified information of objects.
[0042] Pro Fea ,Pro Loc , and Pro Cla Represents an object <o i ,o j >The prompt component can be calculated as:
[0043]
[0044] Loc ij =UnionBox(B i ,B j ),
[0045]
[0046] RoIPooling(,), UnionBox(,) and ClaEmbed() represent three operation kernels. RoIPooling(,) corresponds to RoI pooling in object detectors, which is used to convert candidate regions of various sizes into feature maps of fixed size. Representation based on bounding box from Crop fixed-size features in Indicates that B is included i and B j The maximum bounding box of The details are as follows:
[0047]
[0048] ClaEmbed() represents the class information embedding obtained from the language model, and UnionBox(,) represents the embedding of two object boxes.
[0049] The embodiment of the present invention proposes a new method for embedding position information of an object, such as Figure 3 Specifically, for object o i and jTwo object boxes B i =(x i1 ,y i1 ,x i2 ,y i2 ) and B j =(x j1 ,y j1 ,x j2 ,y j2 ), first construct a two-dimensional feature matrix initialized to zero Shape
[0050] Will This method preserves the local structural information within the bounding box, thereby better capturing the local features and global structure of the object. Specifically: Region 1, the only region belonging to object o i The area of the object is assigned with value p; area 2, which is unique to object o j The area is assigned with value -p; area 3, object o i and j Shared area, assigned value q; area 4, object o i and j The mutually exclusive part is assigned a value of -q. Specifically, in the embodiment of the present invention, p and q are set to 0.1 and 0.01 respectively. The specific expression is as follows:
[0051]
[0052] For object pairs <o i ,o j >Relationship Tips Rel ij 、Loc ij and Cla ij , connect them and embed them into a In vectors of the same shape:
[0053] Pro ij =RelProEmbed([Rel ij ;Loc ij ; Cla ij ])
[0054] RelProEmbed() is an embedding structure based on multi-layer perceptron MLP. In order to effectively utilize global information and Local Details Pro ij , construct these features into a sequence Φ I :
[0055]
[0056] The first element of the sequence is the full image feature, which provides the overall contextual information of the base model; the remaining elements are relation hints, which are used to extract the relation embedding of each object pair from the full image features extracted by the base model.
[0057] For the sequence Φ I , the embodiment of the present invention uses the transformer structure to extract relation embedding. The self-attention mechanism can be expressed as:
[0058]
[0059] Where Q, K, and V represent query, key, and value matrices, which are obtained by different linear transformations of the input. M is used as an attention mask to modify the attention weights before softmax calculation to avoid inappropriate interactions. Since we want global features to affect all local features to convey the overall context of the image while preventing contamination of local information, the attention mask M is designed as:
[0060]
[0061] In order to solve the problem of difficulty in distinguishing similar relationships, the embodiment of the present invention proposes a semantic enhancement method based on a large language model. Specifically, a simple text prompt can be enhanced to:
[0062] TP (1) =SemAugm(TP (0) ,f Δ )
[0063] Where SemAugm() represents the enhancement kernel, and the positions before and after the comma represent the text prompt to be enhanced and the language model to be used. The embodiment of the present invention uses ChatGPT as a language model to expand the text prompt TP (0) , namely TP (1) =SemAugm(TP (0) ,f gpt ). Compared with many other language models, ChatGPT performs well in generating coherent and logically consistent text. In addition, ChatGPT's powerful interactive capabilities enable it to effectively handle user feedback and guidance, such as modifying or regenerating descriptions, to optimize text prompts. However, relation prompt enhancement may cause triple descriptions to be too narrow and unable to cover the various representations of relations in different triple combinations. To solve this problem, it is only necessary to enhance the mutual information between relation description embeddings and relation embeddings in the FMSGG framework. <o i ,o j > as an example, its enhanced text prompts and relationship prompts are and Pro ij , their information entropy is:
[0064]
[0065] therefore, and Pro ij The mutual information between them is:
[0066]
[0067] in represents the joint information entropy of image and text features. In the proposed framework, Remain unchanged, the embodiment of the present invention increases To reduce the conditional entropy in:
[0068]
[0069] To achieve this, each relationship is enhanced multiple times, with each instance from Convert to That is, each relationship is enhanced Z times (Z=3 in the embodiment of the present invention):
[0070]
[0071] These augmented text hints are then averaged to obtain the optimized relation description embedding:
[0072]
[0073] To solve the double bias problem, the embodiment of the present invention introduces triplet-aware logarithmic adjustment (TLAdjustment) to reduce the long-tail bias, and introduces basic model adaptive evidence smoothing (FMAE-Smoothing) to deal with the bias of the basic model. Logarithmic adjustment solves the long-tail distribution problem by adjusting the model prediction to ensure fair impact of each category. In essence, it assigns an adjustment factor to the output of each category, which is usually the inverse of the classification frequency. Specifically, the output of FMSGG A logarithmic adjustment factor can be used Make adjustments:
[0074]
[0075] However, the long-tail distribution in the SGG dataset occurs at the triple level, and triples of the same relation are still skewed, which means that different triples should be assigned different adjustment factors. Figure 5 The key difference between relation-level and triple-level adjustments is explained. The TLAdjustment method proposed in the embodiment of the present invention constructs triple-level adjustment factors The allocation is as follows:
[0076]
[0077] in
[0078]
[0079] For triples <o i ,r k ,o j > adjustment factor. Similar to the classic logarithmic adjustment method, the adjustment factor is set to the inverse of the triple frequency for each relation:
[0080]
[0081] where | <o i , r k , o j >| train Represents the triplet in the training set <o i , r k , o j > count.
[0082] Due to the missing of some triples in the training set, the adjustment parameters cannot be calculated, and the adjustment factors of these triples cannot be calculated. In order to solve this problem, the embodiment of the present invention proposes a ZsT-Guiding strategy, which uses the statistical characteristics of the training data to classify these missing triples.
[0083] The embodiment of the present invention specifically defines two inference rules:
[0084] Rule 1: If o m and M have semantic similarity, and <o m , r k , o n > exists in the training set, then p(<o_M,r_k,o_n> |<o_m,r_k,o_n> )>0. If <o M , r k , o n > is missing in the training set, it is classified as a zero-shot triplet.
[0085] Rule 2: If o n and N have semantic similarity, and <o m , r k , o n > exists in the training set, then p(<o_m,r_k,o_N> |<o_m,r_k,o_n> )>0. If <o m , rk , o N >If it is missing in the training set, it is classified as a zero-shot triplet.}
[0086] To quantify the semantic similarity between Rule 1 and Rule 2, a weighted combination of two independent similarity measures of the embodiment of the present invention, namely, Euclidean distance and cosine similarity, is used:
[0087]
[0088] In the embodiment of the present invention, α=0.7. Finally, a threshold β is set to evaluate o m and M Is there semantic similarity?
[0089]
[0090] In the embodiment of the present invention, β=0.4.
[0091] Based on rules 1 and 2 and the semantic similarity calculation method proposed in the embodiment of the present invention, the missing triples in the training set can be classified as zero samples or impossible triples. Then, their counts in the training set are adjusted from zero to a specific value according to the classification results. Specifically, for the triples in the training set represented by r k The triple relationship formed is The missing triples are set to count as:
[0092]
[0093] In the embodiment of the present invention, the basic model adaptive evidence smoothing (FMAE-Smoothing) uses the object recognition ability of CLIP as evidence for optimizing the adjustment factor to reduce the bias of the basic model. Specifically, assuming that the distribution of objects in the training data set is Follows Dirichlet distribution:
[0094] φ o ~Dir(γ)
[0095] where γ=(γ1,γ2,…,γ K ) represents the parameters of the Dirichlet distribution, which is the adjustment factor As prior knowledge. o Can be initialized as:
[0096]
[0097] Then calculate the evidence extracted from the base model CLIP, that is, the accuracy of each object category:
[0098]
[0099] where x i,j represents the jth instance of the i-th object category, In order to improve the efficiency of evidence calculation, each category only samples N evi In this embodiment of the present invention, N evi =1000.
[0100] To capture the uncertainty of the extracted evidence, Acc is assumed to follow a Beta distribution, since it allows and To express this uncertainty:
[0101]
[0102] Where Beta(,) is the Beta function:
[0103]
[0104] Where Γ() is the Gamma function. Therefore, the parameters of the Dirichlet distribution can be updated as:
[0105] γ i ′ =γ i ×σ i
[0106] in
[0107]
[0108] This Acc-based parameter updating mechanism reflects the relative importance and uncertainty of each category, so that the original adjustment factor is adjusted.
[0109] Since a triplet is composed of two objects, using the accuracy of object pairs as evidence is more suitable for the SGG task. Based on this principle, the parameters of the Dirichlet distribution are reconstructed to adapt to object pairs:
[0110]
[0111] in
[0112]
[0113] In this way, the parameters of the underlying Dirichlet distribution of the object pair can be updated as:
[0114]
[0115] Therefore σ ij It can be calculated as:
[0116]
[0117] Using these updated parameters, the previously obtained tuning parameters can be optimized Therefore, for FMSGG, TLAdjustment and FMAE-Smoothing are used to output The debiasing result is updated as:
[0118]
[0119] is the adjustment factor extracted from the training dataset, and γ i ′ j It is optimized from the base model through evidence learning. Considering the skewed training data and the evidence extracted from the base model, the double bias problem in the proposed framework can be eliminated.
[0120] In order to verify the specific performance of FMSGG proposed in the embodiment of the present invention, the VG150 subset of VisualGenome (VG) popular in the SGG community was selected for experiments, which contains 150 object categories and 50 relationship categories. VG150 contains about 94k images, following the division method used in traditional methods, including about 62k training images, 5k verification images and 27k test images.
[0121] Evaluation Mode: Three standard evaluation modes are used, including: 1) Predicate Classification (PredCls), which predicts the relationship between each pair of objects given the true labels of the object categories and bounding boxes. 2) Scene Graph Classification (SGCls), which predicts the object categories and the relationship between each pair of objects given the true labels of the object bounding boxes. 3) Scene Graph Detection (SGDet), which predicts the object categories and bounding boxes as well as the relationship between each pair of objects without any object-related information.
[0122] Evaluation Metrics. Four evaluation metrics are used: 1) Recall (R@K), which calculates the proportion of correctly predicted relations to the total annotated relations. Here, K represents the number of highest confidence relation predictions considered in each image, and is usually set to 20, 50, and 100, corresponding to R@20, R@50, and R@100, respectively. 2) Mean Recall (mR@K), which averages the recall of each relation. Given the extremely long-tail distribution of the SGG dataset, a high R@K score can be obtained by performing well on a few head categories. mR@K addresses this issue by averaging the recall of all relations, so it has become the main metric for evaluating SGG debiasing in recent years. 3) The Mean of R@K and mR@K (M@K), which calculates the average of R@K and mR@K to evaluate the ability of the SGG model to balance performance between head and tail relations. 4) Zero-shot Recall (zR@K), which is a challenging metric that calculates the recall of zero-shot triples.
[0123] The quantitative results are shown in Tables 1, 2, 3, and 4, which show the comparison under the mR@K, R@K, MR@K, and zR@K indicators, respectively. Based on these results, the following conclusions can be drawn:
[0124] Table 1
[0125]
[0126] Table 2
[0127]
[0128] Table 1 shows that the method proposed in the embodiment of the present invention achieves the best performance in the debiasing index mR@K in most evaluation modes. For example, compared with the single-stage SGG method CV-SGG, the proposed method FMSGG improves the mR@100 index by 5.9%, 7.3% and 6.9% in the PredCls, SGCls and SGDet evaluation modes respectively. Compared with the two-stage SGG method ST-SGG, the method proposed in the embodiment of the present invention improves by 3.8%, 4.2% and 6.1% in these three evaluation modes respectively.
[0129] Table 3
[0130]
[0131] Table 4
[0132]
[0133] Table 2 shows that the method proposed in the embodiment of the present invention achieves state-of-the-art performance on the debiasing index mR@K while retaining the R@K performance to the maximum extent. The FMSGG proposed in the embodiment of the present invention utilizes the basic model CLIP to enhance the features of the tail relations instead of sacrificing the performance of the head relations during training, thereby maintaining the performance of the head relations while pursuing unbiased prediction. For example, in the PredCls evaluation mode, VCTree equipped with debiasing methods such as TransRwt, PPDL, and CFA decreased by 18.2%, 20.1%, and 10.9% in the R@100 index, respectively, while the method proposed in the embodiment of the present invention only decreased by 2.7%. In the SGDet mode, the method proposed in the embodiment of the present invention is even slightly ahead, further highlighting the key difference between the proposed method and other debiasing methods.
[0134] Table 3 shows that the method proposed in the embodiment of the present invention achieves the latest performance on the MR@K indicator. For example, compared with the single-stage SGG method SQUAT, the method proposed in the embodiment of the present invention has the highest performance in the PredCls, SGCls and SGDet evaluation modes. ° They are ahead by 7.1%, 7.1% and 6.7% respectively. For the two-stage SGG method MotifsNet, the method proposed in the embodiment of the present invention outperforms the debiasing technique TBE by 11.1%, 8.6% and 10.2% respectively in these evaluation modes. MR@K is the average of the head relation-friendly index R@K and the tail relation-friendly index mR@K, so the results in Table 3 show that the method proposed in the embodiment of the present invention achieves the best balance between the head and tail relations.
[0135] Table 4 shows that the method proposed in the embodiment of the present invention achieves the latest performance on the zero-sample triple recognition index zR@K. Specifically, in the PredCls, SGCls and SGDet evaluation modes, the proposed method achieves the latest performance on the index The embodiments of the present invention attribute this success to the supplement of basic visual features provided by the base model CLIP, which is an enhancement that the baseline methods cannot achieve. In fact, the zero-shot capability of the base model in downstream tasks has been widely discussed in the literature. Their similar performance in the SGG task further enhances the confidence in leveraging the knowledge of large models to improve the performance of downstream tasks.
[0136] Qualitative results such as Figure 6As shown, it can be observed that the FMSGG proposed in the embodiment of the present invention can predict the tail relationship with rich information. Due to the long-tail distribution problem, the baseline SGG model often predicts the head relationship with less information. For example,<cat,laying on,bed> vs.<cat,on,bed> and<man,sitting on,skateboard> vs.<man,on,skateboard> . An embodiment of the present invention weights tail relations in the output reasoning stage to compensate for their insufficiency in training instances. Secondly, FMSGG is able to predict information-rich interactive relations. The base model integrated in the embodiment of the present invention supplements the relational features of the detector trained on a limited data set, so that relation prediction relies on more complex and complete visual features rather than partial visual clues such as relative object positions. Finally, FMSGG is able to predict subordinate relations between objects, thanks to the large language model ChatGPT used to enhance relational descriptions and optimize text prompts, and the comprehensive corpus covers relational expressions that are scarce in the SGG dataset.
[0137] By conducting ablation experiments on the key components of FMSGG (image prompts and text prompts), the results are shown in Table 5, and the following conclusions can be drawn:
[0138] 1) Each hint component contributes to the performance of FMSGG. ij and have the largest impact on model performance, which is consistent with expectations because they correspond to the image encoding and text encoding inputs in the base model CLIP, respectively. ij and Cla ij There is also a significant effect, indicating that Rel ij It is not enough to locate object pairs from the overall image features, and additional location information guidance is required.
[0139] 2) The difference between R@K and mR@K shows that FMSGG without debiasing is a typical biased model. However, it still performs well in mR@K, R@K and z@K indicators. For example, even without debiasing, the proposed method achieves 34.3% in the mR@K indicator, surpassing many debiasing methods in Table 1. Therefore, it can be considered that this supports the original view of the embodiment of the present invention: the introduction of the base model is intended to supplement the features of the tail relationship, while the existing typical debiasing method weakens the head relationship to enhance the learning of the tail instance.
[0140] 3) The base model can supplement the scarce relation features in the SGG dataset and alleviate the dominance of the head relations in training. However, due to the extreme long-tail distribution problem, the tail relations are still not fully learned. Therefore, equipping the base model with a debiasing method can further improve the performance, especially for tail and zero-shot triplets. To address this issue, TLAdjustment is introduced to weight the tail and zero-shot triplets in the model inference stage to improve the performance of these triplets.
[0141] Table 5
[0142]
[0143] Table 6
[0144]
[0145] In order to verify the effectiveness of TLAdjustment, we first calculated the prediction performance of the biased SGG model at the triple level, as follows: Figure 7 As shown. For the head relation “on”, some triples (such as<woman,on,horse> and<building,on,street>}) perform poorly. In contrast, triples formed by the tail relation “walking on” (e.g.<lady,walking on,sidewalk>}and<man,walking on,sidewalk> The performance is significantly better. These observations further indicate that adjusting logits only at the relation level is ineffective due to the complexity of SGG triple composition. Table 6 further shows that TLAdjustment can reduce the bias of the model. Although the additional knowledge of the base model makes the model pay more attention to the tail relations during training, this is not enough to achieve unbiased training. The embodiments of the present invention attribute this to the extreme long-tail distribution of the SGG training set. Therefore, even if the knowledge of the base model is integrated during training, the SGG task still requires debiasing methods to further improve performance.
[0146] ZsT-Guiding and FMAE-Smoothing are two key components of the debiasing method proposed in the embodiment of the present invention. Figure 8(a) shows the performance comparison between ZsT-Guiding and a simple method (referred to as No Guiding in the table), showing that the proposed method is significantly ahead in the zero-sample indicator zR@K.
[0147] In order to reduce the bias of the base model, an embodiment of the present invention proposes FMAE-Smoothing, which adjusts the frequency-based logarithmic adjustment factor based on evidence smoothing by calculating the importance evidence of different triplets. Figure 8(b) shows that FMAE-Smoothing has a significant advantage over No Smoothing in the mR@K metric. This shows that it is necessary to consider the impact of external knowledge on the distribution of the original data. In addition, quantifying these impacts and adjusting the model accordingly is the key to using the base model to enhance the performance of downstream tasks.
[0148] Example 2
[0149] Embodiment 2 of the present invention provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device may be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.
[0150] The terminal device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in the above-mentioned embodiment 1.
[0151] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.
[0152] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.
[0153] Example 3
[0154] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to the above embodiment 1, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the above embodiment 1 are implemented.
[0155] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.
[0156] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0157] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.
[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0159] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0160] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A scene graph generation method based on a basic model, It is characterized in that The following steps are involved: S1. Take image I as the input of the target detector and get relation prompts. Take image I as input of CLIP to obtain the global features of the relationship S2. Merge the relationship prompts and the relationship global features Get the merge relation sequence; S3. Use the merged relationship sequence as the input of the attention structure to obtain the relationship feature i and j represent the serial numbers of the objects in image I respectively; As input to CLIP, get text features Indicates the text representation after SemAugm() enhancement; SemAugm() represents the enhanced kernel, TP (0) represents the original text representation, f gpt Represents the GPT model; S4. Calculate the relationship feature Distance to text features Using the distance Construct the loss function l clip : in, N represents the number of image-text pairs, τ is the temperature coefficient; S5. Use the loss function to update the parameters of the attention structure to obtain a scene graph generation model.
2. The scene graph generation method based on the basic model according to claim 1, It is characterized in that The relationship prompts include Rel ij 、Loc ij , Cla ij ,Rel ij 、Loc ij , Cla ij Respectively represent the relationship hint, location hint, and classification hint of the i-th and j-th objects: Among them, RoIPooling() represents RoI pooling, UnionBox() represents the embedding of two object boxes, and ClaEmbed() represents the class information embedding obtained from the language model. represents the features obtained by inputting image I into the object detector, Indicates that B is included i and B j The maximum bounding box, B i ,B j Represent the i-th object frame and the j-th object frame respectively, <o i ,o j > represents an object pair consisting of the i-th and j-th objects.
3. The scene graph generation method based on the basic model according to claim 1, It is characterized in that The text features The specific calculation formula is: in, Represents the output of the CLIP text editor, where Z represents the number of enhancements.
4. The scene graph generation method based on the basic model according to claim 1, It is characterized in that Z=3。 5. The scene graph generation method based on the basic model according to claim 1, It is characterized in that in, For triples <o i ,r k ,o j >Adjustment factor; | <o i ,r k ,o j >| train Represents the triplet in the training set <o i ,r k ,o j > count, r k Indicates o i With o j The relationship between, K and C represent the number of relationship classification and object classification respectively; Acc i and Acc j They represent the accuracy of the i-th object category and the j-th object category respectively, x i,j represents the jth instance of the i-th object category, f clip (x i,j ) is the x determined by CLIP i,j The object category, N evi The number of instances to sample for each class.
6. A computer device comprising a memory, a processor and a computer program stored in the memory; It is characterized in that The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.
7. A computer-readable storage medium having a computer program / instructions stored thereon; It is characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.
8. A computer program product comprising a computer program / instructions; It is characterized in that When the computer program / instructions are executed by a processor, the steps of the method of any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Scene graph generation method based on global information and position embedding
CN113836339A
Unbiased scene graph generation method based on candidate predicate relation deviation
CN116645501A
Unbiased scene graph generation method and device based on anti-causal model, and storage medium
CN117557997A
Refill type lipstick container
KR1020250037876A
Cited By
Open domain indoor scene hierarchy generation method and system based on large language model
CN119416330A
