Two-stage causal modeling-based unbiased scene graph generation method
Through visual triple labeling data sets and population losses and adaptive logit adjustments, the problem of long-tail distribution and semantic confusion in scene graph generation is solved, unbiased prediction is achieved, and the performance of scene graph generation model is improved.
Patent Information
- Application Number
- CN202310654130.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-06-05
AI Technical Summary
The existing scenario graph generation method has biases when dealing with long-tail distribution and semantic confusion, resulting in head relationship prediction errors and ignore other biases, affecting model performance.
The data set is marked with visual triple-tubes, and through population loss and adaptive logit adjustment, semantic confusion factors of long-tail distribution are eliminated, and they are divided into two stages: causal representation learning and causal calibration learning, realizing unbiased scene graph generation.
While maintaining the performance of the head category, the prediction accuracy of tail relationships and the ability to distinguish similar relationships are improved, which significantly improves the average recall rate and is better than the existing technology.
Smart Images

Figure CN119090981A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an unbiased scene graph generation method, terminal equipment and storage medium. Background Art
[0002] Existing scene graph generation works have made important progress, but these works often produce biased predictions, that is, informative fine-grained relations (such as "standing at") are predicted as less informative coarse-grained relations (such as "at") due to the long-tail distribution problem. Take VG150 as an example, it is a well-known dataset in the scene graph generation task, but it has a serious long-tail distribution problem. The scene graph generation model naturally cannot learn the characteristics of both head and tail relations from the skewed distribution, so it is easy to make wrong predictions on the head relation. In response to the above biased predictions, many debiasing methods have been proposed to solve this problem. Unlike early works, the main goal of debiasing methods is to pursue unbiased scene graphs. Existing debiasing methods can be roughly divided into four categories: 1) Resampling methods rebalance the training data distribution by upsampling tail relations or downsampling head relations. 2) Reweighting methods correct the contribution of different relations during training. For example, the prediction loss is weighted to enhance the model's representation ability for tail categories. 3) Adjustment methods modify the learned biased model to obtain unbiased predictions, for example, by adjusting the output logits to increase the likelihood of more informative fine-grained relationships. 4) Hybrid methods combine some or all of the above methods. An existing scene graph generation method such as Figure 1 As shown in Figure 2. Although debiasing research in scene graph generation tasks is quite active, the above methods often fall short in pursuing the prediction of informative tail relations while maintaining the performance of the head category. More importantly, current debiasing methods mainly focus on a single bias, namely the long-tail distribution problem, while ignoring other biases, such as ignoring semantic confusion, which makes scene graph generation models prone to produce incorrect predictions for similar relations. Summary of the invention
[0003] The technical problem to be solved by the present invention is to provide an unbiased scene graph generation method, terminal device and storage medium to overcome the long-tail distribution problem and eliminate semantic confusion in view of the shortcomings of the existing technology.
[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: an unbiased scene graph generation method, comprising the following steps:
[0005] S1. Using visual triples Annotated image dataset, where Represent the classification of two objects respectively. Represent the bounding boxes of the two objects respectively, is the number of observations in the sample, is the N in the sample i relations, where N i is the number of relations in the sample, and K is the number of relation categories in the observed data;
[0006] S2, extracting relationship populations from the observation data of the labeled image dataset;
[0007] S3, training a machine learning model using the relationship population;
[0008] S4, extracting adjustment parameters from the observed data and the trained model;
[0009] S5. Using the adjustment parameters to adjust the model output and generate an unbiased scene graph.
[0010] In step S2, for the tth relationship category y in the observation data t , relationship population It is expressed as:
[0011]
[0012] Among them, arg smalα is the calculation kernel, which means selecting similar relationships based on feature distance. To represent non-relational y t The relationship characteristics, and They are all the relations in the observed data<o i ,y t ,o j > is the fusion feature and quantity, C is the number of objects in the observation data, o i and j They are two different objects.
[0013] In step S4, the adjustment parameters The expression is:
[0014]
[0015] Among them, y i and l j denote the i-th relationship and the j-th position of the logit respectively; and Respectively represent the relationship y i Corresponding to the first j Adjustment of the lower and upper bound combinations of positions; Indicates the mth value of adjusting the lower bound combination, represents the nth value of the upper bound of the adjustment; T represents the potential space of the adjustment factor, and t represents the possible values in this space.
[0016] The loss function of the machine learning model is expressed as:
[0017]
[0018] Among them, x and y are the input and output relation categories respectively; It is the relationship i statistical knowledge, is the statistical knowledge of input y; π y represents the quantity distribution of input y, π y′ represents the number distribution of input y′; f y (x) and f y′ (x) represents the logit of y and y′ respectively when the input is x.
[0019] The parameters θ of the machine learning model * The expression is:
[0020]
[0021] in, is the observed data, is the expectation, θ is the potential parameter space of the machine learning model, and f(x) represents the output after inputting x into the machine learning model f.
[0022] As an inventive concept, the present invention also provides a terminal device, which includes:
[0023] one or more processors;
[0024] A memory having one or more programs stored thereon, which, when executed by the one or more processors, enables the one or more processors to implement the steps of the above method of the present invention.
[0025] As an inventive concept, the present invention further provides a computer-readable storage medium storing a computer program, which implements the steps of the above method of the present invention when executed by a processor.
[0026] The present invention first considers semantic confusion and long-tail distribution as confounders of the standard scene graph generation framework, and then uses data-level confounders to model causal relationships for the scene graph generation task. Next, a population loss is proposed to remove semantic confusion confounders and obtain a decoupled factorization of the causal model. Finally, an adaptive logit adjustment (AL-adjustment) is introduced to remove long-tail distribution confounders to obtain unbiased predictions.
[0027] Compared with the prior art, the present invention has the following beneficial effects: the present invention takes long-tail distribution and semantic confusion as confounding factors of the structural causal model, and then divides the causal intervention into two stages. The first stage is causal representation learning, which uses an overall loss to intervene in the semantic confusion confounding factor. The second stage introduces adaptive logit adjustment to eliminate the long-tail distribution confounding factor to complete the causal calibration learning. Both stages are model agnostic and can therefore be used for any scene graph generation model that seeks unbiased predictions. Comprehensive experiments conducted on popular scene graph generation backbones and benchmarks show that the method of the present invention can achieve performance that is superior to the prior art in terms of average recall. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 Generate method schematics for existing scenarios;
[0029] Figure 2 This is a schematic diagram of a scene generation method according to an embodiment of the present invention.
[0030] Figure 3 This is a cause-and-effect structure diagram of an embodiment of the present invention;
[0031] Figure 4 A schematic diagram of the principle of obtaining the adjustment factor according to an embodiment of the present invention;
[0032] Figure 5 This is a flow chart of the training phase of an embodiment of the present invention;
[0033] Figure 6 This is a flow chart of the testing phase of an embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] The main goal of scene graph generation is to model the objects present in the scene and their pairwise relations. Most existing works first detect objects in the scene (e.g., “person”, “horse”) using an object detector and then identify pairwise relations between them (e.g., “riding”, “standing”) using a relation classifier. The object detector extracts information about objects such as their bounding boxes, categories, and features. The relation classifier then predicts relations for each pair of objects. Simply put, a scene graph is a visual triplet of the form <object 1, relation, object 2>. Formally, let Indicates that The observed data of samples, where x i is the i-th picture, is the N in the sample i Relationship, y ij It means x i J th Therefore, the embodiment of the present invention needs to use visual triples Labeled dataset To train the model, and are the classification and object bounding box, respectively. C and K are the number of object and relation categories in the observation data, respectively. Despite the high cost of annotating visual triplets, early efforts have contributed some benchmark datasets to the scene graph generation field, such as Visual Genome, Scene Graph, and Open ImagesV4. However, scene graph generation models trained on these datasets usually face two challenges: (1) semantic confusion, and (2) long-tail distribution.
[0036] In this work, the embodiments of the present invention address the above two challenges from the perspective of causal inference. Specifically, the embodiments of the present invention first consider the above two challenges, namely semantic confusion and long-tail distribution, as confounding factors of the standard scene graph generation framework (see Figure 1 ). Therefore, the method of the embodiment of the present invention uses the confounding factors at the data level to model the causal relationship for the scene graph generation task. Compared with the confounding factors at the model level, the method of the embodiment of the present invention is model agnostic, that is, it can be migrated to any scene graph generation model. Then, the embodiment of the present invention proposes a population loss to eliminate the semantic confusion confounding factors and obtain the decoupled factorization of the causal model (see Figure 2 Next, in Section 3.4, the present invention proposes AL-adjustment to eliminate the long-tail distribution confounding factors, thereby obtaining an unbiased prediction (see Figure 3 Step 2 in the above).
[0037] The unbiased scene graph generation framework proposed in the embodiment of the present invention can identify objects, attributes and relationships from images or videos, and construct a structured scene representation. This representation helps to deeply understand the image content and provide richer semantic information for the computer. The method proposed in the embodiment of the present invention can be applied to the following scenarios: 1) Image description and generation: Scene graph generation can provide rich image information and help to automatically generate image descriptions. On the other hand, information based on the scene graph can also guide the image generation model to generate images with specific structures and contents. 2) Visual question answering: Through scene graph generation, objects, attributes and relationships in the image can be converted into structured knowledge expressions, thereby helping the VQA system to answer questions about the image content more accurately. 3) Robot navigation and interaction: Scene graph generation technology can help robots understand objects and their relationships in their surroundings, so as to navigate and interact more effectively. For example, in order to avoid obstacles or find the correct target object in a picking task, the robot needs to understand the relative position, size and shape of the objects. 4) Video analysis and monitoring: In a video monitoring system, scene graph generation can help analyze the dynamic relationship between objects and the occurrence of events, thereby realizing real-time monitoring and event detection. In addition, this technology can also be used to automatically generate video summaries and extract key frames and key events. 5) Virtual Reality and Augmented Reality: In virtual reality and augmented reality applications, scene graph generation can provide support for generating realistic virtual environments and virtual objects that are integrated with the real world. Through the structured representation of the scene, a more natural interaction and realistic experience can be achieved. 6) Smart Home and Smart Building: Scene graph generation can help smart home and smart building systems understand objects and their relationships in the living environment, thereby achieving more precise control and optimization. For example, in energy management, scene graphs can help the system analyze indoor lighting, ventilation, and heating needs, thereby achieving a balance between energy saving and comfort. In short, the scene graph generation task has broad application prospects in the fields of computer vision and natural language processing, providing rich semantic information and scene understanding for various scenarios.
[0038] Example 1
[0039] The structural causal model modeling stage of the embodiment of the present invention is as follows Figure 3 shown.
[0040] People can use various frameworks to model causal relationships for their systems of interest, such as causal graph models, structural causal models, and potential outcome models. Structural causal models are limited in their causal modeling capabilities because they cannot support counterfactual inferences. Potential outcome models are active in systems with binary treatment variables, but are awkward when dealing with special treatment and outcome variables. Considering the limitations of CGM and PO, in this work, the embodiments of the present invention use causal graph models to model causal relationships, which is a structural method that includes variables, structural functions, and variable distributions (see Definition 1).
[0041] Definition 1 (Structural Causal Model). Structural Causal Model is a 4-tuple in is a set of exogenous variables; is a set of endogenous (observable) variables; Yes A set of structural functions; is the distribution of the exogenous variable.
[0042] Definition 2 (Sub-model). make for The set of variables in for A specific value of . is a 4-tuple: in and retain All other components in .
[0043] Endogenous variables are the basic elements of SCM. However, in the scene graph generation task, determining the variables It is very challenging because its input, i.e., images, is very different from the structured units in traditional causal discovery and reasoning tasks. Inspired by the misprediction of head / tail relationships in graphs, in an embodiment of the present invention, a model-independent data-level variable is proposed to treat semantic confusion and long-tail distribution as confounding factors. Therefore, the submodel can be defined for (See Definition 2). X is the input (image in scene graph generation task), Y is the output (relation), S is the semantic confusion confounding factor, and L is the long-tail distribution confounding factor; P(U X ,U γ ,U S ,U L ) is the distribution of exogenous variables. SCM is Figure 2 As shown, the structural equation is:
[0044] S=P(S)
[0045] L=P(L)
[0046] X=F1(L,P(L))+F2(S,P(S))
[0047] Y=F3(X,P(X))++4(L,P(L))+F5(S,P(S))
[0048] Intuitively, embodiments of the present invention can directly use interventions to eliminate confounding factors in SCM (see Definition 3). However, these interventions do not update Therefore, the intervention results are noisy causal effects in most cases.
[0049] Definition 3 (Intervention in SCM). In the intervention do(V i :=v′) by replacing the i-th structural equation with V i :=v′ to model, where v′ is the same as V i Irrelevant value.}
[0050] Definition 4 (Counterfactuals in SCM). In the counterfactual, the i-th structural equation is replaced by V i :v′ and update to model, where v′ has the same meaning as in Definition 3. The above counterfactual intervention induces the submodel
[0051] Assumption 1 (Insufficient Causality). In the exogenous variable Satisfies: P(U1,…,U n )≠P(U1)×P(U2)×…×P(U n ).
[0052] Fortunately, counterfactual reasoning, as the highest level reasoning process of cognitive ability, overcomes this limitation by imagining the results before and after the intervention (see Definition 4). It should be noted that counterfactual reasoning cannot be falsified because its imagined results cannot be observed. However, important designs in statistics, econometrics, and epidemiology (such as average treatment effects) can estimate counterfactual reasoning and have been proven to be effective. The main difference between intervention and counterfactual reasoning is that the latter is updated when the structural equations are manipulated. Therefore, embodiments of the present invention may partially seek intervention techniques to calculate counterfactual reasoning. Inspired by the above facts, embodiments of the present invention use counterfactual reasoning to eliminate the semantic confusion confounding factor S and the long-tail distribution confounding factor L to obtain an unbiased SCM for the scene graph generation task The counterfactual outcome can be calculated as:
[0053]
[0054] in is the expectation; S and L represent the relationship similarity space and the relationship long-tail space respectively; s and l represent the possible values, which are used to indicate whether they are similar and whether they are long-tail respectively; X and Y are the relationship between the input image and the output respectively; do(S:=s) and, do(L:=l)] respectively represent assigning S and L to s and l; and represents the expectation in the similarity space and the long-tail space; E[Y|X,s,l] represents the expectation of the output result Y under the condition that the similarity relationship is s and the long-tail relationship is l, ∑ s ∑ l represents the conditional sum under condition l; P(s) represents the probability when S takes the value s. Please note that causal sufficiency is the basic assumption of the above equation, that is, the exogenous variable U i are independent of each other: P(U1,…,U n )=P(U1)×P(U2)×…×P(U n ). Due to the causal sufficiency assumption, The endogenous variable v in can be expressed as a causal / decoupled factorization:
[0055]
[0056] Where pa(V i ) is V i The parent node of . In the scene graph generation task, the confounding factors can be observed confounding factors, such as semantic confusion confounding factors and long-tail distribution confounding factors, or unobserved confounding factors caused by missing labeled relations and incorrectly labeled relations. The latter has been discussed in many literatures. Therefore, due to the unobserved confounding factors, the embodiments of the present invention do not expect and cannot model a causally sufficient SCM for the scene graph generation task. Based on this, the embodiments of the present invention assume that is causally insufficient (see Assumption 1), so its endogenous variables can only be expressed as an entanglement factorization:
[0057]
[0058] In summary, the embodiment of the present invention uses counterfactual reasoning to eliminate semantic confusion and long-tail distribution confounding factors in the scene graph generation task, thereby obtaining an unbiased SCM. This will help the embodiment of the present invention to more accurately predict and understand the relationship between images in the scene graph generation task, and also provide a new perspective and method for further studying causal relationships.
[0059] Assumption 2 (Sparse Mechanism Shift). Small distribution changes tend to manifest themselves sparsely or locally in the causal / decoupled factorization, i.e., they do not typically affect all factors simultaneously.
[0060] Assumption 2 tells the present invention that for a decoupled factorization, sparse operations allow learners to remove confounding factors and even generalize to unseen distributions. Unfortunately, however, is causally insufficient because the scene graph generation task inevitably contains unobserved confounding factors, so it cannot benefit from the sparse mechanism transfer assumption. To address this challenge, the embodiment of the present invention divides causal modeling into two stages to achieve the goal of sparsely intervening endogenous variables:
[0061]
[0062] Among them, step 1 exploits the intrinsic sparse property of similar relations. Even under the assumption of insufficient causality, it can also achieve sparse perturbations on the variable S to remove semantically confusing confounding factors and learn Based on the obtained decoupling factor decomposition, the embodiment of the present invention operates the variable L in a sparse manner in step 2 to remove the long-tail distribution confounding factors.
[0063] The causal representation learning part of the embodiment of the present invention is introduced as follows.
[0064] In the scene graph generation task, similarity relations refer to relations that have only slight differences in visual and semantic features. However, existing scene graph generation models perform poorly in distinguishing these similarity relations, for example, it is easy to misjudge "standing" as "walking" and vice versa. This is not surprising, because even for humans, distinguishing these similarity relations is challenging. Naturally, people may be curious and imagine: if "standing" is misjudged as "walking" and is no longer similar, will the above error still occur? Although this only occurs in the imagination space of the embodiment of the present invention, it can be formally calculated by the counterfactual in the causal inference paradigm (see Definition 4):
[0065] P(y|x,do(S:=s))=P(y|x,do(S:=s1))-P(y|x,do(S:=s0)),
[0066] Where (x,y) is a specific value of (X,Y), s1 and s0 indicate that relation y is similar or dissimilar to other relations, respectively. In fact, the counterfactuals in the above simulate the potential outcomes of different interventions, i.e., do(S:=s1) and do(S:=s0). This is crucial because people can often benefit from imagination, for example, Einstein's thought experiment brought the theory of special relativity to the world. Although promising, computing the above requires a lot of cost. TDE is highly relevant to the work of embodiments of the present invention, which simulates two interventions by inferring pre-processing / post-processing inputs to obtain counterfactual results. However, it requires two model inferences for each input, which brings an unbearable cost. In contrast, the average treatment effect estimates the counterfactuals at one time by leveraging statistical knowledge. Inspired by ATE, embodiments of the present invention estimate counterfactuals using statistical knowledge of observed data:
[0067]
[0068] Definition 5 (Population in scene graph generation). Let y = {y1, y2, …, y K} is the observed data The relationship category in is the population of $y_i$. Then, is a string containing y i The relation set of the most similar α relations.
[0069] Formally, the embodiment of the present invention first extracts knowledge from the observation data It is the relationship i The statistical knowledge of is a population of relations that contains similar relations, see Definition 5. Inspired by the penalty head category in logit adjustment, the embodiment of the present invention uses the knowledge Penalize similar relationships. Specifically, the embodiment of the present invention abandons the widely used cross entropy loss and supervises the scene graph generation model through the proposed population loss:
[0070]
[0071]
[0072] Among them, π is the observed data The class frequency on θ * are the parameters used to parameterize the scene graph generation model. x and y are the input and output relation categories, respectively. For example, for relation y i , population loss in the observed data The statistical knowledge π and relational population extracted from The penalty term in the formula can be viewed as do(S:=s), because it intervenes in the sparse Make the model more able to distinguish similar relationships. In other words, population loss can eliminate The confounding factor S in . Therefore, the counterfactual can be obtained by the relationship population To estimate:
[0073]
[0074] In summary, the embodiment of the present invention proposes a method based on causal inference, which improves the performance of the scene graph generation model in distinguishing similar relationships by utilizing statistical knowledge in the observed data. The embodiment of the present invention first extracts knowledge about similar relationships from the observed data, and uses this knowledge to penalize similar relationships. The embodiment of the present invention proposes a population loss to supervise the scene graph generation model so that it is more capable of distinguishing similar relationships. This method has the advantages of causal inference in theory, can effectively eliminate confusion factors, and improve the performance of the scene graph generation model.
[0075] Assumption 3 (Similarity relations are sparse). It is observational data For any relation y i , there are k similar relations. Then, k<<K.
[0076] Although the goal of calculating counterfactuals is achieved, it is crucial to note that the insufficient causal assumption of the embodiments of the present invention determines that the manipulation do(S:=s) in the equation of the embodiments of the present invention may simultaneously disturb other variables, because The entanglement factorization does not satisfy the SMS assumption. Fortunately, in an embodiment of the present invention, the embodiment of the present invention empirically finds that similar relations have sparse properties (see Assumption 3). According to the observation of an embodiment of the present invention, the relations in the scene graph generation dataset are usually only similar to a few specific relations, rather than to most other relations. For example, "standing on" is only similar to "above...", "walking on", etc., but different from most other relations. Therefore, Assumption 3 shows that similar relations have the sparse property emphasized by SMS, and even entanglement factorization can perform sparse intervention on the confusion factor S. In other words, even if It is insufficiently causal. The population loss proposed in the embodiment of the present invention can still be intervened on S without worrying about disturbing other exogenous variables.
[0077] In addition, the embodiment of the present invention considers that do(S:=s) is Local disentanglement is performed because it eliminates the confusion factor S, enabling embodiments of the present invention to obtain a better causal representation. Therefore, the endogenous variables in the sub-model can be roughly expressed as:
[0078]
[0079] Disentanglement is considered to be the key to representation learning because of its potential in abstract reasoning, interpretability, generalization to unseen scenes, etc. Although it has received a lot of attention, evaluating disentangled representations remains challenging. In summary, by applying population loss, embodiments of the present invention improve the model's ability to discriminate in similar relations in scene graph generation tasks, while disentangling the confusion factors in causal representations.
[0080] The following shows how to calculate the relationship population for The embodiment of the present invention provides three assumptions (assumptions 4-6) based on the nature of causal relationships and the characteristics of relationships in the scene graph generation task.
[0081] Assumption 4 (Relation population is independent of learners). and are the scene graph generation models parameterized by θ1 and θ2 respectively. Then,
[0082] Assumption 5 (Relationship population is not sensitive to distribution). and are two observation data sets. Then,
[0083] Assumption 6 (The relationship population is asymmetric). Let y i and j are two relations. Then,
[0084] Assumption 4 indicates that whether two relations are similar or not has nothing to do with the scene graph generation model. For example, no matter what model the present invention uses, “standing on” and “walking on” should have the same features. No scene graph generation model should be used. Assumption 5 states that there is no correlation between the distribution of two relations and their similarity. This is emphasized in the embodiments of the present invention because the embodiments of the present invention observe that scene graph generation datasets often suffer from long-tail distribution problems at the relation and object levels, which may disrupt Assumption 6 is inspired by the fact that causality is directed, i.e., causation determines effect, but not vice versa.
[0085] To meet Assumption 4, the present invention designs a model-independent relational feature extraction method. Consider two objects o i and j , whose bounding boxes are and with o i As an example, is the center point, and are width and height. The embodiment of the present invention represents the model-independent features of the relationship between the two objects as
[0086]
[0087] The model-independent features proposed in the embodiments of the present invention emphasize the relative positions between pairs of objects. For example, the pair of objects "standing" is in an up-down relationship, while "behind" is in a front-back relationship. Thanks to the numerator of the equation, is position-insensitive because the top-left corners of all object pairs are moved to the same coordinates. In addition, the denominator of the equation normalizes the object pairs, ensuring that The position / scale insensitive design in the model-independent feature extraction method of the embodiment of the present invention can overcome the problem that the lens distance causes a huge change in the same relationship, thereby generalizing to unseen object pairs.
[0088] However, before using the above method to extract relational features in observational data, there is a problem that needs to be solved: the object-level long-tail distribution problem may disrupt the extraction of model-independent features. Consider an example in which 90% of the observations are <person, standing, road> and 10% are <person, standing, beach>. Fusion of the "standing" feature will undoubtedly favor <person, standing, road> because it dominates the observational data, which is not conducive to extracting the "standing" feature. The embodiment of the present invention extracts the object-level feature ξ y , and then normalize them to solve this problem. The method of the embodiment of the present invention is inspired by inverse probability weighting, which is a bias correction method commonly used in statistics, econometrics, epidemiology and other fields. Due to this improvement, the proposed method satisfies Assumption 5 because it eliminates the interference of distribution on feature extraction. Specifically, is a four-dimensional statistic:
[0089]
[0090] in It's a relationship <o i ,y t ,o j >, which can be calculated as:
[0091]
[0092] and The observation data are All relations <oi ,y t ,o j Then, the embodiment of the present invention calculates the features of each relationship. For example, for the tth relationship ξy t :
[0093]
[0094] For the relationship population y t The population can be calculated as:
[0095]
[0096] Among them, arg smalα is a computational kernel that selects similar relationships based on feature distance. For example, the above equation is The method of the embodiment of the present invention ensures that the head and tail relationship categories have the same population size, thereby satisfying assumption 6. Therefore, different relationships produce different feature distance thresholds in the above equation, resulting in an asymmetric relationship population.
[0097] The causal calibration learning phase of the embodiment of the present invention is introduced as follows.
[0098] There is a serious long-tail distribution problem in the scene graph generation task. Therefore, the current scene graph generation model can easily predict information-rich fine-grained relationships as coarse-grained relationships with less information. For example, "see" is predicted as "nearby". To solve this problem, let the embodiment of the present invention apply counterfactuals again: if balanced data is collected, or in particular, "see" and "nearby" have the same distribution, will the above error still occur? Obviously, this question can also be answered with counterfactuals:
[0099] P(y|x,do(L:=l))=P(y|x,do(L:=l1))-P(y|x,do(L:=l0)),
[0100] Where l1 and l0 represent the head and tail categories, respectively. Therefore, the above simulates the potential results of different interventions, namely do(L:=l1) and do(L:=l0). Inspired by the logit adjustment, in which the class prior knowledge (also called adjustment factor) extracted from the training data is used to adjust the model results, the embodiment of the present invention extracts statistical knowledge from the observed data through the model to estimate the counterfactual:
[0101]
[0102] Specifically, the embodiment of the present invention uses the extracted statistical knowledge to adjust the factor T β, to maximize the recall rate of the learned model on the observed data. Compared with the existing logit adjustment method, the adjustment factor of the embodiment of the present invention not only extracts knowledge from the observed data, but more importantly, it can adaptively adapt to the scene graph generation model. With this advantage, the adjustment factor T of the embodiment of the present invention is β Significantly better than traditional adjustment methods. Despite this, the knowledge extracted directly from the model and observation data is still not ideal. The embodiment of the present invention believes that this is because the background relationship interferes with the model training, resulting in 1) the logits of the foreground relationship becoming insufficiently distinguishable; 2) alternating positive and negative logits making it impossible for the learned factors to correctly adjust certain predictions. Among them, in the scene graph generation task, the foreground relationship is the relationship within the triplet annotated in the observation data, and the background relationship is the relationship that does not exist between pairs of objects. In order to overcome these problems, the embodiment of the present invention proposes a logit enhancement method, which can enhance the logits output by the model:
[0103]
[0104] in Yes θ The output is the logits of the corresponding background relations. Acting as a guide, it can enhance the logits More discriminative, which is inspired by the excellent performance of traditional adjustment methods in simple classification tasks. The enhanced logits allow embodiments of the present invention to obtain better adjustment factors, and for input x, its final prediction y x It can be calculated as:
[0105]
[0106] Consider a typical misprediction: for a class y that belongs to the tail i The largest and second largest output logits of input x correspond to y j and i , where y j However, the adjustment factor proposed in the embodiment of the present invention can correct this wrong prediction by penalizing the logits corresponding to the head category and encouraging the tail category, thereby eliminating the negative impact caused by the long-tail distribution problem.
[0107] The following shows how to generate a model from the observed data D and the scene graph Extracting statistical knowledge from β , this knowledge can be used to adjust the logits to eliminate the confounding factor L in the sub-model. For the adjustment factor T β , this embodiment of the present invention has two assumptions (assumptions 7-8).
[0108] Assumption 7 (Adjustment effects should be sparse). Let and are the adjustment factors for the ith and jth predicted logits, respectively. Then,
[0109] Assumption 8 (Adjustment factors should be independent of each other). Let and are the adjustment factors for the ith and jth predicted logits, respectively. Then,
[0110] Assumption 7 is inspired by the SMS hypothesis, which holds due to the decoupled factorization obtained in stage 1. This assumption also stems from the observation of embodiments of the present invention that in most cases, the incorrect predictions belong to the largest few logits. Therefore, assumption 7 allows embodiments of the present invention to correct incorrect predictions with sparse adjustment factors. However, existing methods adjust all logits. Assumption 8 views the SMS hypothesis through a relational hierarchy to highlight the causal relationship between relations. The intuition of this assumption is that to correct any incorrectly predicted logits for a binary classification task, embodiments of the present invention only need to adjust one of the two logits. Assumption 8 allows embodiments of the present invention to independently learn the adjustment factor for each relation.
[0111] The adaptive adjustment factor T proposed in the embodiment of the present invention β is a two-dimensional matrix:
[0112]
[0113] in Adjusting the jth maximum predicted logit to correspond to the ith relationship, it can be calculated as:
[0114]
[0115] in Calculate the number of true predictions made by the model on the dataset (e.g., recall (R@K) in scene graph generation tasks). is all the relations between the i-th category and the j-th maximum predicted logit obtained by model inference, which can be further divided into the true prediction and misprediction Therefore, the method of the embodiment of the present invention is to maximize the model f by adjusting the factors learned in the above equation. θ Recall on the observed data.
[0116] Next, the embodiment of the present invention proposes a method based on upper and lower bounds to quickly calculate the adjustment factor. Figure 4 As shown, for For each relationship in , the embodiment of the present invention can calculate the lower bound that maintains the correct prediction. Similarly, the embodiment of the present invention can Each relationship in obtains an upper bound that can be adjusted to a correct prediction. The lower and upper bounds of and Obviously, the embodiment of the present invention only needs to allow Satisfy the most bounds to maximize the number of correct predictions. Therefore, by adjusting the factor to maximize the model The recall rate on the observed data is equivalent to and Another task is to find the factor that satisfies most bounds. Therefore, the adjustment factor can also be calculated as follows:
[0117]
[0118] in is an indicator function (equal to 1 when the expression is true and equal to 0 when it is false), and |·| is the length / size of a given set / list. However, the present invention further finds that the long-tail distribution problem may disrupt Specifically, if y i is a head category, If it is a tail category, then This is biased training due to skewed distribution. To address this problem, for the relationship y i , the embodiment of the present invention randomly extracts the same number of bounds from the original lower bound / upper bound. Therefore, the adjustment factor can be further calculated as:
[0119]
[0120] Finally, for each relation, the embodiment of the present invention only learns β adjustment factors corresponding to the 1-β positions in the predicted logits. Therefore, this sparse adjustment mechanism enables the method of the embodiment of the present invention to meet assumption 7. At the same time, the adjustment factor of each relation is learned independently through the above equation, so the method of the embodiment of the present invention meets assumption 8.
[0121] Example 2
[0122] Embodiment 2 of the present invention provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device may be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.
[0123] The terminal device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in the above-mentioned embodiment 1.
[0124] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.
[0125] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.
[0126] Example 3
[0127] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to the above embodiment 1, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the above embodiment 1 are implemented.
[0128] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.
[0129] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0132] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0133] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. An unbiased scene graph generation method, It is characterized in that The following steps are involved: S1. Using visual triples Annotated image dataset, where Represent the classification of two objects respectively. Represents the bounding boxes of the two objects, N D is the number of observations in the sample, is the N in the sample i relations, where N i is the number of relations in the sample, and K is the number of relation categories in the observed data; S2, extracting relationship populations from the observation data of the labeled image dataset; S3, training a machine learning model using the relationship population; S4, extracting adjustment parameters from the observed data and the trained model; S5. Using the adjustment parameters to adjust the model output and generate an unbiased scene graph.
2. The unbiased scene graph generation method according to claim 1, It is characterized in that In step S2, for the tth relationship category y in the observation data t , relationship population It is expressed as: Among them, arg smalα is the calculation kernel, which means selecting similar relationships based on feature distance. To represent the t′th relationship category y t′ The relationship characteristics, and They are all the relations in the observed data. <o i ,y t ,o j > is the fusion feature and quantity, C is the number of objects in the observation data, o i and j They are two different objects.
3. The unbiased scene graph generation method according to claim 1, It is characterized in that In step S4, the adjustment parameters The expression is: Among them, y i and l j denote the i-th relationship and the j-th position of the logit respectively; and Respectively represent the relationship y i Corresponding to the first j Adjustment of the lower and upper bound combinations of positions; Indicates the mth value of adjusting the lower bound combination, represents the nth value of the upper bound of the adjustment; T represents the potential space of the adjustment factor, and t represents the possible values in the potential space. is the indicator function.
4. The unbiased scene graph generation method according to claim 1, It is characterized in that The loss function of the machine learning model is expressed as: Among them, x and y are the input and output relation categories respectively; It is the relationship i statistical knowledge, is the statistical knowledge of input y; π y represents the quantity distribution of input y, π y′ represents the number distribution of input y′; f y (x) and f y′ (x) represents the logit model that predicts y and y′ when the input is x, respectively.
5. The unbiased scene graph generation method according to claim 4, It is characterized in that The parameters θ of the machine learning model * The expression is: in, is the observed data, is the expectation, θ is the potential parameter space of the machine learning model, and f(x) represents the output after inputting x into the machine learning model f.
6. A terminal device, It is characterized in that include: one or more processors; A memory having one or more programs stored thereon, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the steps of the method according to any one of claims 1 to 5.
7. A computer-readable storage medium, It is characterized in that It stores a computer program, which implements the steps of the method according to any one of claims 1 to 5 when executed by a processor.
Citation Information
Patent Citations
Unbiased scene graph generation method based on hierarchical structure
CN115269925A
Deviation reduction scene graph generation method and system oriented to data double imbalance
CN115546626A
Unbiased scene graph generation method based on double-branch hybrid learning network
CN115565052A
Unbiased dynamic scene graph generation method based on multi-task learning framework
CN115565098A
Cited By
Unbiased scene graph generation method for relieving long-tail distribution
CN120526254A
