Scene graph generation method based on mutual viewing angle inverse relationship calibration and hypernym guiding prompt

Abstract

This invention discloses a scene graph generation method based on mutual perspective inverse relation calibration and hypernym guidance prompts. First, it establishes the principle of mutual perspective inverse relations and utilizes an adaptive inverse relation enhancement strategy. Through two paths—logical sample expansion and random logical permutation—the inverse perspective logic is internalized into the training process. For large-scale visual language models, hypernym guidance prompts are introduced, abstracting fine-grained entities into a coarse-grained set of hypernyms. This adaptively constructs text features containing inverse logic in the latent space, making the generated representations more consistent with the invariance of mutual perspectives. Finally, the cross-modal mutual perspective representations are jointly optimized using a cross-entropy objective function, serving as the final model prediction parameters. This invention significantly improves the inference robustness of scene graph generation models when facing long-tailed distributed data and complex interactive scenarios, effectively solving the problems of inaccurate identification of rare relations and semantic ambiguity caused by perspective switching.

Description

Technical Field

[0001] This invention relates to computer vision technology, specifically a method for generating scene graphs based on mutual perspective inverse relation calibration and hypernym guidance prompts. Background Technology

[0002] Scene graph generation aims to resolve visual scenes into a graph structure composed of nodes (objects) and directed edges (relational predicates), serving as the cornerstone of advanced reasoning tasks such as image retrieval and visual question answering. Existing technological paradigms rely on object detectors to identify entities in images, employing methods such as message passing and tree models to discriminate and classify relationships between entities. Currently, some more innovative methods are gradually exploring the use of cue-based optimization techniques to introduce large-scale visual language models to enhance relationship recognition capabilities, especially in open semantic contexts.

[0003] However, existing methods have two major, mutually restrictive drawbacks:

[0004] (1) Severe one-way bias: Existing models model it as a strict one-way mapping from subject to object ( This approach ignores the inherently reciprocal nature of visual interactions (physical facts). When the anchor perspective of reasoning shifts from the subject to the object, the model often produces semantically broken or logically contradictory predictions. This is also closely related to the annotation bias inherent in the dataset itself; when labeling relations, typically only a subset of active and salient relations are labeled, affecting the model's understanding and recognition of inverse relations. Almost all existing scene graph generation technologies ignore the problem of one-way bias.

[0005] (2) Context Loss and Combinatorial Explosion Dilemma in Cue Learning: When introducing large models to handle the above complex relationships, if simple short cue templates (such as those containing only the predicate) are used, the identities of the subject and object closely related to the predicate will be lost, causing the large model to be unable to distinguish the anchor point and direction described by the predicate; while if complete triple templates are enumerated to provide context, it will cause combinatorial explosion in the search space, with too many possible combinations between different subjects and objects, resulting in computational complexity reaching This will make model optimization computationally infeasible.

[0006] For example, in similar technical solutions in the field of scene graph generation, some solutions only use the most basic prompt templates, making it impossible for the model to recognize the two fixed contexts of subject and object, resulting in serious deviations and errors in the predicate reasoning stage; some solutions experimentally introduced subject-object and long prompt templates during the experiment, which caused problems such as memory overflow. Summary of the Invention

[0007] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a scene graph generation method based on mutual perspective inverse relation calibration and hypernym guidance prompts. The method calibrates the inference logic of the model through mutual perspective constraints and innovatively proposes hypernym guidance prompts, effectively overcoming the computational explosion problem faced when directly applying visual cues optimization to complex scene graphs.

[0008] Technical solution: The present invention provides a method for generating scene graphs based on mutual perspective inverse relationship calibration and hypernym guidance prompts, comprising the following steps:

[0009] Step 1: For the visual scene image to be processed Define the principle of mutual perspective inverse relationship, for each standard triplet relationship in the image. Construct logically equivalent inverse triples Output the inverse relation of the original relation that conforms to the semantic logical classification; where, , Subject, As an object, As a predicate, It is a reverse predicate;

[0010] Based on the relation classification rules in the field of scene graph generation, a large language model is used to classify inverse relations based on the semantic and logical attributes of predicates. The predicate space is divided into symmetric relations and asymmetric relations, and asymmetric relations are further subdivided into three superclasses: semantic relations, geometric relations, and possession relations to adapt to the inverse mapping rules. Here, only the relations are classified, which is essentially a definition or rule constraint. The result can be regarded as an inverse mapping rule.

[0011] Step 2: Based on the inverse relation that conforms to the semantic logic classification in Step 1, an adaptive inverse relation enhancement calibration strategy is adopted to dynamically apply mutual perspective constraints to calibrate the model.

[0012] The filtering mechanism is selected based on the architecture type of the baseline model, including the following two filtering mechanisms: For closed-set models constrained by a fixed vocabulary, a validity filter is applied, adding an inverse sample only if the generated inverse predicate exists in the predefined vocabulary; otherwise, the original sample data is retained to filter out noise outside the distribution. For open-set models without vocabulary constraints, inverse mapping is applied without restriction. Then, for semantic relations or tail-class predicates, a logical sample expansion strategy is applied to supplement the training set with the generated valid inverse triples; for geometric relations and head-class predicates, a random logical permutation strategy is applied to probability-wise permutation of the original triples. Replace with a valid inverse triplet;

[0013] The baseline model in this invention refers to various standard basic network architectures that have been widely adopted in the field of scene graph generation. The relevant modules of this invention have highly decoupled "plug and play" characteristics, which can be seamlessly mounted on any baseline model and directly endow it with bidirectional logical cognition and reasoning calibration capabilities without changing its original underlying topology.

[0014] Step 3: Introduce hypernym guidance prompts to the visual language model to construct a dual-view context. Utilize a large language model to abstract and map fine-grained entity categories to a small-scale, coarse-grained set of semantic hypernyms. In the middle; combining learnable context parameters, the positions of the hypernyms of the subject and object are explicitly swapped, and original perspective prompt templates are generated respectively. With reverse perspective prompt template This is to alleviate the combinatorial explosion problem caused by the enumeration of prompt templates;

[0015] Step 4: Use a pre-trained visual language large model for joint training and recognition prediction. Calculate the similarity between the constructed dual-view cue vector and the image visual features in the latent space, perform cross-entropy joint optimization, and finally output a multi-label scene graph generation result with cross-view consistency.

[0016] Furthermore, in step 1, the scene diagram is represented as follows: Node set Representative entities, edge sets Representational relationship; the principle for defining mutual perspective relationships is as follows: any objective interaction must exist independently of the observation anchor point, if physical facts... If true, then the mapping is satisfied: This establishes the logic of bijective transformation. The predicate space is divided into symmetric relations ( ) and asymmetric relations ( For example, asymmetric relationships can be classified into semantic relationships (such as "riding" mapping to "ridden by"), geometric relationships (such as "above" mapping to "below"), and possession relationships (such as "belonging to" mapping to "own").

[0017] Next, adaptive data augmentation techniques are used to transform the mutual perspective inverse relation principle into training signals; considering the long-tail distribution and information density of predicates (semantic relations are rich in information, while geometric relations have low information density and high redundancy), two calibration paths are specifically adopted:

[0018] (1) Path A (Logical Sample Expansion): For semantic categories or tail category relations with high information density but extremely scarce information, this path expands the original dataset (i.e., the triples in step 1). Based on this, we expand upon it by adding data with inverse relationships, that is, introducing inverse triples as incremental monitoring signals. For such effective triples... Generate inverse triplet And expanded to the training set middle:

[0019] ;

[0020] (2) Path B (Random Logical Permutation): For geometric relationships and head categories with high redundancy and prone to overfitting positional deviations, this path performs random logical permutations with a certain probability, using inverse relational data as perspective perturbations to enhance the model's inference ability. This is achieved by sampling perturbation triplets using a Bernoulli distribution. Finish:

[0021] ;

[0022] in, obey This refers to the Bernoulli distribution. Here, we introduce a control variable that follows a Bernoulli distribution. , where parameters This represents the probability threshold for triggering enhancement (e.g., it can be set to 0.5). In each training iteration, the system obtains random values ​​through sampling to determine whether the current sample should execute a "random logical permutation" strategy based on reverse logic or retain the original data. This decision-making mechanism based on a stochastic process ensures that while retaining some original positive perspective features, the model can be regularized in a controlled proportion through the reverse perspective, thereby forcing the invariance of mutual perspectives in the latent space and effectively preventing the model from overfitting to a single training distribution.

[0023] In traditional cue-based learning methods, enumerating all full triplet templates leads to an excessively large search space. However, simple templates lose the subject-object context; to solve these problems, the specific method for constructing hypernym guidance prompts in step 3 is as follows:

[0024] Using a large language model to refine fine-grained entity sets Mapping to a small set of hypernyms (satisfy The projection function is defined as follows: ;

[0025] Based on the above mapping relationship, fusion A learnable continuous context vector For each interaction, construct a pairing cue between the original viewpoint and the reverse viewpoint:

[0026] (1) Original perspective prompt template: ;

[0027] (2) Reverse perspective prompt template: ;

[0028] in, This represents the fixed text word embedding vectors of the pre-trained visual language model. The hypernym guidance prompting mechanism structurally aligns the text prompts with the visual inverse transformation.

[0029] Next, vocabulary-aware adaptive filtering is performed: when executing the above path, the rules for adopting inverse relations need to be determined based on whether the model is a closed or open set; for open set models (such as the architecture of the existing large visual language model CLIP), mappings are applied without restriction to introduce new inverse predicates; for models limited by a fixed predefined vocabulary... In this invention, a validity filter is innovatively applied to the closed-set model, that is, the generated inverse triples are only applied to their inverse predicates. They are only injected or replaced when needed, which ensures that the classification performance of the closed-set model will not collapse due to the introduction of predicate labels outside the distribution.

[0030] Furthermore, the detailed method for step 4, cross-modal recognition prediction and joint model optimization, is as follows:

[0031] Joint optimization of cross-view features is performed based on the cross-entropy function, on a calibration dataset enhanced by an adaptive inverse relation enhancement strategy. The goal is to minimize the expected negative log-likelihood loss. :

[0032] ;

[0033] Among them, symbols Characterizes the mathematical expectation operation on the calibration dataset augmented by the adaptive inverse relation enhancement strategy; variables The label represents the real physical association fact corresponding to the current input subject-object triplet sample, i.e., the labeled real relation predicate.

[0034] For the large open-set visual language model adopted, combined with visual features Embedded with text prompts generated through hypernym guidance The probability of relationship classification is calculated using the temperature-scale cosine similarity between the two values. The expression is:

[0035] ;

[0036] in, The function represents the learnable temperature scaling parameter. Defined as a measure of image visual features The cosine similarity function is used to calculate the degree of matching between text spatial features; set Represents the predefined vocabulary space of relational predicates; variables Specifically refers to the target relation category that the current model is positively predicting; subscript To traverse the set of predicate vocabularies Index parameters for all potential relation categories; correspondingly, variables Used to indicate in a predefined vocabulary within, no. Each category is mapped to a corresponding text cue embedding vector after being constructed using a cue template. These variables work together to form a complete logical framework for aligning bidirectional cue vectors with visual features in the latent space, allowing the model to output a scene graph with bidirectional consistency.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0038] (1) In response to the inherent one-way bias problem in scene graph generation, this invention corrects the model by using a unique mutual perspective inverse relationship principle and an adaptive inverse relationship enhancement strategy, thereby avoiding the collapse of the prediction perspective and significantly enhancing the two-way logical self-consistency of the model at the cognitive level.

[0039] (2) In response to the complexity dilemma of introducing large visual models for prompt optimization, the present invention innovatively designs a hypernym guidance prompt module, which extracts the functional features of hypernyms and successfully compresses the huge prompt search space of O(|V|^2×|P|), taking into account both the semantic accuracy of relational direction and the computational efficiency of training inference.

[0040] (3) As a plug-and-play module, this invention is not affected by factors such as the relation headers used by the model or the specific implementation methods. Moreover, since this solution mainly works in the data augmentation and hint construction stage, it does not require modification of the underlying architecture of the model. Therefore, it can be integrated into any scene graph generation task as a general plugin, and has extremely high commercial and application value.

[0041] (4) This invention has strong generalization and plug-and-play characteristics. The combination of cross-perspective inverse relation calibration and hypernym guidance prompts can be seamlessly integrated into the open set scene graph generation architecture without changing the underlying backbone network structure, and it has shown leading performance in both reverse stress test and long-tail distribution test. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0043] Figure 2 This is the overall network model structure and processing flowchart of the present invention.

[0044] Figure 3 This is a schematic diagram of the inverse relation classification mechanism and inverted mapping logic proposed in this invention.

[0045] Figure 4 This is an example diagram of a guiding prompt template for predicate superclass classification using a large language model, as described in this embodiment of the invention. Detailed Implementation

[0046] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0047] To standardize the expression, the key concepts of this invention are further explained in detail below: The expression regarding "mutual perspective inverse relationship" in this invention directly corresponds to the "physical fact invariance" of interactive attributes. For example, in a visual scene, the visual fact "man holding surfboard" logically necessarily accompanies its inverse state "surfboard held by man." The expression regarding "superordinate word guidance and prompting" in this invention directly corresponds to the "functional semantic abstraction" of subject and object entities in a zero-sample visual large-scale model. For example, biologically different "man" and "woman" share the superordinate word role of "person" in interactive behaviors such as riding and holding. By abstracting it into a superordinate word, it can effectively guide the large model to perceive subject-object reversal in the prompting space, avoiding the collapse of computational combinations of a large vocabulary.

[0048] like Figure 1 As shown, the scene graph generation method based on mutual perspective inverse relation calibration and hypernym-guided prompts of the present invention includes the following steps: acquiring visual scene images and establishing mutual perspective inverse relation principles; dividing the predicate space into symmetric and asymmetric relations; adopting an adaptive inverse relation enhancement strategy; selecting two methods—application logic sample expansion and random logic permutation—according to different categories; introducing a large language model for hypernym abstraction mapping; projecting entities as hypernyms to alleviate combinatorial explosion during prompt construction; exchanging subject and object at the text prompt encoding end; adaptively constructing paired prompts containing inverse logic for each interaction relation; and using CLIP to jointly optimize bidirectional prompt vectors and visual features to collaboratively output a scene graph with inverse relation recognition capabilities. Specifically, the method includes the following steps:

[0049] Step 1: For the visual scene image to be processed Define the principle of mutual perspective inverse relationship, for each standard triplet relationship in the image. Construct logically equivalent inverse triples Output the inverse relation of the original relation that conforms to the semantic logical classification; where, , Subject, As an object, As a predicate, It is a reverse predicate;

[0050] Based on the relation classification rules in the field of scene graph generation, a large language model is used to classify inverse relations based on the semantic and logical attributes of predicates; the predicate space is divided into symmetric relations and asymmetric relations, and asymmetric relations are further subdivided into three superclasses: semantic relations, geometric relations and possession relations, thereby adapting to the inverse mapping rules;

[0051] Step 2: Based on the inverse relation that conforms to the semantic logic classification in Step 1, an adaptive inverse relation enhancement calibration strategy is adopted to dynamically apply mutual perspective constraints to calibrate the model.

[0052] The filtering mechanism is selected based on the architecture type of the baseline model, including the following two filtering mechanisms: For closed-set models constrained by a fixed vocabulary, a validity filter is applied, adding an inverse sample only if the generated inverse predicate exists in the predefined vocabulary; otherwise, the original sample data is retained to filter out noise outside the distribution. For open-set models without vocabulary constraints, inverse mapping is applied without restriction. Then, for semantic relations or tail-class predicates, a logical sample expansion strategy is applied to supplement the training set with the generated valid inverse triples; for geometric relations and head-class predicates, a random logical permutation strategy is applied to probability-wise permutation of the original triples. Replace with a valid inverse triplet;

[0053] Step 3: Introduce hypernym guidance prompts to the visual language model to construct a dual-view context. Utilize a large language model to abstract and map fine-grained entity categories to a small-scale, coarse-grained set of semantic hypernyms. In the middle; combining learnable context parameters, the positions of the hypernyms of the subject and object are explicitly swapped, and original perspective prompt templates are generated respectively. With reverse perspective prompt template ;

[0054] Step 4: Use a pre-trained visual language large model for joint training and recognition prediction. Calculate the similarity between the constructed dual-view cue vector and the image visual features in the latent space, perform cross-entropy joint optimization, and finally output a multi-label scene graph generation result with cross-view consistency.

[0055] This invention significantly improves the robustness of scene graph generation models in reasoning when faced with long-tailed distributed data and complex interactive scenarios. It effectively solves the problems of inaccurate identification of rare relationships and semantic ambiguity caused by perspective switching. The resulting scene graph can be widely used in downstream tasks, providing support for them. For example, it has the potential to help robot environmental perception, autonomous driving visual semantic understanding, intelligent security monitoring, and natural language-based image retrieval, providing accurate, stable, and mutually inversely logically consistent semantic topological support for high-level visual reasoning tasks.

[0056] like Figure 3 As shown, step 1 of this embodiment establishes a bijective transformation logic based on the interactive and reciprocal nature of physical facts. Construct its logically equivalent inverse triplet For scene diagrams Node set Representative entities, edge sets Representational relationship; the principle for defining mutual perspective relationships is as follows: any objective interaction must exist independently of the observation anchor point, if physical facts... If true, then the mapping is satisfied: This establishes the logic of bijective transformation. The predicate space is divided into symmetric relations and asymmetric relations.

[0057] The expression for the symmetry relation is: The predicate remains unchanged after reverse mapping. (e.g., "near" and "and"); when the roles of subject and object are reversed, the predicate used to describe the relationship between the subject and object remains unchanged.

[0058] The expression for asymmetric relations is: Inverse mapping requires specific semantic inversions; these include three superclasses: semantic (e.g., action interaction), geometric (e.g., spatial location), and possessive (e.g., membership). These categories require specific semantic inversion rules to maintain the stability of physical facts.

[0059] Table 1. Some predicate superclasses and their mapping logic

[0060] predicate superclass Typical predicate (p) <![CDATA[Corresponding inverse predicate (p −1 )]]> Transformation Logic Semantics (SE) riding, holding ridden by, held by Active / passive switching Geometry (GE) on, above, behind under, below, over Spatial Dual Flip Possession (PO) has, belonging to part of, owning Overall-to-part role reversal Symmetry (SY) near, and near, and Identity mapping

[0061] To internalize the inverse logic into the model training process through dynamic data calibration, step 2 employs an adaptive inverse relation enhancement calibration strategy to transform the mutual perspective inverse relation principle into training signals. This enables differentiated data augmentation based on the attributes of different training data. The differentiated augmentation path helps identify different attributes, primarily focusing on the frequency of data occurrence (long-tail distribution) and superclasses. Blindly performing the same addition or flipping operations on all samples would exacerbate inherent biases such as the long-tail distribution and information imbalance of the data itself. The adaptive inverse relation enhancement strategy, based on the position of the predicate in the long-tail distribution (Head, Body, Tail) and its semantic features, specifically divides into two calibration paths, considering the long-tail distribution and information density of the predicate:

[0062] (1) Path A, namely logical sample expansion, is to expand the original dataset based on semantic categories or tail category relationships that are high in information density but extremely scarce, by adding data on inverse relationships, that is, introducing inverse triples as incremental supervision signals. For such effective triples Generate inverse triplet And expanded to the training set middle, The expression is:

[0063] ;

[0064] Path A is equivalent to adding data with inverse relationships. It neither replaces the original data nor does it replace the rich information of low-frequency or semantic classes of tail relationships. It also reduces the proportion of head and low-information relationships to a certain extent, which can significantly alleviate the problem of insufficient supervision signals caused by long-tail distribution and enhance the model's cognitive depth of complex action interactions.

[0065] (2) Path B, i.e., random logical permutation, is used to perform random logical permutation with corresponding probabilities for geometric relationships and head categories that are highly redundant and prone to overfitting positional deviations. The inverse relationship data is used as a viewpoint perturbation and the perturbation triples are sampled through Bernoulli distribution. Completed, the expression is as follows:

[0066] ;

[0067] Among them, control variables obey This is the Bernoulli distribution. This mutually exclusive sampling mechanism can effectively break the model's statistical dependence on the spatial arrangement of objects, forcing the model to learn more discriminative visual features. In addition, it also adjusts the data distribution to a certain extent, alleviating the data distribution bias problems of long-tailed distribution and information imbalance.

[0068] like Figure 2As shown, the specific method for constructing the hypernym guidance prompt in step 3 of this embodiment is as follows:

[0069] Using a large language model to refine fine-grained entity sets Mapping to a small set of hypernyms ,satisfy The projection function is defined as follows: ;

[0070] Based on the above mapping relationship, fusion A learnable continuous context vector For each interaction, construct a pairing cue between the original viewpoint and the reverse viewpoint:

[0071] The expression for the original perspective hint template is: ;

[0072] The expression for the reverse perspective hint template is: ;

[0073] in, This represents the fixed text word embedding vectors of the pre-trained visual language model. The hypernym guidance prompting mechanism structurally aligns the text prompts with the visual inverse transformation.

[0074] Next, adaptive filtering based on word awareness is performed:

[0075] When executing the above path, the rules for adopting inverse relations need to be determined based on whether the model is a closed or open set. For open set models, mappings are applied without restriction to introduce new inverse predicates; for closed set models, a validity filter is applied, meaning that the generated inverse triples are only applied to their inverse predicates. It is only injected or replaced when needed.

[0076] The detailed method for step 4 of this embodiment, cross-modal recognition prediction and joint model optimization, is as follows:

[0077] Joint optimization of cross-view features is performed based on the cross-entropy function, on a calibration dataset enhanced by an adaptive inverse relation enhancement strategy. The goal is to minimize the expected negative log-likelihood loss. :

[0078] ;

[0079] Among them, symbols Characterizes the mathematical expectation operation on the calibration dataset augmented by the adaptive inverse relation enhancement strategy; variables The label represents the real physical association fact corresponding to the current input subject-object triplet sample, i.e., the labeled real relation predicate.

[0080] For the large open-set visual language model adopted, combined with visual features Embedded with text prompts generated through hypernym guidance The probability of relationship classification is calculated using the temperature-scale cosine similarity between the two values. The expression is:

[0081] ;

[0082] in, This represents the learnable temperature scaling parameter. (Function) Defined as a measure of image visual features The cosine similarity function is used to calculate the degree of matching between text spatial features; set Represents the predefined vocabulary space of relational predicates; variables Specifically refers to the target relation category that the current model is positively predicting; subscript To traverse the set of predicate vocabularies Index parameters for all potential relation categories; correspondingly, variables Used to indicate in a predefined vocabulary within, no. Each category is mapped to a corresponding text cue embedding vector after being constructed using a cue template. These variables work together to form a complete logical framework for aligning bidirectional cue vectors with visual features in the latent space.

[0083] The scene graph generation method based on mutual perspective inverse relationship calibration and hypernym guidance in this embodiment has the following core system architecture and processing flow: Figure 2 As shown, the network modalities of this invention enforce bidirectional logical constraints on physical facts at the feature level through mutual perspective inverse relationship calibration. For example... Figure 2 As shown in the lower half, the cross-view inverse relation calibration is performed on the triples extracted from the input image, based on... Figure 3 The classification mechanism shown identifies relationships as symmetric or asymmetric (semantic, geometric, possessive) relation categories. The model then executes an adaptive inverse relation enhancement strategy: logical sample augmentation is performed on relations belonging to the "semantic class (Se)" or "tail class (T)" to enrich the sparse interaction labels; random logical permutations are performed on relations belonging to the "geometric class (Ge)" or "head class (H)" with a probability (e.g., set to 0.5), thereby weakening the statistical bias caused by fixed positions and high-frequency words. Figure 2 As shown in the upper part, in the cross-modal alignment stage, hypernym-guided prompts are used to generate paired bidirectional contextual text, and visual features and linguistic features are aligned by multiplicative similarity, thereby generating a robust scene graph network.

[0084] To execute the above logic accurately, such as Figure 3As shown, this embodiment divides predicates into symmetric and asymmetric relations. Asymmetric relations are further divided into semantic, geometric, and possessive categories, and clear reverse inversion inference rules are formulated for each.

[0085] To avoid subjective bias in manual definitions and improve scalability, this embodiment utilizes a large language model for knowledge distillation: such as Figure 4 As shown, a guiding prompt template for predicate superclass partitioning was designed, which guides the model to output a stable classification by providing reference anchor points. Figure 4 By providing reference anchors and strict definitions, the large model is guided to stably output predicate classification to guide subsequent adaptive inverse relation enhancement. It is divided into sections such as role assignment, task setting, reference examples, constraints, and input / output specifications.

[0086] To quantitatively verify the superiority of the technical solution of this invention, this embodiment conducted detailed experimental tests on the classic "Visual Genome (VG)" scene graph dataset. In the experimental evaluation, this invention organically combines mutual perspective inverse relation calibration with hypernym-guided prompts, demonstrating significant performance advantages. For example, in CLIP-based backbone testing, it not only significantly reduces the computational load of parameter tuning but also achieves an excellent balance in traditional recall (R@K) and mean recall (mR@K) metrics. Crucially, in the "Inverse Consistency (IR@K)" protocol test specifically designed for unidirectional bias, the mutual perspective model with hypernym-guided prompts was fully introduced. This means that this invention achieved the highest level of performance, showing a significant improvement over traditional visual models and completely avoiding visual blind spots and logical collapses during inference anchor point switching.

[0087] The above experimental results fully demonstrate the dual superiority of the technical solution of this invention in terms of complex cognition and computational compression. Specific experimental data tables and related explanations are as follows:

[0088] Table 2 Overall performance across architectures

[0089] As shown in Table 2, the calibration strategies of this invention (closed-set expansion / replacement, open-set expansion / replacement) exhibit high plug-and-play compatibility with classic baseline models such as Motifs, VCTree, and PE-Net, as well as the CLIP open-set variant. This is particularly true for the open baseline CLIP, which incorporates hypernym guidance. MIn the model, combined with the open set replacement strategy of this invention, the model achieved a recall average score of 43.9% / 45.2% on the comprehensive index M@50 / 100, which is significantly better than the original baseline of 39.9% / 42.5%, proving that this invention effectively solves the problems of one-way bias and long-tail distribution.

[0090] Table 3. Verification data of mutual perspective reverse cognition ability

[0091] To evaluate the model's ability to handle "mutual perspective cognition," this invention innovatively proposes the reverse recall rate (IR@K) and the reverse comprehensive index (MI). As shown in Table 3, traditional baseline models (such as Motifs) have an MI@100 of only 4.4% in reverse perspective, indicating that the model faces significant performance degradation when perspectives are reversed and cannot effectively perform reverse semantic reasoning. However, after introducing the framework of this invention, CLIP... M The MI@100 of the (open set expansion strategy) is improved to 36.4%. This proves that the proposed scheme significantly improves the model's ability to reason about physical common sense, enabling it to understand perspective invariance.

[0092] Table 4. Selection and Optimization of Hypernym Guidance and Prompt Model

[0093] As shown in Table 4, after comparing various large language models such as GPT-4o and DeepSeek-V3, the strategy of "Gemini-pro + reasoning enhancement" achieved the best overall score of 43.1% / 44.7% (M@50 / 100). This indicates that eliminating common sense conflicts through reasoning enhancement of large models can maximize the accuracy of feature projection, thereby significantly alleviating the problem of cue combinatorial explosion.

[0094] Table 5. Scope of Application of Adaptive Inverse Relationship Enhancement Strategy

[0095] This embodiment designs expansion strategies (Add) and replacement strategies (Rep) for different predicate categories. As shown in Table 5, ablation experiments demonstrate that blindly applying the same strategy globally does not yield the best results. For the expansion strategy, when applied only to the information-sparse "tail + semantic class (T&Se)," the model achieves the highest performance metrics of 48.8% / 51.2%; while for the replacement strategy, the best performance is achieved when applied only to the highly redundant "head + geometric class (H&Ge)," reaching 48.7% / 51.0%. These experimental results highlight the scientific rigor and advancement of this invention's adaptive dynamic calibration based on data attributes.

[0096] In summary, compared with traditional one-way mapping scene graph generation methods, this invention starts directly from the reciprocity of physical interactions and completely changes the problem of semantic collapse of the model when the reasoning perspective changes due to the inherent "one-way bias" in existing methods through mutual perspective calibration. Compared with directly using full triple enumeration for visual language large model prompt learning, this invention uses "hypernyms" extracted from the large language model for semantic abstraction, which effectively alleviates the combinatorial explosion problem in the prompt template construction process, and preserves the precise context of the subject and object while significantly reducing the search space.

[0097] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for generating scene graphs based on mutual perspective inverse relation calibration and hypernym guidance prompts, characterized in that, Includes the following steps: Step 1: For the visual scene image to be processed Define the principle of mutual perspective inverse relationship, for each standard triplet relationship in the image. Construct logically equivalent inverse triples Output the inverse relation of the original relation that conforms to the semantic logical classification; where, , Subject, As an object, As a predicate, It is a reverse predicate; Based on the relation classification rules in the field of scene graph generation, a large language model is used to classify inverse relations based on the semantic and logical attributes of predicates; the predicate space is divided into symmetric relations and asymmetric relations, and asymmetric relations are further subdivided into three superclasses: semantic relations, geometric relations and possession relations, thereby adapting to the inverse mapping rules; Step 2: Based on the inverse relation that conforms to the semantic logic classification in Step 1, an adaptive inverse relation enhancement calibration strategy is adopted to dynamically apply mutual perspective constraints to calibrate the baseline model. The filtering mechanism is selected based on the architecture type of the baseline model, including the following two filtering mechanisms: For closed-set models constrained by a fixed vocabulary, a validity filter is applied, adding an inverse sample only if the generated inverse predicate exists in the predefined vocabulary; otherwise, the original sample data is retained to filter out noise outside the distribution. For open-set models without vocabulary constraints, inverse mapping is applied without restriction. Then, for semantic relations or tail-class predicates, a logical sample expansion strategy is applied to supplement the training set with the generated valid inverse triples; for geometric relations and head-class predicates, a random logical permutation strategy is applied to probability-wise permutation of the original triples. Replace with a valid inverse triplet; Step 3: Introduce hypernym guidance prompts to the visual language model to construct a dual-view context. Utilize a large language model to abstract and map fine-grained entity categories to a small-scale, coarse-grained set of semantic hypernyms. In the middle; combining learnable context parameters, the positions of the hypernyms of the subject and object are explicitly swapped, and original perspective prompt templates are generated respectively. With reverse perspective prompt template ; Step 4: Use a pre-trained visual language large model for joint training and recognition prediction. Calculate the similarity between the constructed dual-view cue vector and the image visual features in the latent space, perform cross-entropy joint optimization, and finally output a multi-label scene graph generation result with cross-view consistency.

2. The scene graph generation method based on mutual perspective inverse relationship calibration and hypernym guidance prompts according to claim 1, characterized in that, In step 1, the scene diagram is represented as follows: Node set Representative entities, edge sets Representational relationship; The principle for defining mutual perspective relationships is as follows: any objective interaction must exist independently of the observation anchor point, if physical facts... If true, then the mapping is satisfied: This establishes the logic of bijective transformation. The predicate space is divided into symmetric and asymmetric relations; the expression for a symmetric relation is: The expression for the asymmetric relation is: ; Next, an adaptive inverse relation enhancement calibration strategy is adopted to transform the mutual perspective inverse relation principle into a training signal; considering the long-tail distribution and information density of predicates, it is specifically divided into two calibration paths: (1) Path A, namely logical sample expansion, is to expand the original dataset based on semantic categories or tail category relationships that are high in information density but extremely scarce, by adding data on inverse relationships, that is, introducing inverse triples as incremental supervision signals. For such effective triples Generate inverse triplet And expanded to the training set middle, The expression is: ； (2) Path B, i.e., random logical permutation, is used to perform random logical permutation with corresponding probabilities for geometric relationships and head categories that are highly redundant and prone to overfitting positional deviations. The inverse relationship data is used as a viewpoint perturbation and the perturbation triples are sampled through Bernoulli distribution. Completed, the expression is as follows: ； Among them, control variables obey That is, the Bernoulli distribution.

3. The scene graph generation method based on mutual perspective inverse relationship calibration and hypernym guidance prompts according to claim 1, characterized in that, Step 3, the specific method for constructing hypernym guidance prompts, is as follows: Using a large language model to refine fine-grained entity sets Mapping to a small set of hypernyms ,satisfy The projection function is defined as follows: ; Based on the above mapping relationship, fusion A learnable continuous context vector For each interaction, construct a pairing cue between the original viewpoint and the reverse viewpoint: The expression for the original perspective hint template is: ; The expression for the reverse perspective hint template is: ; in, This represents the fixed text word embedding vectors of the pre-trained visual language model, and the hypernym guidance prompting mechanism structurally aligns the text prompts with the visual inverse transformation; Next, adaptive filtering based on word awareness is performed: When executing the above path, the rules for adopting inverse relations need to be determined based on whether the model is a closed or open set. For open set models, mappings are applied without restriction to introduce new inverse predicates; for closed set models, a validity filter is applied, meaning that the generated inverse triples are only applied to their inverse predicates. It is only injected or replaced when needed.

4. The scene graph generation method based on mutual perspective inverse relationship calibration and hypernym guidance prompts according to claim 1, characterized in that, The detailed method for step 4, cross-modal recognition prediction and joint model optimization, is as follows: Joint optimization of cross-view features is performed based on the cross-entropy function, on a calibration dataset enhanced by an adaptive inverse relation enhancement strategy. The goal is to minimize the expected negative log-likelihood loss. : ； Among them, symbols Characterizes the mathematical expectation operation on the calibration dataset augmented by the adaptive inverse relation enhancement strategy; variables The label represents the real physical association fact corresponding to the current input subject-object triplet sample, i.e., the labeled real relation predicate. For the large open-set visual language model adopted, combined with visual features Embedded with text prompts generated through hypernym guidance The probability of relationship classification is calculated using the temperature-scale cosine similarity between the two values. The expression is: ； in, The function represents the learnable temperature scaling parameter. It refers to the measurement of visual features of an image. The cosine similarity function is used to calculate the degree of matching between text spatial features; set Represents the predefined vocabulary space of relational predicates; variables Specifically refers to the target relation category that the current model is positively predicting; subscript To traverse the set of predicate vocabularies Index parameters for all potential relation categories; variables Used to indicate in a predefined vocabulary within, no. The corresponding text hint embedding vector generated after each category is constructed using a hint template.

Images