A novel multi-modal semantic-spatial representation method for embodied perception and spatial reasoning

Abstract

This invention discloses a novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning, belonging to the field of embodied perception. Addressing the two major pain points of existing methods—insufficient spatial coherence in semantic reasoning and poor relation consistency and interpretability—this invention designs an optimization mechanism: on the one hand, it designs a transitive relation learning mechanism to correct relation conflicts and missing relations in reasoning under weak supervision, improving the completeness and logical consistency of spatial representation; on the other hand, relying solely on RGB images and depth maps, it integrates monocular depth estimation as a spatial modality into embodied perception, injecting implicit three-dimensional spatial attributes into the semantic structure, constructing a unified multimodal semantic-spatial representation, and enhancing the spatial coherence of semantic reasoning. This invention improves the performance of structured scene modeling and spatial grounding description generation through advanced spatial representation, and can be widely used in embodied spatial perception scenarios.

Description

Technical Field

[0001] This invention belongs to the field of embodied perception and spatial reasoning technology, specifically relating to a novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning. Background Technology

[0002] Embodied agents perform perception, decision-making, and action tasks in dynamic and complex environments. Their core capabilities rely on effective environmental understanding, which is typically built upon multimodal embodied perception and long-term spatial memory. Recent advancements in multimodal representation learning and visual-language models have significantly improved cross-modal perception and reasoning capabilities, enabling embodied systems to more efficiently integrate visual, linguistic, and spatial information. From the perspective of pattern recognition and computer vision, this challenge can be modeled as learning structured and reasonable scene representations from multimodal perceptual inputs. In these representations, semantic information and spatial structure are jointly modeled within a unified framework. Compared to visual representations that focus only on local appearance features, scene representations that explicitly encode semantic relationships, spatial structure, and multimodal spatial cues provide more stable and interpretable support for reasoning and decision-making. Therefore, learning efficient scene representations has become a core issue in the fields of visual scene understanding, multimodal spatial cognition, and embodied intelligence.

[0003] Many studies on visual scene understanding and spatial representation have attempted to incorporate 3D geometric information or structured relationship modeling methods. Methods based on LiDAR point clouds or depth sensors have achieved high spatial accuracy through explicit 3D modeling. However, these methods typically rely on dedicated hardware and large-scale labeled 3D data, limiting their scalability in general environments. In contrast, vision-based methods often employ deep neural networks to learn implicit feature representations.

[0004] In recent years, significant progress has been made in the fields of neural scene representation and neural rendering. Gaussian representation-based 3D scene modeling methods have further enhanced the geometric perception and 3D reconstruction capabilities of visual systems. While these methods have achieved remarkable progress in perception tasks, spatial relationships are often implicitly encoded in a distributed manner, resulting in a lack of explicit structural constraints and making it difficult to guarantee relationship consistency and interpretability. Some methods further introduce symbolic or graph-based representations to enhance expressive power, but these typically rely on strong supervision or manually designed rules, making it difficult to generalize to complex real-world scenes. In summary, these methods demonstrate the potential for visual scene modeling while also exposing the fundamental limitations of constructing unified, structured spatial representations under weak supervision. Summary of the Invention

[0005] The purpose of this invention is to provide a novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning, in order to solve the technical problems in the prior art where scene representation relies on dedicated sensors and large-scale 3D annotation lacks explicit structural constraints on spatial relationships.

[0006] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0007] A novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning, the method comprising the following steps:

[0008] Step S1: The initial scene graph generation module sequentially performs visual feature encoding, entity extraction, and triple parsing on the input RGB image to generate an initial scene triple containing subject-predicate-object, and completes the global triple information summary.

[0009] Step S2: The transitive prior learning module constructs a learnable directed relation graph using a predefined set of predicates. Based on the co-occurrence frequency of relations in image triples and subject-object overlap information, iteratively updates the edge weights of the relation graph to obtain a converged transitive relation graph. Simultaneously, based on the statistical regularity of global triples, high-confidence transitive patterns are selected to construct a prior predicate relation table. The relation graph and the prior relation table are fused to obtain the final transitive weights. Logical conflicts are corrected and missing relations are filled in for the initial triples to obtain an optimized set of triples.

[0010] Step S3: The implicit 3D spatial attribute injection module converts the RGB image into a depth map through monocular depth estimation, aligns the depth features with the entity region, provides pixel-level geometric cues, injects geometric constraints into the semantic structure, and forms a unified semantic-spatial multimodal representation.

[0011] Step S4: Optimize the triplet-driven text generation module by randomly shuffling the triplet sequence to perform data augmentation. Encode the augmented triplets into the model input sequence, fine-tune it using a pre-trained T5 model, and generate text descriptions through autoregression. In the inference stage, use bundle search decoding with diversity constraints to output semantically faithful and spatially consistent natural language scene descriptions.

[0012] ...

[0013] Compared with the prior art, the present invention has the following beneficial technical effects:

[0014] To address the technical problems of existing technologies where scene representation relies on dedicated sensors and large-scale 3D annotation lacks explicit structural constraints on spatial relationships, this invention proposes a novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning. This method first utilizes an initial scene graph generation module to achieve end-to-end entity detection and initial triple generation. This module first extracts multi-level visual features from the input RGB image using a convolutional neural network to generate an initial feature map. Then, the initial feature map is flattened into a sequence and superimposed with pixel space coordinate encoding. A Transformer encoder is introduced to enhance pixel-level global dependencies, modeling long-distance pixel dependencies across the entire image to obtain a unified global visual representation. Subsequently, learnable entity queries are set based on the entity density of the dataset. Entity information is extracted using a Transformer entity decoder with self-attention and cross-attention mechanisms to generate a refined entity representation. This is then combined with a multilayer perceptron prediction head to complete entity category classification and bounding box regression. Finally, through learnable subject queries, object queries, relation encoding, subject encoding, and object encoding, combined with a multi-stage attention mechanism, a triplet structure is constructed to generate initial triplets containing subject-predicate-object. The triplets from a single image and all images are then summarized to complete the global triplet information aggregation.

[0015] Then, a transitive prior learning module is used to construct transitive prior knowledge and optimize the initial triples. First, a learnable directed relation graph is constructed based on a predefined set of predicates. The edge weight matrix of the relation graph is initialized with a uniform distribution to determine the initial propagation probability between each relation. Then, for the triple set derived from the image, the scoring function of the predicate pair is calculated per image unit (composed of normalized relation co-occurrence frequency and subject-object overlap weights). The edge weights of the graph are iteratively updated until convergence, resulting in a learnable transitive relation graph. Next, based on the predicate co-occurrence frequency and subject-object overlap of the global triple set, high-confidence transitive relations that meet the preset threshold are selected to construct a prior predicate relation table. Finally, the learnable transitive relation graph and the prior predicate relation table are fused and encoded to obtain the final transitive weights. Based on these weights, logical conflicts in the initial triple set are corrected and missing relations are filled in, finally outputting the optimized triple set.

[0016] Next, spatial attributes are injected and multimodal representations are constructed using an implicit 3D spatial attribute injection module. First, the input RGB image is converted into a dense depth map through monocular depth estimation. The depth features are then aligned pixel-level with the entity regions in the initial triples, and object-level geometric attributes such as relative depth order are extracted to provide pixel-level geometric cues for entities and relationships. Then, geometric constraints are injected into the semantic structure to form a unified semantic-spatial multimodal representation, providing spatial constraints for transitive prior learning. Finally, this multimodal representation is input into the subsequent triple-driven text generation module for optimization.

[0017] Finally, the optimized triplet-driven text generation module generates a natural language scene description. First, the optimized triplet sequence is randomly shuffled to perform data augmentation, and the augmented triples are encoded with the target text into a model-compatible word sequence. Then, a pre-trained T5 model is fine-tuned to encode the global "subject-relation-object" semantics, generating a text description based on the autoregressive cross-entropy objective. Finally, during the inference phase, a bundle search with diversity control is used to decode the optimized triples, outputting a semantically faithful and spatially consistent natural language scene description.

[0018] Existing embodied spatial perception methods rely on dedicated sensors and large-scale 3D annotations for scene representation, lack explicit structural constraints on spatial relationships, suffer from insufficient spatial coherence in semantic reasoning, and exhibit poor relationship consistency and interpretability. This invention constructs a unified semantic-spatial structure representation from a single RGB image, models structured relationships, characterizes objects and their interactions, and forms a scene description with transferable reasoning capabilities. Furthermore, an implicit spatial perception mechanism is introduced to supplement the semantic structure with necessary implicit 3D spatial attributes. By combining semantic relationship reasoning with spatial attribute modeling, this method can achieve effective representation and long-term memory of complex scenes without dedicated sensors or 3D annotations. Addressing the two major pain points of existing methods—insufficient spatial coherence in semantic reasoning and poor relationship consistency and interpretability—this invention designs optimization mechanisms: on the one hand, a transferable relationship learning mechanism is designed to correct relationship conflicts and missing relationships under weak supervision, improving the completeness and logical consistency of the spatial representation; on the other hand, relying only on RGB images and depth maps, monocular depth estimation is integrated as a spatial modality into embodied perception, injecting implicit 3D spatial attributes into the semantic structure, constructing a unified multimodal semantic-spatial representation, and enhancing the spatial coherence of semantic reasoning. This invention improves the performance of structured scene modeling and spatial grounding description generation with advanced spatial representation, and can be widely used in embodied spatial awareness scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning of the present invention.

[0021] Figure 2 This is a schematic diagram of the detection results of the module that generates the initial indoor scene image.

[0022] Figure 3 This is an indoor monocular depth estimation map.

[0023] Figure 4 A schematic diagram of the detection results of the module that generates the initial outdoor scene image.

[0024] Figure 5 This is an outdoor monocular depth estimation map.

[0025] Figure 6 This is a descriptive image with perception of people and vehicles.

[0026] Figure 7 This is a descriptive image with animal and environmental awareness. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] This invention proposes a novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning, such as... Figure 1 As shown, the method includes the following steps:

[0029] Step S1: The initial scene graph generation module sequentially performs visual feature encoding, entity extraction, and triple parsing on the input RGB image to generate an initial triple containing subject-predicate-object, and completes the global triple information summary.

[0030] Step S11: From the input image Extracting local spatial details and global contextual information. A convolutional neural network is used to extract multi-level visual features and generate an initial feature map. ,in, Indicates the initial feature map height. Indicates the initial feature map width. Indicates the number of channels in the initial feature map;

[0031] Initial feature map Flatten into a sequence and encoded with pixel space coordinates Superimposed, input to a Transformer solid encoder containing self-attention and feedforward networks. Modeling long-distance pixel dependencies across the entire image establishes the relationships between different regions and distant pixels in the image, generating a unified global visual representation:

[0032]

[0033] Step S12: Set according to the entity density of the dataset Learnable entity query , Indicates the total number of entity queries. Indicates the feature dimension;

[0034] Learnable entity queries The Transformer entity decoder is input with a unified global visual representation F, and self-attention is used to model the contextual dependencies between entities. Cross-attention is used to extract visual features related to entities, thereby generating a refined entity representation. Refined entity representation Input a parallel multilayer perceptron prediction head to perform entity category classification, bounding box regression, and generate low-dimensional entity features for subsequent relationship prediction.

[0035] Step S13: Refine the entity representation output by entity decoding Based on this, learnable subject queries are introduced. Object query Relational encoding Main body code Object coding Modeling subject queries and object queries using a multi-stage attention mechanism. The interaction between them ensures that the subject and object are real entities, ultimately generating a subject with triplet awareness. and object The calculation formula is as follows:

[0036]

[0037] subject and object The predicate relationship between the predicted subject and object Finally, they combine to form a complete triplet. In this context, 's' represents the subject and 'o' represents the object. The set of all generated triples within an image is the sum of all the triples generated for that image. All images After summarizing, it becomes the global set of triples. .

[0038] Step S2: The transitive prior learning module constructs a learnable directed relation graph using a predefined set of predicates. Based on the co-occurrence frequency of relations in image triples and subject-object overlap information, iteratively updates the edge weights of the relation graph to obtain a converged transitive relation graph. Simultaneously, based on the statistical regularity of global triples, high-confidence transitive patterns are selected to construct a prior predicate relation table. The relation graph and the prior relation table are fused to obtain the final transitive weights. Logical conflicts are corrected and missing relations are filled in for the initial triples to obtain an optimized set of triples.

[0039] Step S21: The transitive prior learning module starts from a randomly initialized relation graph, derives relation co-occurrence statistics and subject-object overlap statistics for triples based on the image, and iteratively updates edge weights. The predefined relation set is denoted as... , The number of representative relations, Represent the Mth relation; the initial directed graph is defined as:

[0040]

[0041] in, This is the edge weight matrix, where each element represents a relation. arrive The initial propagation probability. Initial edge weights. Through uniform distribution initialization.

[0042] Step S22: Based on the global triplet set of the dataset Calculate predicate pairs The scoring function is calculated as follows:

[0043]

[0044] in, This represents a relational transitive scoring function. This represents the normalized co-occurrence frequency of the relation. This represents the normalized subject-object overlap. It is the balance coefficient.

[0045] Derivation of triplet sets from images The iterative update formula for the relation edge weights is as follows:

[0046]

[0047] in, Indicates the first Relationship at the next iteration and Edge weights between them The learning rate is used to control the update magnitude.

[0048] Repeat the iterations until the edge weights converge to obtain a learnable transitive relation graph. , Represents a set of relations; This represents the convergent edge weight matrix.

[0049] Step S23: Construct the prior predicate relation table as follows:

[0050]

[0051] in, Representing relation pairs The normalized prior confidence level. and For predefined thresholds.

[0052] The learnable transitivity relation graph and the prior predicate relation table are fused to obtain the final transitivity weights. :

[0053]

[0054] in, This represents the edge weights after convergence. Used to control the fusion weight between learnable graphs and prior confidence.

[0055] Finally, logical conflicts and missing relationships are corrected for the initial triples, based on... The transitivity strength of the relationships is determined, and for relationships with conflicting relationships, the relationships with stronger transitivity are retained first, resulting in an optimized set of triples.

[0056] Step S3: The implicit 3D spatial attribute injection module converts the RGB image into a depth map through monocular depth estimation, aligns the depth features with the entity region, provides pixel-level geometric cues, injects geometric constraints into the semantic structure, and forms a unified semantic-spatial multimodal representation.

[0057] Step S31: Convert the input RGB image into a dense depth map through monocular depth estimation, align the depth features with the entity regions in the initial triplet generated in step S1 at the pixel level, extract object-level geometric attributes such as relative depth order, and provide pixel-level geometric clues for entities and relationships.

[0058] Step S32: Inject geometric attributes into triples, further optimize the optimized triple set to form a unified semantic-spatial multimodal representation, providing spatial constraints for the transitive prior learning in step S2.

[0059] Step S33: Finally input the optimized triplet-driven text generation module from step S4.

[0060] Step S4: Optimize the triple-driven text generation module by randomly shuffling the triple order of the semantic-spatial multimodal representation to perform data augmentation. Encode the augmented triples into the model input sequence, fine-tune it using a pre-trained T5 model, and generate text descriptions through autoregression. In the inference stage, use bundle search decoding with diversity constraints to output semantically faithful and spatially consistent natural language scene descriptions.

[0061] Step S41: Optimize the triple-driven text generation module by randomly shuffling the semantic-spatial multimodal representations into triple sequences to perform data augmentation, encoding them into a T5 model-compatible word sequence; fine-tune the pre-trained T5 model, which is a pre-trained text-to-text Transformer model proposed by Google, that encodes global "subject-verb-object" semantics and generates text descriptions based on an autoregressive cross-entropy objective.

[0062] Step S42: The autoregressive cross-entropy loss function is defined as follows:

[0063]

[0064] in, The number of valid lexical units, The length of the output sequence generated for the text. The word-level prediction distribution of the decoder. For the first in the target sequence One effective word element, Indicates the preceding A prefix of a real word element, For indicator functions, For the scene Figure 3 The input sequence of the model is composed of tuples.

[0065] Step S43: During the inference phase, the optimized triples are decoded using a bundle search with diversity control to output a fluent and semantically consistent text description.

[0066] The effectiveness of this invention will be demonstrated through the following experiments:

[0067] 1. Experimental data:

[0068] Visual Genome is the most representative large-scale benchmark dataset in the field of scene graph generation. It contains dense and fine-grained object annotations, attribute annotations, and visual relation annotations, which can fully reflect the model's ability to detect objects, predict relations, and make structured inferences in complex scenes. It is suitable for evaluating the improvement effect of this method in object-level relation modeling, transitive reasoning consistency, and semantic structure integrity.

[0069] MSCOCO Caption is a classic standard dataset in the field of image text generation. Each image corresponds to multiple high-quality human-written descriptive texts. It focuses on evaluating the interpretability of the model under high-level semantic understanding, semantic coherence, and spatial constraints. It is used to verify whether the unified semantic-spatial multimodal representation constructed by this method can effectively support the generation of high-quality, spatially consistent natural language scene descriptions.

[0070] 2. Experimental Results and Analysis

[0071] As shown in Table 1, two-stage models perform relatively well in terms of recall@50 (R@50), but typically have a lower mean recall@50 (mR@50) score and often suffer from large model size and slow inference speed. These limitations mainly stem from the two-stage pipeline: entity detection is performed first, followed by relation inference. This process is prone to error propagation and weakens the ability to model long-tail relationships. In contrast, single-stage models (such as SGTR) perform better in mean recall@50 (mR@50), but have a relatively lower recall@50 (R@50), reflecting the trade-off between recall and prediction balance.

[0072] Among all methods, Initial Scene Graph Generation + Transitive Prior Learning (ISG+TPL) achieved the best performance in terms of recall@50 (R@50), while its recall@50 (R@50) was only 4.2% lower than the best-performing BGT-Net. This indicates that the method achieves an effective balance between high recall and consistent relation prediction. Furthermore, comparing the results of Initial Scene Graph Generation (ISG) and Initial Scene Graph Generation + Transitive Prior Learning (ISG+TPL), it is evident that the introduction of the Transitive Prior Learning (TPL) module improves the average model performance by approximately 1.0 percentage point, validating the effectiveness of learnable transitive priors in scene graph generation tasks.

[0073] Table 1. Experimental results comparing existing methods with ISG+TPL

[0074] Model ↑ @20↑ @50↑ @20↑ @50↑ Avg.↑ Motifs-TDE, a time-aware scene graph model based on neural patterns. - 12.4 16.9 5.8 8.2 10.8 VTransE-TDE Visual Translation Embedded Time-Aware Model - 13.5 18.7 6.3 8.6 11.8 VCTree-TDE Dynamic Tree Structure Time-Aware Model 28.1 14.0 19.4 6.9 9.3 12.4 Motifs Neural Model 20.0 21.4 27.2 4.2 5.7 14.7 FCSGG Fully Convolutional Scene Graph Generation Model 28.5 16.1 21.3 2.7 3.6 11.0 KERN Knowledge Embedded Routing Network 20.0 22.1 27.1 - 6.4 14.0 VCTree Dynamic Visual Context Tree Model - 22.0 27.9 5.2 6.9 13.0 BGT-Net bidirectional GRU-Transformer network 28.1 25.5 32.8 5.7 7.8 17.0 SGTR is an end-to-end scene graph generation model based on Transformer. - 21.0 20.6 - 10.8 15.8 Relative Transformer Model 26.4 21.6 27.0 7.7 10.8 18.7 SGTR+ Enhanced Transformer End-to-End Scene Graph Model - 21.8 27.9 7.7 11.0 17.1 RRM Relationship Refinement Model - 22.0 28.1 7.9 11.0 17.3 ISG Initial Scene Graph Construction 26.4 21.2 27.5 6.8 10.8 16.6 ISG+TPL Initial Scene Graph Construction + Optimized Transmission 26.4 22.3 28.6 8.1 11.5 19.3

[0075] As shown in Table 2, to enhance the practical relevance of the experimental conclusions, we supplemented the results with resolution comparison experiments using various scaling factors. Test resolutions included 0.25×(160×120), 0.5×(320×240), and the original baseline resolution of 1×(640×480). All resolution adjustments were implemented using bilinear interpolation to avoid geometric distortion and maintain feature continuity. The results in Table 2 show that the performance of the proposed method is significantly affected by image resolution: lower image resolution weakens the feature extraction capability of the object detection module in the framework. Nevertheless, evaluation metrics show that using 0.5× downsampled images still maintains competitive performance.

[0076] Table 2. Experimental results comparing existing methods with ISG+TPL

[0077] Model ↑ @20↑ @50↑ @20↑ @50↑ Avg.↑ ISG+TPL(0.25) 18.7 15.4 19.3 5.2 7.6 13.2 ISG+TPL(0.5) 23.1 19.2 24.7 6.9 9.8 16.7 ISG+TPL(1) 26.4 22.3 28.6 8.1 11.5 19.3

[0078] like Figure 2-5 As shown, we visualize the two core outputs of our method: structured entity-relation triples and their corresponding scene depth estimation results. The image uses a three-row layout: the first row shows human-interaction scenes, the second row shows human-pet interaction scenes, and the third row shows dynamic motion scenes. Each image contains four images of labeled triples (with compact bounding boxes labeled for the relevant entities) and one depth map capturing the spatial layout of the scene.

[0079] This figure plays two key roles in validating this study:

[0080] First, these images validate the effectiveness of our proposed structured semantic modeling method. For example, in the first row (human interaction), by annotating the human's hand and the bottle with compact bounding boxes, the triple "human holding a bottle" was located, achieving precise localization of the relational entity. Similarly, in the second row (human-pet interaction), the relationship "human and dog" was located in the human's legs and the dog's body area, capturing the spatial proximity in a companion relationship. In the third row (sports scene), triples such as "human wearing a shirt" annotated the athlete's clothing with compact bounding boxes—even in dynamic scenes, the localization results maintain context awareness. These examples validate that our method can reliably associate entities (such as humans and dogs) with relationships (such as "holding" and "accompanying") through compact and precise bounding boxes, highlighting its robustness and effectiveness in structured scene understanding tasks.

[0081] Second, the image validates the quality of the depth estimation method. Each row contains a scene depth map: warm colors (such as yellow and orange) represent closer objects, and cool colors (such as blue and purple) represent farther objects. For example, in the first row, the warm color of the bottle matches its foreground position relative to the person's torso, presenting fine spatial details; in the second row, the dog's body and the person's legs are on the same depth plane, reflecting their physical proximity; in the third row, the athletes labeled "person wearing a shirt" and "person wearing shorts" are closer to the camera than the background (such as the "house window" in the fourth image), fitting the foreground focusing characteristics of moving scenes. These depth maps validate that the method can generate accurate, scene-consistent depth estimation results, which is the core foundation for anchoring semantic understanding in a three-dimensional spatial context.

[0082] like Figure 6 , 7 As shown, we qualitatively compared the image description results of the reference annotation, the ViECap model, and our proposed method for different scenarios. The ViECap model only focuses on object existence and coarse-grained semantic information, lacking explicit modeling of scene spatial structure and relationship consistency.

[0083] In contrast, this method, based on reasoning-oriented semantic-spatial joint representation, can generate more detailed and spatially clear descriptive text. By explicitly constructing object-relation triples and introducing a transitive relation learning mechanism, the model can correct relation conflicts and infer missing spatial relations. At the same time, monocular depth estimation provides the model with implicit spatial cues, enabling the model to describe the relative positions and distances between objects.

[0084] This method can accurately capture fine-grained spatial relationships, such as "in front of", "next to", "about 0.5 meters apart", etc., which are often missing or incorrectly described in the descriptions generated by ViECap.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A novel multi-modal semantic-spatial representation method for embodied perception and spatial reasoning, characterized in that, Includes the following steps: Step S1: The initial scene graph generation module sequentially performs visual feature encoding, entity extraction, and triple parsing on the input image to generate an initial triple containing subject-predicate-object, and completes the global triple information summary. Step S2: The transitive prior learning module constructs a learnable directed relation graph using a predefined set of predicates. It iteratively updates the edge weights of the relation graph based on the co-occurrence frequency of the relation in the image triples and the subject-object overlap information to obtain a converged transitive relation graph. At the same time, it selects high-confidence transitive patterns based on the statistical regularity of global triples and constructs a prior predicate relation table. The relation graph is fused with the prior relation table to obtain the final transitive weights. Logical conflicts are corrected and missing relations are filled in the initial triples to obtain the optimized triple set. Step S3: The implicit 3D spatial attribute injection module converts the input image into a depth map through monocular depth estimation, aligns the depth features with the entity region, provides pixel-level geometric cues, and injects an optimized set of triples to form a unified semantic-spatial multimodal representation. Step S4: Optimize the triple-driven text generation module by randomly shuffling the triple order of the semantic-spatial multimodal representation to perform data augmentation. Encode the augmented triples into the model input sequence, fine-tune it using a pre-trained T5 model, and generate text descriptions through autoregression. In the inference stage, use bundle search decoding with diversity constraints to output natural language scene descriptions.

2. A novel multi-modal semantic-spatial representation method for body-awareness and spatial reasoning as claimed in claim 1, wherein, The specific steps of S1 are as follows: Step S11: extracting local spatial details and global context information from the input image extracting local spatial details and global context information from the input image, generating an initial feature map; flattening the initial feature map and superimposing it with pixel spatial coordinate encoding, inputting it into a Transformer entity encoder to generate a unified global visual representation; Step S12: setting according to data set entity density number of learnable entity queries , denotes the total number of entity queries, denotes the feature dimension; Learning to query entities and unified global visual representation F input transformer entity decoder, with self-attention modeling inter-entity context dependency, through cross-attention to extract entity-related visual features, and then generate refined entity representation ; Refining entity representations Input parallel multi-layer perceptron prediction heads, respectively implementing entity class classification, bounding box regression; Step S13: Introduce learnable subject queries Object query Relational coding Main body code Object coding Modeling subject queries and object queries with refined entity representation through a multi-stage attention mechanism. The interaction between them generates a subject with triplet awareness. and object The final calculation formula is as follows: subject and object The predicate relationship between the predicted subject and object Finally, these are combined to form a complete triplet. The sum of all the triplets generated in an image is the set of triplets for that image. All images After summarizing, it becomes the global set of triples. .

3. A novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning according to claim 2, characterized in that, Step S2 is as follows: Step S21: Predefine the set of relations , The number of representative relations, Let M be the Mth relation; define the initial directed graph as follows: ,in, Let be the edge weight matrix, where each element represents the edge weight from the predicate. arrive Initial propagation probability, initial edge weight Through uniform distribution initialization. Step S22: Based on the global triple set of the dataset Co-occurrence statistics of relations and overlap statistics of subject and object, calculating predicate pairs The scoring function; then based on the set of triples of the image. The scoring function iteratively updates the edge weights of the relationships; this process is repeated multiple times until the edge weights converge, resulting in a learnable transitive relationship graph. , This represents the convergent edge weight matrix; Step S23: Construct the prior predicate relation table as follows: in, Indicates predicate pair The normalized prior confidence level. and For predefined thresholds. The learnable transitivity relation graph and the prior predicate relation table are fused to obtain the final transitivity weights. : in, This represents the edge weights after convergence. Used to control the fusion weight between learnable graphs and prior confidence. Finally, according to The transitivity strength of the relationships is determined, and for relationships with conflicting relationships, the relationships with stronger transitivity are retained first, resulting in an optimized set of triples.

4. A novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning according to claim 3, characterized in that, Step S4 is as follows: The optimized triplet sequence was randomly shuffled to perform data augmentation and encoded into a T5 model-compatible word sequence. A pre-trained T5 model was used for fine-tuning to encode the global subject-verb-object semantics and generate text descriptions based on the autoregressive cross-entropy objective. During the inference phase, a bundle search with diversity control is used to decode the optimized triples, outputting a fluent and semantically consistent text description.

5. A novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning according to claim 4, characterized in that, The scoring function is calculated as follows: in, This represents a relational transitive scoring function. This represents the normalized co-occurrence frequency of the relation. This represents the normalized subject-object overlap. It is the balance coefficient. Image-derived triple set The iterative update formula for the relation edge weights is as follows: in, Indicates the first Relationship at the next iteration and Edge weights between them This is the learning rate.

6. A novel multimodal semantic-spatial representation method for embodied perception and spatial reasoning according to claim 5, characterized in that, The loss function for the autoregressive cross-entropy objective is as follows: in, The number of valid lexical units, The length of the output sequence generated for the text. The word-level prediction distribution of the decoder. For the first in the target sequence One effective word element, Indicates the preceding A prefix of a real word element, For indicator functions, The input sequence is a model consisting of scene graph triples.

Images