A method, device and storage medium for enhancing spatial perception capability of a robot

Abstract

The application discloses a robot space perception ability enhancement method and device and a storage medium, relates to the field of embodied intelligence, and aims to improve the space perception ability and precision of a robot in a 2D image. A monocular camera is used to continuously collect a 2D image sequence; a key frame sequence is obtained from the 2D image sequence; the key frame sequence is subjected to feature coding in the plane dimension and the space dimension; the coded fusion features are subjected to multi-scale feature fusion; multi-task reasoning output is performed based on the fusion features and the multi-scale fusion features; in the multi-scale feature fusion stage, the fusion features are injected into a multi-modal large language model (MLLM) in layers, and the MLLM is configured to perform a scale perception MoE mechanism on each hidden layer. The application reduces the dependence of robot space perception on expensive 3D sensors, realizes efficient and accurate online space perception, and significantly improves the accuracy and robustness of space reasoning.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method, apparatus and storage medium for enhancing the spatial perception capabilities of robots. Background Technology

[0002] With the deepening application of robotics technology in fields such as intelligent manufacturing, home services, and autonomous driving, the ability of robots to perceive, understand, and interact with complex three-dimensional environments (i.e., spatial perception) has become crucial in determining their intelligence level. Traditional robot spatial perception solutions mainly rely on expensive 3D sensors (such as LiDAR and depth cameras) or complex multi-sensor fusion systems. These solutions are not only costly but also suffer from performance limitations in scenarios involving strong light, transparent objects, and long distances. They also require significant computing power, making them unsuitable for deployment on low-cost, lightweight robot platforms. Furthermore, they lack semantic understanding capabilities and cannot perform complex spatial reasoning using natural language. In contrast, 2D vision solutions based on monocular or binocular RGB cameras offer advantages such as low cost and ease of deployment. Therefore, how to endow robots with powerful 3D spatial perception capabilities from ordinary 2D image input is a current research hotspot in the fields of computer vision and robotics.

[0003] Currently, the known technical approaches for enhancing a robot's spatial perception capabilities from 2D images mainly fall into the following categories: Route 1: SLAM (Simultaneous Localization and Mapping) system based on monocular depth estimation; Route 2: Perception scheme based on end-to-end 3D reconstruction model; Route 3: Spatial reasoning based on Multimodal Large Language Model (MLLM).

[0004] However, the above-mentioned technical approaches also have some shortcomings: Known spatially-aware optimization methods for MLLM rely solely on fine-tuning of the dataset without introducing explicit spatial geometric constraints. This results in significant biases in predicting the spatial distance and relative orientation of objects in 2D images, and insufficient depth prediction accuracy in the near-field operation area. Known streaming 3D reconstruction techniques only complete geometric reconstruction and do not integrate with the large semantic understanding model end-to-end. They cannot convert geometric information into semantic instructions that robots can execute. Geometric perception and high-level semantic decision-making are completely separated, resulting in poor semantic reasoning ability. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for enhancing the spatial perception capability of a robot, thereby improving the robot's spatial perception capability and accuracy in 2D images, addressing all or part of the aforementioned problems.

[0006] The technical solution adopted in this invention is as follows: In a first aspect, this application provides a method for enhancing the spatial perception capability of a robot, comprising: Continuous acquisition of 2D image sequences using a monocular camera; Obtain a keyframe sequence from the 2D image sequence; encode the keyframe sequence using planar and spatial features; perform multi-scale feature fusion on the encoded fusion features; and perform multi-task inference output based on the fusion features and the multi-scale fusion features. In the multi-scale feature fusion stage, the fused features are injected into the multimodal large language model MLLM in layers, and the MLLM is configured to perform a scale-aware MoE mechanism on each hidden layer.

[0007] Optionally, the fused features can be injected hierarchically into the MLLM, including: Determine the injection point from the MLLM backbone network; Configure an adapter for each injection point and align the temporal temperature of the fused feature with the hidden state of the corresponding layer of the injection point; The hidden state is enhanced and updated by using the adapters at each injection point to fuse features.

[0008] Optionally, the keyframe sequence is subjected to feature encoding in both planar and spatial dimensions, including: Semantic feature encoding is performed on the keyframes; Spatial feature encoding is performed on the keyframes; Perform feature alignment and gating fusion of the semantic features and the spatial features.

[0009] Optionally, obtaining a keyframe sequence from the 2D image sequence includes: The camera frustum of each frame in the 2D image sequence is converted. Keyframes are selected from the 2D image sequence based on the principle of maximizing the union of the frustum sets of the selected frame images by selecting a predetermined number of frame images. Spatial metric data augmentation is then performed on the keyframes.

[0010] Optionally, spatial metric data augmentation is performed on the keyframes, including: Select the target object from the keyframe; Calculate the spatial metric parameters of the target object; Based on the target object and the spatial metric parameters, generate spatial metric question-and-answer pairs for keyframes.

[0011] Optionally, multi-task inference output based on multi-scale fusion features and multi-scale feature fusion includes: Using the first prediction head, and based on the multi-scale fusion features, an inference chain is generated using a Chain-of-Point mechanism; Using the second prediction head, based on the fused features, and employing the first multilayer perceptron (MLP), the relative camera pose change of the current keyframe relative to the first keyframe is generated. Using the third prediction head and based on the fused features, the second MLP is employed to generate spatial metric parameters for the current keyframe.

[0012] Optionally, the composite loss of the multi-task inference output of MLLM includes the language model cross-entropy loss of the first prediction head, the relative pose regression loss of the second prediction head, the spatial metric regression loss of the third prediction head, and the spatial consistency loss; the relative pose regression loss, spatial metric regression loss, and spatial consistency loss are weighted and summed by hyperparameters.

[0013] Optionally, the method for calculating the spatial consistency loss includes: Multiple sets of feature point pairs are selected from the fusion features; Predict the 3D coordinates of each feature point; Obtain the true 3D coordinates of each feature point; Calculate the predicted distance of each feature point to the predicted 3D coordinates, and the true distance to the actual 3D coordinates. Calculate the mean difference between the predicted distance and the true distance for each feature point pair.

[0014] In a second aspect, this application also provides a robot spatial perception enhancement device, which includes a processor and a storage medium storing a computer program, and the processor runs the computer program to execute the above-described robot spatial perception enhancement method.

[0015] In a third aspect, this application also provides a computer-readable storage medium comprising a computer program that, when executed by a processor, performs the above-described method for enhancing robot spatial perception.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: This application reduces the robot's reliance on expensive 3D sensors for spatial perception, achieves efficient and accurate online spatial perception, significantly improves the accuracy and robustness of spatial reasoning, endows MLLM with true 3D spatial understanding capabilities, and alleviates the problems of multi-scale knowledge conflict and catastrophic forgetting. Attached Figure Description

[0017] The present invention will be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the implementation of methods to enhance the spatial perception capabilities of robots.

[0018] Figure 2 This is a flowchart of the sub-steps of the preprocessing step.

[0019] Figure 3 This is a flowchart of the keyframe sequence filtering process.

[0020] Figure 4 This is a flowchart of the keyframe enhancement process.

[0021] Figure 5 This is a flowchart of the sub-steps of the feature encoding process.

[0022] Figure 6 This is a flowchart of the sub-steps of the feature fusion step.

[0023] Figure 7 This is a data flow diagram of the feature fusion step.

[0024] Figure 8 This is a flowchart of the sub-steps of the reasoning optimization process.

[0025] Figure 9 This is a data flow graph of the inference optimization steps. Detailed Implementation

[0026] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0027] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0028] The term "robot" as used in this application may include, but is not limited to, the following types of mobile robots and operational robots: (1) Home service robots: Identify objects (such as tables, chairs, and appliances) in complex home environments, understand room layouts (such as "go around the sofa to turn on the TV"), and plan collision-free paths.

[0029] (2) Industrial logistics robots: identify shelves and pallets in the warehouse, accurately grab and place goods, and conduct inventory counts.

[0030] (3) Autonomous vehicles: Real-time perception of the precise three-dimensional position and motion status of surrounding vehicles, pedestrians, lane lines and traffic signs from onboard monocular cameras.

[0031] Common robot spatial perception solutions mainly rely on 3D sensors or multi-sensor fusion systems, which have high hardware costs, require a lot of computing power, cannot be deployed on low-cost, lightweight robot platforms, and have weak semantic understanding capabilities, making it impossible to combine natural language to complete complex spatial reasoning.

[0032] Currently, the known technical approaches for enhancing a robot's spatial perception capabilities from 2D images mainly fall into the following categories: Route 1: SLAM system based on monocular depth estimation.

[0033] Such systems typically perform the following steps: ① Utilize a monocular depth estimation network (such as MiDaS, DepthAnything V2, etc.) to perform depth estimation for each frame (index is...). t Input image Predicted relative depth map ② Combining visual odometry (VO) or SLAM (such as ORB-SLAM3) techniques, extracting and matching feature points between images to estimate the camera's own motion (pose). ); ③ Project pixels with depth information onto 3D space to construct a dense or semi-dense 3D map. The core of this approach is to chain together the two separate tasks of depth estimation and pose estimation to recover the 3D structure of the scene.

[0034] Route 2: Perception scheme based on end-to-end 3D reconstruction model.

[0035] Inspired by works such as DUSt3R and VGGT, this approach uses a unified feedforward neural network to directly regress the 3D attributes of a scene from a set of 2D images. The steps are as follows: ① Input multiple frames of images as a whole into the model (such as VGGT, i.e., the "visual geometry foundation model"); ② The model directly outputs each frame of image (indexed as...) through a Transformer architecture. i The corresponding camera parameters (intrinsic and extrinsic parameters), dense depth map Even pixel-aligned 3D point clouds ; ③ With the help of a pre-trained Transformer, the geometric information of all input viewpoints can be inferred at once through alternating intra-frame and global self-attention mechanisms, without the need for traditional feature matching and geometric optimization, thus achieving efficient 3D reconstruction.

[0036] Route 3: Spatial Reasoning Based on Multimodal Large Language Models. MLLMs such as Qwen-VL and GPT-4V perform well in 2D image understanding and text generation, but their visual encoders (based on the CLIP paradigm) primarily capture semantic information, lacking understanding of fine 3D structure and spatial relationships. Known improvement schemes mainly fall into two categories: a. Introduce 3D prior information: Before inputting into the MLLM, explicit 3D data such as 3D coordinates, depth maps or point clouds are used as additional input channels or tokens and fused with RGB image features.

[0037] b. Introduce an auxiliary 3D geometric encoder: Design a dual encoder architecture, one branch (such as CLIP) extracts 2D semantic features, and the other branch uses a pre-trained 3D visual geometric encoder (such as VGGT) to extract implicit 3D structural features from 2D images. The two are then fused and fed into LLM for spatial reasoning and question answering.

[0038] Although the aforementioned 2D vision-based spatial perception provides multiple approaches, they generally suffer from one or more of the following technical shortcomings: 1. Spatial understanding and semantic understanding are separated: Route 1 and Route 2 separate 3D geometric reconstruction and 2D semantic understanding (object recognition, command comprehension, etc.) into two independent stages: the map constructed by the SLAM system lacks object-level semantic labels, and the point cloud output by the 3D reconstruction model is difficult to interface with high-level semantic commands. Route 3's fusion method is crude and does not fully utilize the structured information of geometric features to enhance semantic reasoning.

[0039] 2. Strong dependence on explicit 3D data: Most high-performance spatial reasoning models (such as Video-3D LLM and GPT4Scene) rely on pre-acquired depth maps, point clouds or camera poses during training or inference. These data acquisition costs are high and the computational load is large, which limits their real-time application on resource-constrained mobile robot platforms.

[0040] 3. Lack of direct supervision of spatial metric relationships: The known training data of MLLM lacks direct and dense supervision of precise spatial metrics (distance, orientation, size, etc.), and relies on vague text descriptions (such as "left", "far away", etc.), and cannot output precise metric distance and three-dimensional orientation angle. In precise spatial positioning tasks (such as "move forward 30 cm"), the performance drops sharply.

[0041] 4. Insufficient robustness to dynamic and sparse viewpoints: Known methods are prone to error accumulation and geometric collapse when processing dynamic scenes (such as TUM-dynamics) or image sequences with low viewpoint overlap. For example, global attention methods such as VGGT have high accuracy in static scenes, but the computational cost increases quadratically with the number of images, which cannot meet real-time requirements; streaming methods (such as CUT3R, StreamVGGT, etc.) have high computational efficiency, but are prone to long sequence drift and lack robustness when handling large viewpoint changes.

[0042] 5. Feature conflict and catastrophic forgetting: When learning spatial knowledge across multiple scales (from millimeters to kilometers), feature patterns at different scales may conflict, leading to unstable model output; when fine-tuning to enhance spatial perception capabilities, the model is prone to losing its original capabilities for general visual-language tasks, resulting in catastrophic forgetting.

[0043] To address all or some of the aforementioned problems, this application proposes a method for enhancing the spatial perception capability of robots. This method aims to enable robots to accurately perceive three-dimensional spatial information (position, distance, etc.) based solely on 2D visual images, thereby facilitating the planning and execution of actions such as movement and grasping when performing spatial tasks.

[0044] The core logic of the robot spatial perception enhancement method proposed in this application is as follows: deeply integrating the semantic understanding capability of general MLLM with the 3D structural perception capability of visual geometric basic model, supplemented by innovative loss function and data augmentation strategy, relying only on monocular RGB image sequence, to improve the robot's spatial perception capability and the accuracy of distance prediction between objects in 2D images. The implementation process follows a closed-loop process of "data input → preprocessing → feature encoding → feature fusion → inference optimization → data output". Among them, "preprocessing → feature encoding → feature fusion → inference optimization" can be classified as the "processing" stage, which is closely connected through four core modules: "data preprocessing and enhancement module → dual-stream feature encoding and alignment module → multi-layer spatial perception feature fusion module → multi-task spatial inference and optimization module".

[0045] like Figure 1 As shown, in one optional implementation, the robot spatial perception enhancement method proposed in this application includes: (1) Data input stage.

[0046] 2D image sequences are continuously acquired using a monocular camera (such as a monocular RGB camera) at a predetermined sampling frequency. , Indicates the frame image index. T The total number of frames captured. These represent the height and width of the frame image, respectively. Both sample data acquisition and the acquisition of data to be inferred are performed in this manner.

[0047] (2) Data processing stage.

[0048] As mentioned earlier, the image processing stage includes four steps: preprocessing, feature encoding, feature fusion, and inference optimization.

[0049] S1 and 2D image sequences are preprocessed to obtain enhanced keyframe sequences.

[0050] The preprocessing step only applies to the sample data; no preprocessing is performed on the data to be inferred. Alternatively, each frame of the data to be inferred is treated as a keyframe.

[0051] In one optional implementation, the preprocessing step is performed using a data preprocessing and augmentation module. This step preprocesses and specifically augments the input 2D image sequence to improve the robustness and spatial awareness accuracy of subsequent model training. The preprocessing step outputs augmented keyframe sequences and spatial metric question-answer pairs.

[0052] like Figure 2 As shown, the preprocessing step includes two sub-steps: S101, Spatial Awareness Keyframe Acquisition.

[0053] This sub-step primarily addresses the issues of information redundancy and incomplete scene coverage. Continuously acquired 2D image sequences by the robot typically contain a large number of redundant frames (such as redundant information generated when the camera is stationary or moving slowly). Simple uniform sampling cannot guarantee that the selected frames can completely cover the 3D structure of the scene. To solve this problem, this sub-step proposes a greedy sampling strategy based on maximizing 3D spatial coverage. This strategy automatically selects a fixed number (let's say N) of optimal keyframes from the long 2D image sequence, ensuring that the keyframes of the input model can capture the richest 3D scene information with the minimum amount of data.

[0054] The keyframe acquisition method is as follows: The camera view frustum of each frame image in the 2D image sequence is pixelated, and then the keyframes are acquired based on the selected... N The keyframes (sequences) are selected from the 2D image sequence based on the principle of "maximizing the union of the set of view frustums of the selected frame images" (i.e., a predetermined number of frame images).

[0055] Specifically, such as Figure 3 As shown, the implementation method of this sub-step includes: (a) Candidate frame generation and fast geometric estimation.

[0056] Sparse candidate frame extraction: For the input 2D image sequence, perform uniform sampling at fixed time intervals (e.g., every 2 seconds) to obtain a set of sparse candidate frames. , M The total number of candidate frames. i For candidate frame index, .

[0057] This operation can significantly reduce the computational cost of subsequent pose estimation operations while maintaining effective field of view coverage.

[0058] Fast pose and depth estimation: Utilizing lightweight pre-trained visual geometry models (such as mature models like VGG and VGGT) for candidate frame sets. For each candidate frame, fast forward inference is performed to obtain camera intrinsic parameters, pose / extrinsic parameters, and depth map. Furthermore, based on the inferred depth map and camera parameters (intrinsic and extrinsic parameters), the depth map can be back-projected into 3D space to calculate the point cloud.

[0059] (b) View frustum simplification and coverage calculation.

[0060] For the candidate frame set For each candidate frame, using its estimated camera intrinsic parameters, pose / extrinsic parameters, and depth map, the depth map of the observed 3D spatial region of each candidate frame is back-projected onto the 3D world coordinate system. Based on a preset voxel resolution (e.g., 5cm×5cm×5cm), the voxel index of each candidate frame is calculated, resulting in the 3D field-of-view voxel set for that voxel frame. For each candidate frame... Its set of voxels is denoted as .

[0061] (c) Keyframe iteration selection.

[0062] Based on the greedy maximum coverage principle, keyframes are iteratively selected. The set of selected keyframes is initialized. The voxel set has been covered. . Used to record the union of voxels covered by all selected keyframes.

[0063] Repeat the following steps N times (corresponding to the desired predetermined number) to filter out all keyframes: Iterate through each currently unselected candidate frame and calculate its relative value. The number of newly added coverage voxels is determined by selecting the candidate frame that yields the maximum number of newly added coverage voxels from all candidate frames traversed, and denoted as the keyframe for the current round. .

[0064] After each iteration, update the set respectively. and : , .

[0065] After N iterations, the set The contents contained therein are the selected ones. N One keyframe. Based on the acquisition order. N The keyframe sequence formed by these keyframes is denoted as . This keyframe sequence serves as the raw input data for subsequent feature encoding steps.

[0066] Based on the greedy strategy described above, each keyframe added in each iteration can observe previously uncovered 3D spatial regions to the greatest extent possible. Thus, within a limited frame count budget, the input model's coverage of the scene's 3D structure is maximized, effectively eliminating information redundancy caused by stationary or slow-moving cameras.

[0067] S102, Perform spatial metric data augmentation on keyframes.

[0068] This sub-step primarily addresses the problem of insufficient supervision of spatial metrics.

[0069] For subsequent model inference, some distance-related data is required, which is constructed through this sub-step. This sub-step utilizes the 3D information (depth map, or also containing corresponding point cloud data) generated for each keyframe by a pre-trained visual geometry model (same as the previous example) to construct keyframes and spatial metric question-answer pairs (enhanced keyframe sequences) with precise numerical values ​​online, providing spatial metric supervision for subsequent model training.

[0070] Specifically, such as Figure 4 As shown, the implementation method of this sub-step includes: (a) Select the target object from the keyframe.

[0071] Identify any two objects from keyframes. Let these be abstractly defined as object A and object B. The identification results for each object include its center point and bounding box.

[0072] (b) Calculate the spatial metric parameters of the target object.

[0073] The calculated spatial metric parameters include: (1) Distance: Based on point cloud data or depth map, calculate the shortest distance between object A and object B, that is, the Euclidean distance between the nearest points of the bounding boxes of object A and object B, defined as d, which is obtained by the straight-line distance between two points in space.

[0074] (2) Orientation: Construct a spherical coordinate system with the center point of object A as the origin, and calculate the azimuth angle of the center point of object B in the spherical coordinate system. (i.e., horizontal rotation angle) and polar angle (i.e., vertical pitch angle).

[0075] (3) Dimensions: Calculate the actual three-dimensional dimensions of object A or object B, namely length, width and height.

[0076] (c) Generate keyframes and spatial metric question-and-answer pairs.

[0077] Based on the selected target objects and calculated spatial metric parameters, generate question-and-answer pairs about the spatial metric parameters between spatial objects. For example: Question: How far is object A from object B? Answer: d meters (answer with the actual calculated value).

[0078] Question: In what position is object B relative to object A? Answer: Object B is at the lower right corner of object A (or provide the specific azimuth angle). and polar angle ).

[0079] Similarly, multiple sets of keyframes and spatial metric question-and-answer pairs can be generated based on the calculated spatial metric parameters. These question-and-answer pairs contain spatial metric parameters such as distance and orientation between objects in the 2D image, which is equivalent to integrating 3D information into the 2D image and providing accurate spatial supervision information for subsequent model training.

[0080] S2. Perform feature encoding on the keyframe sequence in both planar and feature dimensions.

[0081] Feature encoding is handled by two feature extraction models, where the first model... The second model is responsible for extracting planar features from 2D images. This step is responsible for extracting spatial (geometric) features from 2D images and aligning planar and spatial features through adaptive fusion, laying the foundation for subsequent semantic feature injection. The output of this step is the fused feature.

[0082] like Figure 5 As shown, feature encoding includes three sub-steps: S201. Encode the semantic features of the keyframes.

[0083] Visual encoders employing pre-trained vision-language models (such as those using the Qwen3-VL-8B architecture) ), for keyframe sequences Perform the following processing: (a) Perform feature extraction on each keyframe.

[0084] (b) Output the semantic features of each keyframe. For the index is t The keyframes, whose semantic features are represented as .in, These represent the height and width in the spatial dimensions of the semantic features, respectively. This represents the number of channels for a semantic feature. This semantic feature includes information such as the object's shape, texture, and outline.

[0085] semantic features It is a dense feature map, with spatial dimensions This represents the number of grids after the original keyframes are divided into patches (e.g., 14×14 or 16×16 grids), with each grid position corresponding to a local region of the original image. Channel Dimension This represents a high-dimensional semantic embedding vector for the local region. For example, the feature vector of the grid containing the "blue cup" in a 2D image would be similar to concepts like "cup," "blue," and "cylinder" in the vector space. Feature extraction in Qwen3-VL-8B uses Transformers based on the vit architecture. Feature extraction is performed on each 2D image. Extracting 2D features and Extracting 3D features facilitates the subsequent fusion of corresponding feature blocks.

[0086] S202. Perform spatial feature encoding on the keyframes.

[0087] A pre-trained visual geometric model (same as the previous example) is used as the structural encoder for spatial features. The spatial features of the keyframe sequence are extracted. This sub-step involves processing the keyframe sequence using the following methods: (a) Keyframe sequence Integral input structure encoder ; (b) through An internal alternating self-attention mechanism enables cross-frame information interaction; (c) Output spatial features of spatial geometry perception. For index 1 t The keyframes, whose spatial features are represented as .in, These represent the height and width in the spatial dimensions that define spatial characteristics. The number of channels represents the semantic features. This spatial feature implicitly encodes scene depth, normals, camera pose, and cross-frame correspondences, eliminating the need to display 3D information and making it easier to integrate into MLLM.

[0088] S203. Perform feature alignment and gating fusion of semantic and spatial features.

[0089] This sub-step is responsible for aligning semantic and spatial features, as well as adaptive gating fusion between the two features.

[0090] Due to semantic features and spatial features The spatial resolution and channel dimensions may be inconsistent, so feature alignment must be performed first, and then feature fusion must be performed.

[0091] (a) Resolution alignment.

[0092] Spatial features are obtained through bilinear interpolation. The resolution is adjusted to match the semantic features. Consistent. The reverse is also possible.

[0093] (b) Gated graph generation.

[0094] Using a single-layer linear projection layer and a sigmoid activation function Calculate the gating graph ,in, This indicates the operations performed on Linear operations. This indicates feature concatenation or element-wise addition. This indicates the Sigmoid activation function, which compresses the output to... Interval. Index of frame image t Correspondingly, and The spatial dimensions are consistent, and the element values ​​are in The interval represents the model's dependence weight on semantic features.

[0095] Gating diagram The projection transformation is a per-position transformation. For each of the H×W spatial positions, an independent 7168-dimensional (example) vector is mapped to a 3584-dimensional gate value (and then processed). Compress to The output gate value has a shape of [N, H, W, 3584], and each position and each channel has an independent gate value. gate[c] → 1: This channel and position c trust semantic features (such as texture, color, object category, etc.) more; gate[c] → 0: This channel and position c trust geometric features (such as depth, spatial structure, 3D position, etc.) more. The gate values ​​for different spatial positions and different channels can be completely different, forming a spatially adaptive gating graph.

[0096] (c) Feature fusion.

[0097] Based on computational gating graphs, semantic features and spatial features Perform adaptive fusion and output fused features. (Corresponding index) t ).

[0098] ; In the formula, This indicates the concatenation of semantic features along the channel dimension. and spatial features Output splicing features; This represents element-wise multiplication; Indicates the kernel size as The convolutional layer operations are used to perform local fusion and channel dimensionality reduction on the spliced ​​features. Based on the gating graph, the model dynamically adjusts the fusion weights of semantic and spatial features to adaptively adjust the dependencies between them. For example, at object edges, it relies more on spatial features, while in areas with rich textures, it relies more on semantic features.

[0099] S3, Multi-scale feature fusion.

[0100] In the feature fusion step, the fused features are injected into the MLLM layer by layer to solve the multi-scale knowledge conflict problem and achieve spatial-semantic depth alignment. This step outputs the enhanced MLLM hidden state.

[0101] like Figure 6 The feature fusion process includes two sub-steps.

[0102] S301, Injecting hierarchical features into MLLM for fusion features.

[0103] In this embodiment, unlike injecting the fusion feature only once at the input, the fusion feature is... After being converted into a token sequence, it is injected into the MLLM backbone network in a layered manner through an adapter. For example, for the Qwen3-VL-8B architecture model, it is injected into its LLM part in a layered manner.

[0104] like Figure 7 As shown, the methods for hierarchical injection of fused features include: (a) Determine the injection site.

[0105] Assuming the MLLM backbone network has L hidden layers, choose its... Three key layers serve as injection points.

[0106] (b) Feature adaptation of each injection point.

[0107] Configure a lightweight adapter for each injection point. ( (Indicates the injection point index); fused features Downsampling or pooling is used to correlate the temporal dimension with the hidden state at the corresponding level of the injection point. Alignment, the fused feature after alignment is represented as .

[0108] Since layers of different depths have different requirements for information granularity, the injected features should match the "understanding scale" of that layer. For example, shallow layers require high resolution to preserve fine-grained spatial information; mid-layers require medium resolution and local aggregation; and high-layers require low resolution and high abstraction. Therefore, it is necessary to adjust the fused features accordingly. Perform progressive downsampling. In this embodiment, an adapter is used. Achieve adaptation between injection features and injection point scale.

[0109] adapter A bottleneck structure is adopted, with the dimensions of each layer represented as: hidden_size → hidden_size / 4 → hidden_size. That is, the input and output both match the feature dimension hidden_size of the pre-trained model of the Transformer architecture, and the dimension of the intermediate bottleneck layer is reduced to 1 / 4 of the original dimension. The number of parameters of a single-layer adapter is only 1% to 2% of that of a complete Transformer layer. They do not share weights.

[0110] (c) Status update at each injection point.

[0111] The hidden state is enhanced and (residual connection) updated using the adapters at each injection point with fused features.

[0112] The hidden states of each layer are updated using the following method to achieve continuous fusion of spatial and semantic features: ; In the formula, Indicates the updated hidden state; Indicates adapter Through cross-attention layers, As a key-value pair, enhanced Spatial perception ability.

[0113] S302. Configure MLLM to perform scale-aware MoE mechanism for each hidden layer.

[0114] This sub-step is responsible for resolving conflicts in multi-scale perception. It dynamically adapts to different scene scales using a scale router and a LoRA expert module.

[0115] The implementation method of this sub-step includes: (a) LoRA expert module training.

[0116] Based on the scene scale (e.g., four scales: small objects, near-field desktop, indoor environment, and outdoor environment), multiple LoRA expert modules are trained online, denoted as... , where K is the total number of experts. According to the above 4 scales, K = 4. Each LoRA expert module is represented by a pair of low-rank factorization matrices . The LoRA expert module is used to solve the "knowledge conflict" problem, that is, the same visual pattern is interpreted differently at different scales.

[0117] (b) Scene scale judgment.

[0118] Perform global pooling on the fused features to obtain , and input it into the lightweight scale router R , and output the expert selection probability distribution through the following method: ; In the formula, is the probability vector, represents the weight for activating the i-th expert.

[0119] The above scale router R can be constructed by using a lightweight MLP (Multilayer Perceptron) in conjunction with the Softmax function. Its role is: input the features related to the current sample (or current layer) (here it is ), and output the dynamic scaling factor of each LoRA expert module to weight the contributions of different LoRA expert modules. (c) Dynamically activate experts according to the judged scene scale.

[0120] The outputs of each FFN (Feed-Forward Neural Network) layer of MLLM are enhanced by the weighted sum of the top k (Top-k, k is configured in advance, k < K) experts, and the operation is as follows: ; In the formula, respectively represent the input features and output features of the FFN layer; is the original pre-trained weight; is the i th LoRA expert module's low-rank ( r dimension, for example, 32) factorization matrix ( , is the input / output feature dimension of the original projection layer in the LLM), to ensure that the model adapts to different scale scenarios and avoid feature conflicts.

[0121] The MLLM backbone network does not merge the adapters at the three injection points and the outputs weighted by each layer of experts as independent and parallel input sources. Instead, it adopts a hierarchical and phased fusion strategy.

[0122] The feature maps after gating and fusion are converted into a token sequence, which serves as the initial input to the bottom layer of the MLLM backbone network. This is the only initial data stream received by MLLM. The injection of the three adapters occurs during the intermediate hidden state enhancement stage of the MLLM process. The injection location is at the [missing information - likely a specific location in MLLM]. Layers L / 2 and 3L / 4. Each adapter injects spatial information by overlaying it onto the hidden state of the current layer through residual connections.

[0123] In other words, MLLM does not receive three independent inputs, but rather, during the inference process, a specific intermediate layer actively fuses features. Spatial cues are extracted to correct the current feature representation. The merged features of the expert-weighted output occur during the computation process of the FFN layer within the MLLM.

[0124] The weighted sum output of the scale-aware MoE mechanism directly replaces or enhances the computation results of the original FFN. Expert signals are directly added to the output of each layer in the FFN computation, rather than being used as additional input tokens.

[0125] S4, Reasoning Optimization.

[0126] The inference optimization step defines the training objective and inference output format, optimizes the model through a composite loss function, and achieves multi-task collaborative improvement. The inference optimization step outputs the final spatial perception result.

[0127] like Figure 8 As shown, inference optimization includes two sub-steps: S401, Multi-task Parallel Reasoning.

[0128] This sub-step is responsible for outputting multi-dimensional spatial information. For example... Figure 9 As shown, three parallel prediction heads are constructed after the MLLM backbone network to handle different spatial tasks and achieve multi-dimensional output.

[0129] First prediction head: Visual-Language Inference Head. This prediction head receives the hidden states of the last layer of the MLLM backbone network (i.e., multi-scale fused features obtained from multi-scale feature fusion), generates text answers through autoregression, and uses a Chain-of-Point (CoP, Spatial Action Sequence Inference Chain) mechanism to output an inference chain containing precise coordinates. For example, the output of this prediction head is: First locate the book... Find the cup again Calculations show that the cup is approximately 15 centimeters northeast of the book, with coordinates ( ). cx , cy This achieves the combination of semantic and spatial reasoning.

[0130] Second prediction head: Relative pose regression head. This prediction head receives fused features. global pooling results The current frame image (index) is regressed using the first MLP. t Relative camera pose change relative to the first frame (keyframes are not extracted during inference). , It represents a 3D special Euclidean group to eliminate dependence on a fixed reference frame. Specifically, the second prediction head (first MLP) outputs six values: three-axis translation, Euler angles, quaternions, and axis angles, providing inter-frame relative motion estimation for tasks such as SLAM and navigation, independent of the global coordinate system. Key features: continuous regression, small and fixed output dimension; no text generation involved, fast speed; only outputs relative pose changes, without including scene semantic information.

[0131] The third prediction head: a spatial metric regression head. This prediction head also receives fused features. global pooling results The second MLP regresses spatial metric parameters of the current frame image, such as the distance to the nearest object and the 3D dimensions of the target object, to provide direct numerical supervision for action planning. Specifically, the third prediction head (second MLP) outputs physical space metrics, providing accurate metric parameters for actions such as grasping and obstacle avoidance. Key features: Pure numerical output, no geometric transformation parameters, and the ability to simultaneously infer metric parameters of various dimensions through multi-task (multi-branch) output design. Training the MLP can be accomplished using absolute error loss.

[0132] S402, Composite Loss Calculation (Training Phase).

[0133] In this embodiment, a composite loss function is designed to show the constraint space relationship and balance the losses of each task.

[0134] The loss is defined as follows: ; In the formula, Indicates compound loss; This represents the cross-entropy loss of the language model. This loss supervises the text output of the first prediction head (i.e., the visual-language inference head), specifically monitoring the difference between the autoregressive text output generated by the first prediction head and the ground truth text annotations (i.e., the actual text corresponding to the output content, such as: Question: How far is object A from object B? Answer: 1.2 meters; Question: Where is the red cup on the table relative to the apple? Answer: The red cup is northwest of the apple). This ensures the model's basic semantic understanding and text generation capabilities.

[0135] This represents the relative pose regression loss. It calculates the relative camera pose change predicted by the second prediction head. Relative changes in camera pose The difference between them. The specific calculation method is as follows: ; In the formula, They represent and The translation vector, This represents L1 loss (L1 norm). They represent and rotation vector, This represents the L2 loss (L2 norm). This loss explicitly supervises the camera's motion trajectory, freeing the model from dependence on a fixed reference frame and improving its robustness to changes in viewpoint.

[0136] This represents the spatial metric regression loss. It calculates the difference between the spatial metric predicted by the third-predictor head and the true spatial metric. For example, this loss can be quantified using the SmoothL1 loss function. For instance, it calculates the predicted distance. Distance from reality The loss is: .

[0137] Finally, as one of the key design considerations for composite loss, This represents the spatial consistency loss, which is calculated based on the (enhanced) keyframe sequence. It constrains the 3D distance consistency after backprojection of 2D feature points, aiming to enhance the geometric distance consistency of points (salient feature points or object center points, etc.) on the 2D image feature map after backprojection into 3D space.

[0138] These are all hyperparameters, representing respectively , and The weighted summation weights are used to balance the contribution of each loss to the composite loss.

[0139] The calculation methods include: (a) Select multiple pairs of feature points from the fused feature points.

[0140] In fusion features Two groups of high-response pixels were selected.

[0141] For example, computational fusion features The feature response map is generated; candidate feature points are initially screened based on the set threshold; within the set local window, the feature points with the largest response values ​​are retained according to the non-maximum suppression method; and the feature points are sorted according to the size of the response values ​​and the top feature points are selected.

[0142] Of course, other methods can also be used to select high-response pixels. For ease of explanation, we will use the center point of object A as an example here. and the center point of object B For example.

[0143] (b) Predict the 3D coordinates of each feature point.

[0144] Using the depth map predicted by the 3D geometric encoder (same as the previous embodiment) and camera parameters, the selected feature points (such as...) are... and The back projection onto 3D space yields the corresponding 3D coordinates, which are represented as follows: .

[0145] (c) Obtain the true 3D coordinates of each feature point.

[0146] Feature points (such as those obtained through camera pose or cross-frame triangulation) are acquired. and The true 3D coordinates of ) are represented as follows: .

[0147] (d) Calculate the loss .

[0148] Calculate the predicted distance Distance from reality The loss is calculated using the following method: ; In the formula, G represents the number of feature point pairs collected; g is the feature point pair index. and Let represent the predicted distance and the true distance of the g-th feature point pair, respectively.

[0149] Explicitly enhancing the model's sensitivity to spatial metrics can effectively improve positioning accuracy. This loss explicitly constrains the geometric consistency of the 2D visual features learned by the model in 3D space, forcing the model not only to recognize objects in 2D images but also to understand their spatial metrics in the real world, greatly enhancing the model's sensitivity and accuracy to spatial distance and orientation.

[0150] During the model training phase, iterative training is performed with the goal of minimizing the composite loss. Training stops once the composite loss converges or the maximum number of iterations is reached, and the model parameters are retained. The trained model is then used for task inference optimization.

[0151] For example, the model's training data uses SpaceVista-1M. After acquiring information such as object position and size, 19 types of spatial reasoning tasks were designed, including object counting, distance estimation, path planning, and spatial relationships. Approximately 1 million question-answer pairs (QA) were generated, covering more than 38,000 video scenes.

[0152] The training phase can be broadly described as follows: 1. Feature enhancement and training of the fusion layer provide the model with geometric, depth, and structural information that goes beyond pure semantics; 2. Supervised fine-tuning: The model was fine-tuned for 2 epochs using the SpaceVista-1M dataset with Chain of Thought (CoT) annotations; 3. Reinforcement Learning: Based on the supervised fine-tuning model, reinforcement learning training is performed using data in multiple choice and regression formats, for a total of 2500 steps.

[0153] (3) Data output stage.

[0154] The final model outputs the inference results from the inference optimization phase. For example, the inference results from the three prediction heads include: textual answers (such as inference chains with precise coordinates), camera relative pose changes, and spatial metric parameters, providing accurate and comprehensive decision-making basis for the robot's subsequent motion planning.

[0155] The enhancement method designed in this application has the following significant advantages compared to the known solutions listed above: 1. Significantly improves the accuracy and robustness of spatial reasoning: By deeply fusing 2D semantic and 3D structural features, and supplementing them with spatial consistency loss and orientation-distance (spatial metric) data augmentation, the model's ability to understand the precise spatial metric relationships of objects, such as distance, orientation, and size, is greatly enhanced. This allows for more accurate execution of precise commands such as "move to a location 30 centimeters away from the object." Simultaneously, hierarchical feature injection and scale-aware MoE mechanisms enhance the model's robustness across different viewpoints and scales.

[0156] 2. Achieve efficient and accurate online spatial awareness: The spatial awareness frame sampling strategy ensures that the input frame can maximize scene coverage under limited computing resources. Combined with the feedforward network structure, this solution can process video streams online and in real time to generate dense and semantically rich 3D environment understanding without the need for expensive offline global optimization.

[0157] 3. Empowering MLLM with True 3D Spatial Understanding: By introducing a powerful 3D geometry encoder and employing innovative gating mechanisms and hierarchical feature fusion, this application successfully "implants" geometric priors into MLLM, which excels in semantic understanding. This enables MLLM to answer questions requiring cross-frame perspective reasoning (e.g., what is seen from another angle?) and complex spatial relationship reasoning (e.g., is object A southeast of object B and above object C?).

[0158] 4. Mitigating multi-scale knowledge conflict and catastrophic forgetting: The scale-aware MoE mechanism enables the model to activate different expert knowledge modules when dealing with different scenarios, effectively avoiding knowledge interference. At the same time, through parameter-efficient adapters and LoRA fine-tuning strategies, the powerful general vision-language capabilities of the original MLLM are preserved to the maximum extent.

[0159] 5. Reduced reliance on expensive 3D sensors: The entire process of this application only requires monocular RGB image input to achieve high-performance spatial perception, which greatly reduces the cost of robot hardware and the deployment threshold.

[0160] Based on the ideas of this application, this application embodiment also provides a robot spatial perception enhancement device, which includes a processor and a storage medium storing a computer program. The processor runs the computer program to execute the robot spatial perception enhancement method of the above embodiment.

[0161] Furthermore, this application also provides a computer-readable storage medium including a computer program that, when executed by a processor, performs the robot spatial perception enhancement method described above.

[0162] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A method for enhancing the spatial perception capability of a robot, characterized in that, include: Continuous acquisition of 2D image sequences using a monocular camera; Obtain a keyframe sequence from the 2D image sequence; perform feature encoding on the keyframe sequence in both planar and spatial dimensions; Multi-scale feature fusion is performed on the encoded fusion features; multi-task inference output is performed based on the fusion features and the multi-scale fusion features; In the multi-scale feature fusion stage, the fused features are injected into the multimodal large language model MLLM in layers, and the MLLM is configured to perform a scale-aware MoE mechanism on each hidden layer.

2. The method for enhancing robot spatial perception as described in claim 1, characterized in that, Injecting fused features hierarchically into MLLM, including: Determine the injection point from the MLLM backbone network; Configure an adapter for each injection point and align the temporal temperature of the fused feature with the hidden state of the corresponding layer of the injection point; The hidden state is enhanced and updated by using the adapters at each injection point to fuse features.

3. The method for enhancing robot spatial perception as described in claim 1, characterized in that, The keyframe sequence is subjected to feature encoding in both planar and spatial dimensions, including: Semantic feature encoding is performed on the keyframes; Spatial feature encoding is performed on the keyframes; Perform feature alignment and gating fusion of the semantic features and the spatial features.

4. The method for enhancing robot spatial perception as described in claim 1, characterized in that, Obtaining a keyframe sequence from the 2D image sequence includes: The camera frustum of each frame in the 2D image sequence is converted. Keyframes are selected from the 2D image sequence based on the principle of maximizing the union of the frustum sets of the selected frame images by selecting a predetermined number of frame images. Spatial metric data augmentation is then performed on the keyframes.

5. The method for enhancing robot spatial perception as described in claim 4, characterized in that, Spatial metric data augmentation is performed on the keyframes, including: Select the target object from the keyframe; Calculate the spatial metric parameters of the target object; Based on the target object and the spatial metric parameters, generate spatial metric question-and-answer pairs for keyframes.

6. The method for enhancing robot spatial perception as described in claim 1, characterized in that, Multi-task inference output based on multi-scale fusion features and multi-scale feature fusion includes: Using the first prediction head, and based on the multi-scale fusion features, an inference chain is generated using a Chain-of-Point mechanism; Using the second prediction head, based on the fused features, and employing the first multilayer perceptron (MLP), the relative camera pose change of the current keyframe relative to the first keyframe is generated. Using the third prediction head and based on the fused features, the second MLP is employed to generate spatial metric parameters for the current keyframe.

7. The method for enhancing robot spatial perception as described in claim 6, characterized in that, The composite loss of MLLM's multi-task inference output includes the language model cross-entropy loss of the first prediction head, the relative pose regression loss of the second prediction head, the spatial metric regression loss of the third prediction head, and the spatial consistency loss of the keyframe sequence; the relative pose regression loss, spatial metric regression loss, and spatial consistency loss are weighted and summed by hyperparameters.

8. The method for enhancing robot spatial perception as described in claim 7, characterized in that, The method for calculating the spatial consistency loss includes: Multiple sets of feature point pairs are selected from the fusion features; Predict the 3D coordinates of each feature point; Obtain the true 3D coordinates of each feature point; Calculate the predicted distance of each feature point to the predicted 3D coordinates, and the true distance to the actual 3D coordinates. Calculate the mean difference between the predicted distance and the true distance for each feature point pair.

9. A device for enhancing the spatial perception capability of a robot, characterized in that, It includes a processor and a storage medium, the storage medium storing a computer program, the processor running the computer program to perform the robot spatial perception enhancement method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It includes a computer program, which, when executed by a processor, performs the robot spatial perception enhancement method as described in any one of claims 1-8.

Images