VLA-based body robot SLAM method and device and storage medium

Abstract

According to the VLA-based body robot SLAM method and device and the storage medium, a VLA large model is introduced on the basis of multi-modal fusion SLAM of traditional point cloud geometry, vision and the like, perception is improved from a geometric layer to semantic concept alignment, and the SLAM is more accurate. A VLA large model is used for carrying out dynamic prediction updating on dynamic interference filtering, key frame screening, factor graph relation construction, noise estimation, loopback detection and the like, the real-time requirement is met in the modes of incremental optimization and the like, and the dislocation problem of geometric constraints is corrected through global semantic constraints. The stability of the robot SLAM in extreme scenes such as excessive environmental dynamic interference, loud sensor noise and environmental degradation is improved, and the constructed hierarchical situation map can meet the requirement of a high-order navigation task while the geometric accuracy is met, so that the robot SLAM can be deployed to carriers such as a body-equipped intelligent carrier for subsequent application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent inspection robots, in particular to a wheeled inspection robot, and more particularly to a body robot SLAM method based on VLA, equipment and a storage medium. BACKGROUND

[0002] The geometric theoretical framework of traditional SLAM is relatively mature, and the algorithm has been optimized for many years. It has high positioning accuracy and real-time performance in static and structured environments, high computational efficiency, and modular design, which makes different algorithm components (such as feature extractors and optimizers) can be replaced and upgraded relatively independently, facilitating the fusion with IMU, lidar and other sensors. However, it is sensitive to environmental texture and dynamic objects. In weak texture, repeated texture or severe light change environment, feature extraction and matching will become difficult, leading to tracking failure. Dynamic objects can easily introduce error data, causing pose estimation drift. The map constructed by traditional SLAM is mainly a geometric map (such as point cloud map, feature map), which lacks semantic information and is difficult to support high-level tasks (such as "go to the kitchen to get a cup"). For long-term running cumulative error, although loop detection can correct the error, in large-scale or appearance change obvious scene, it may still be unable to effectively identify the loop, resulting in inconsistent map.

[0003] SLAM based on visual-linguistic-action (motion) model (VLA) is a further development of semantic SLAM. Its core idea is to use the powerful open vocabulary semantic understanding and cross-modal alignment capability of VLA to promote SLAM from the geometric level to the semantic level, so that the machine can understand "what" and "the relationship between objects" like a person; it has strong scene understanding and reasoning ability, can understand unknown objects and scenes in an open environment, and supports high-level task instructions based on natural language, making it possible for the robot to truly make autonomous decisions; by distinguishing static structures and dynamic objects through semantic information and processing them separately, it can effectively suppress the interference of dynamic objects on pose estimation, and has excellent dynamic environment robustness.

[0004] Term explanation: SLAM: simultaneous localization and mapping, real-time positioning and mapping; PTP: Precision Time Protocol, precise network time protocol; PPS: Pulse Per Second, hardware pulse; ICP: Iterative Closest Point, Iterative Closest Point algorithm; Odom: odometer, odometer; IMU: Inertial Measurement Unit EKF: Extended Kalman Filter VLA: Vision-Language-Action GAT: Graph Attention Network SUMMARY

[0005] The embodiment of the present application provides a VLA-based embodied robot SLAM method, device and storage medium, which can at least solve one of the technical problems in the background art.

[0006] To achieve the above-mentioned purpose, the following technical solutions are adopted in the present application: A VLA-based embodied robot SLAM method, comprising the following steps executed by a computer device, S1, calibrating the internal parameters of a sensor and the external parameters between multiple sensors, and performing space-time synchronization on the data published by multiple modes to ensure that the data is collected at the same time; and performing distortion removal on the radar point cloud according to the IMU data; S2, performing semantic and prediction analysis on the distortion-removed point cloud and visual data based on a VLA large model, outputting a weight map, filtering data frame noise according to different weights, retaining data with high weights, and then estimating and outputting the front-end odometry pose of the robot by calculating the IMU pre-integral, point cloud residual and visual residual; S3, constructing key frames and local maps according to the front-end odometry pose; combining the VLA large model and visual information to perform semantic extraction on the point cloud, and abstracting the point cloud into a hierarchical situation map; in addition to geometric features, semantic-related labels and semantic relationships are extracted, and the semantic information is completed and optimized by the VLA, so that the object pose and relationship in the scene are saved together for use in high-level navigation tasks; S4, through the AVA-VLA mechanism based on belief state active attention update, the objects appearing in the scene are updated in real time and dynamically, and it is judged whether the environmental objects in the scene are temporary dynamic objects, and the influence of these objects is ignored in the final map and subsequent S6 factor graph optimization; S5, generating fusion features of key frames based on laser point cloud features, visual features and VLA semantic features in the key frames and a weight matrix, and calculating loop similarity through the fusion features; when the loop similarity based on the fusion features meets a threshold value, geometric feature fine registration is performed, more accurate loop factors are obtained, and the loop factors are inserted into the S6 factor graph; S6 inserts odometer factors based on the front-end odometer in S2 and loop closure factors based on loop closure detection in S5. It also performs semantic keyframe filtering, inter-factor connection prediction, and adaptive noise estimation based on VLA to construct a multivariate factor graph. Incremental global optimization is used to update the relationships in the factor graph, thereby updating the global map.

[0007] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0008] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0009] As can be seen from the above technical solutions, the embodied robot SLAM method based on VLA of the present invention introduces a large vision-language-action (VLA) model (such as RT-2, OpenVLA), which elevates robot perception from "point cloud matching" to "concept alignment". The VLA model encodes visual observation, instructions and actions into tokens, enabling the SLAM system to possess "semantic common sense"—for example, it knows that "televisions are usually placed against the wall." This prior knowledge can serve as a powerful global constraint to correct geometric pose in extreme environments with high sensor noise or environmental degradation. This technical approach retains the geometric accuracy of traditional sensors while overcoming the global consistency bottleneck through semantic enhancement; moreover, the hierarchical situation map can also support higher-level navigation tasks (such as going to the kitchen to get a cup).

[0010] In summary, this invention introduces a large VLA model on the basis of traditional multimodal SLAM fusion, including point cloud geometry and vision. This elevates perception from the geometric layer to semantic concept alignment. The large VLA model is used to dynamically predict and update dynamic interference filtering, keyframe selection, factor graph relationship construction, noise estimation, and loop closure detection. Real-time requirements are met through incremental optimization, and global semantic constraints are used to correct geometric constraint misalignment. This improves the stability of robot SLAM in extreme scenarios such as excessive dynamic interference, high sensor noise, and environmental degradation. Furthermore, while maintaining geometric accuracy, the constructed hierarchical situation map can also meet the needs of high-order navigation tasks, enabling its deployment on embodied intelligence and other platforms for subsequent applications. Attached Figure Description

[0011] Figure 1 This is a flowchart of the present invention; Figure 2 The final mapping result of this embodiment of the invention is a point cloud map with RGB color. Figure 3 This is a schematic diagram of laser point cloud map updating during the SLAM process in an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0013] like Figure 1 As shown, the embodied robot SLAM method based on VLA described in this embodiment executes the following steps via computer equipment: S1. Calibrate the intrinsic parameters of sensors such as IMU and camera and the extrinsic parameters between multiple sensors, and perform spatiotemporal synchronization of multimodal data to ensure that the data is collected at the same time; at the same time, perform distortion correction processing on radar point cloud based on IMU data.

[0014] S2. Based on the VLA large model, semantic and predictive analyses are performed on the distortion-free point cloud and visual data, and a weight map is output. Data frame noise is filtered according to different weights, and data with high weights is retained. Then, the robot front end odometry pose is estimated and output by calculating IMU pre-integration, point cloud residuals and visual residuals.

[0015] S3. Based on the odometry pose, construct keyframes and local maps; combine VLA large model and visual information to extract semantics from point clouds and abstract them into a hierarchical situation map. In addition to geometric features, extract semantically related labels and semantic relationships, and use VLA to complete and optimize semantic information. Save the poses and relationships of objects in the scene together for use in advanced navigation tasks.

[0016] S4. Through the AVA-VLA mechanism, the objects appearing in the scene are dynamically updated in real time. It is determined whether the environmental objects are temporary dynamic objects. The influence of these objects is ignored in the final map and subsequent S6 factor graph optimization.

[0017] S5. Generate fusion features of keyframes based on laser point cloud features, visual features, VLA semantic features and weight matrix in keyframes. Calculate loop closure similarity using fusion features. When the loop closure similarity based on fusion features meets the threshold, perform fine registration of geometric features to obtain more accurate loop closure factors, and insert them into the factor map in S6.

[0018] S6 inserts odometer factors based on the front-end odometer in S2 and loop closure factors based on loop closure detection in S5. It also constructs a multivariate factor graph based on VLA, including semantic keyframe filtering, inter-factor connection prediction, and adaptive noise estimation. In addition, to meet real-time requirements, incremental global optimization is adopted to update the factor graph relationships and thus update the global map.

[0019] The following are detailed explanations: Step 1: Sensor accuracy calibration and spatiotemporal synchronization of multimodal data The first step in implementation is to establish a sub-millisecond spatiotemporal reference to ensure the spatiotemporal consistency of laser point clouds, visual images, and inertial measurement data.

[0020] 1.1 Hardware Layer Synchronization: A PTP or PPS-based synchronization scheme is used to synchronize the data from the inertial navigation measurement unit (IMU), LiDAR, and camera. The IMU samples at a high frequency of 300Hz, using its time as the master clock reference. A unified external trigger signal is output through a hardware synchronization board to control the exposure and sampling time of the LiDAR and camera. The process is as follows: Figure 1 As shown: The STM32 board sends a PWM signal and a GPRMC timestamp signal (where the timestamp contained in the GPRMC protocol is the IMU master clock) to hard trigger radar acquisition and assigns the timestamp to the current point cloud frame. At the same time, it sends a PWM signal to hard trigger camera acquisition. While acquiring and publishing data, the radar driver writes the current frame timestamp to shared memory. When the camera driver receives the hard trigger signal and performs exposure and data acquisition, it reads the timestamp from shared memory as the time of the current exposure image frame, thereby completing hard synchronization of multiple sensors.

[0021] 1.2 Camera intrinsic parameter calibration and dynamic extrinsic parameter calibration: First, 10-20 sets of images with different poses are acquired using a checkerboard calibration board, sub-pixel corner points are extracted, the correspondence between the world coordinate system and the image coordinate system is established, and the distortion parameters of the camera lens are obtained, thus obtaining the precise intrinsic parameters of the camera; The system defines the robot's body center coordinate system W, the lidar coordinate system L, the camera coordinate system C, and the IMU coordinate system b.

[0022] The camera-IMU extrinsic parameter calibration is achieved by rapidly rotating and translating the handheld sensor array, and constructing constraint equations using visual feature points and IMU pre-integrated values. The core formula is as follows:

[0023] Reprojection error:

[0024] in For IMU pre-integration displacement, For camera projection function, Image feature point observations By jointly optimizing the reprojection error of multiple feature points and the IMU pre-integration constraint, the extrinsic parameters from the camera to the IMU are obtained. ,in This is the extrinsic parameter matrix for the camera and IMU.

[0025] For the extrinsic parameter calibration of the camera and LiDAR, planar feature points are extracted from the LiDAR point cloud and feature points are extracted from the camera image. Then, each LiDAR point is projected onto the camera pixel coordinate system. ,in K Represents the camera intrinsic parameter matrix The extrinsic parameter calibration is then transformed into minimizing the point-to-plane distance calculation, the expression of which is: ,in This represents the corresponding point cloud region in the image. This is the extrinsic parameter matrix for the camera and radar.

[0026] 1.3 Laser Point Cloud Motion Distortion Correction: Considering the point cloud scanning characteristics of lidar, during the acquisition of a single frame of point cloud data (approximately 100ms), the system utilizes the high-frequency angular velocity and acceleration provided by the IMU to compensate each point for its pose at the start of the frame; let the start time of the laser scan be... The reference coordinate system is For laser points emitted / received at any time during the scan (In the carrier coordinate system at that time), obtained through IMU integration. (time (Coordinate transformation from the carrier coordinate system to the reference coordinate system W) transforms the current point to the reference coordinate system: By performing transformation processing on all points in the current frame, the distortion-free point cloud frame can be obtained. .

[0027] Step 2: Tightly Coupled Odometer Front End with VLA Inference Guidance After processing the point cloud data, front-end calculations can be performed. The front-end odometry is based on the Error State Iterative Kalman Filter (ESIKF) framework, which utilizes the saliency prediction capability of the VLA model to enhance the robustness of state estimation.

[0028] 2.1 VLA Active Visual Guidance: Based on environmental semantics and task instructions, a "semantically enhanced observation weight map" is generated to guide the selection and weighting of sensor data. Unlike traditional methods that treat all visual and semantic features equally, VLA encoders (such as DepthVLA) output a visual saliency weight map. The model uses its pre-trained knowledge to identify objects with "high semantic value" in the environment; it can also select only feature points located on static objects (such as walls and the ground) while avoiding the use of feature points on dynamic objects (such as pedestrians and vehicles).

[0029] Output semantic tokens using VLA models (such as QDepth-VLA). and the corresponding three-dimensional coordinates Then define VLA semantic confidence. Laser and visual weights are respectively , And satisfy

[0030] 2.2 Multi-sensor prediction: The LiDAR uses VLA semantic weights W_sem to filter dynamic noise points (such as pedestrians and moving objects) while preserving static semantic features (such as "wall" and "pillar"), generating a filtered point cloud. The camera extracts semantically labeled ORB feature points after acquiring images. The inertial navigation system outputs the raw IMU measurements [ω_imu, a_imu], which are then pre-integrated to obtain the relative pose. .

[0031] 2.3 Sensor residual calculation and optimization: Update the robot pose using data filtered by VLA weights, and calculate the residuals of each sensor; The residual from the lidar point to the plane is Where N is the number of points-to-faces matched. Let be the normal vector and distance parameters of the i-th plane.

[0032] Visual luminance residual is ,in For pixel coordinates, For camera projection function, For three-dimensional coordinate points in the world coordinate system, and For translation and rotation transformations from the world coordinate system to the robot coordinate system, and The extrinsic parameters for the lidar to the camera.

[0033] Therefore, fusion optimization of pose is equivalent to minimizing the residual: ,in The predicted coordinates of the VLA semantic point in the current pose.

[0034] The system iteratively updates and alternately processes the geometric residuals of the radar and the photometric residuals of the vision, ultimately obtaining the optimized front-end pose. This dual update mechanism ensures that the system can still maintain pose tracking even when visual texture is missing or radar motion degradation occurs (such as in long corridor environments).

[0035] Step 3: Automated construction of VLA semantic extraction and hierarchical situation graphs (S-Graphs) This solution abstracts geometric point clouds into situational graphs with a four-layer structure, leveraging the logical reasoning capabilities of the VLA model to achieve scene modeling. The layered situational graphs (S-Graphs) transform traditional 3D point cloud maps into machine-understandable structured concepts through a hierarchical progression of "geometric abstraction → semantic association → logical reasoning → global situation." The VLA model plays two main roles in this process: Semantic parser: Extracts semantic labels and relationships (e.g., "the table is in front of the chair") from images / point clouds; Logical reasoner: Dynamically constructs a hierarchical graph structure based on language instructions or scenario rules.

[0036] 3.1 Semantic Feature Extraction: Inputting 3D point cloud P and synchronized RGB data, using the QDepth-VLA architecture, the output is a geometry-enhanced semantic token and relation description, with the following semantic-geometric joint representation: ,in For semantic tags (such as pillars, doors, etc.), As the initial confidence level, The centroid of the point cloud in 3D spatial coordinates; The confidence level is corrected by optimizing the residual between the lidar depth and the VLA predicted depth. ,in Predict depth for VLA, For lidar depth, experience points 3.2 Layered Architecture: This solution consists of a four-layer S-Graphs architecture: a geometric point cloud layer, a semantic entity layer, a logical relationship layer, and a global consistency layer. Geometric point cloud layer: laser point cloud octree map The resolution is customizable and used as a priori map for positioning and navigation. Semantic entity layer: composed of nodes and edges Node: Semantic entity extracted by VLA parsing;

[0037] Edges: Geometric relationships between entities (such as "containment", "adjacency", etc.), generated through VLA relationship descriptions; The relationship definition rules are implemented through AABB bounding box intersection and union comparison; Logical relation layer: Composed of nodes and edges, it is abstracted based on the second semantic entity layer and a rule relation library is established. Node: A higher-level abstraction of semantic entities.

[0038] Edge: Logical reasoning relationships based on a rule base.

[0039] Such as semantic entity layer This indicates that the table and chair are adjacent; at the logical relationship level, it is... , indicating a table and chair set.

[0040] Global Consistency Layer: Semantic Topology Graph ,node For key semantic targets (such as "meeting room door" or "corridor corner"), edges Constraints on the relative distance / direction between targets.

[0041] The VLA model constructs "functional connections" by recognizing interactive elements (such as switches and door handles). This connection method, which transcends geometry, enables the system to establish logical topologies such as "the kitchen is connected to the dining room through this door," and even when the door is closed, the robot can still recognize path accessibility on the logical layer.

[0042] 3.3 VLA Semantic Completion and Optimization: Dynamic semantic filtering is performed by marking dynamic targets through VLA temporal reasoning (such as TrackVLA) to remove them from the map; then semantic size prior constraints are applied, and the categories are known. Target size (e.g., column height) Optimize semantic unit size ,in For the observation size, For the weighting coefficient Based on the above processing, different situation maps are abstracted and extracted from the original point cloud geometric map. The poses and relationships of objects in the map scene can be saved together for use in subsequent advanced navigation tasks.

[0043] Step 4: Active Attention Update Based on Belief State (AVA-VLA) To enable real-time dynamic updates of objects in the scene, the solution introduces the AVA-VLA framework, which treats the mapping process as a partially observed Markov decision process (POMDP). It utilizes historical context to optimize the current perception weights. The core idea is to model environmental uncertainty through dynamic belief states and combine it with an active inference mechanism to optimize the attention sampling strategy, thereby achieving efficient visual information processing.

[0044] 4.1 Belief State Representation: First, the system maintains a neural approximation of the belief state vector to represent the robot's probabilistic understanding of the environmental layout; assuming that there exists in the visual scene... K A potential goal, a state of belief. Defined as target state The posterior probability distribution: ,in For historical observation sequence, For historical attention action sequences (such as window coordinates and scale) 4.2 Active Attention Action Selection: Based on the principle of minimizing variational free energy (VFE), the next attention action is selected. :

[0045] This can be simplified to maximizing the expected information gain (EIG). ,in Let be the entropy function, representing the uncertainty of the belief state.

[0046] 4.3 Belief State Update: New Data Observed Then, the belief state is updated using Bayesian filtering: If we use a particle filter approximation, it can be expressed as an update of particle weights:

[0047] The system predicts the "motion lifetime" of environmental entities using a VLA model. For objects identified as "temporary dynamics" (such as packages or moving vehicles), the system marks them as "transient factors" during mapping and automatically ignores the geometric contributions of these nodes during global loop closure and factor graph optimization, ensuring the long-term effectiveness of the map.

[0048] Step 5: Cross-modal global loop closure detection and relocation Loop closure detection employs a two-stage strategy from "multimodal detection with semantic embedding" to "geometric verification." It leverages the cross-modal capabilities of a large model to achieve zero-sample relocalization. In situations where a single modality is susceptible to environmental interference (such as visual failure under varying illumination or laser degradation in textureless regions), multimodal fusion can enhance detection reliability through information complementarity.

[0049] 5.1 Multimodal feature fusion: First, extract laser point cloud features, visual features and VLA semantic features. Generate fusion features of key frames based on pre-trained weight matrices. Determine and store key frames based on fusion feature attributes and inter-frame distance for use in subsequent loop closure detection. Laser point cloud features: Global features of the point cloud are extracted. Scan Context, a polar coordinate-based circular histogram, transforms the 3D point cloud into a 2D matrix. This is achieved by dividing the point cloud into rings and sectors in a polar coordinate system, forming an N×M matrix where each element represents the maximum height value of a point within its corresponding spatial bucket, thus encoding the global geometry of the scene. Its core attribute is the height distribution matrix of the polar coordinate grid, and its similarity can be measured using cosine distance or Hamming distance.

[0050] in Given the Scan Context matrices (N x M dimensions) for two scenes, The trace of the matrix (the sum of the elements on the main diagonal). It is the Frobenius norm.

[0051] Visual Features: CLIP extracts global visual features and trains on massive image-text pairing datasets. This makes similar images and texts closer together in the embedding space, while samples of different modalities are spaced further apart. This enables CLIP to achieve zero-shot inference, meaning it can classify or understand any visual task without fine-tuning for specific tasks. CLIP maps images to 512-dimensional real-valued vectors through contrastive learning. Its core property is alignment with the language embedding space, and similarity is measured by cosine distance.

[0052] in Let be the CLIP embedding vector of image I.

[0053] VLA semantic features: Extract semantic graph structure features of VLA modalities, construct spatial logical relationships between targets through graph structure encoding, and the spatial modeling relationship is as follows: ,node For key semantic targets (such as "meeting room door" or "corridor corner"), edges Relative distance / direction constraints between targets. Node features are... (Category, center coordinates, size), edge weight is ,in Indicates whether entities i and j are adjacent.

[0054] After obtaining multimodal features and feature metrics, loop closure similarity scores are generated separately for each modality, and then a comprehensive decision is made through methods such as voting and weighted summation. The loop closure similarity scores are calculated based on a pre-trained weight matrix, and the weights are dynamically adjusted based on modality reliability.

[0055] The reliability of loop closure similarity scores in combination with modal attributes: ,in For modality Similarity (laser L, visual V, VLA semantic S) The weights are dynamic and adaptively adjusted based on modal reliability. , ,in For modality The standard deviation of feature similarity.

[0056] 5.2 Loop Closure Detection and Geometric Verification: After obtaining the fused feature similarity between keyframes, determine whether their similarity meets the loop closure threshold. ,in If the threshold is set, then the loop closure requirement is met, and this frame is included in the loop closure comparison frame range. After obtaining the loop closure candidate frames, the system performs 6-DoF geometric alignment. Based on the system point cloud features, it uses the ICP fine registration algorithm to perform point cloud registration and obtain the inter-frame transformation matrix. This allows for the acquisition of more accurate inter-frame relationships, and the establishment of closure factors is incorporated into factor graph optimization.

[0057] Step 6: Global Optimization of Factor Graph Based on VLA Dynamic Fusion In SLAM, the core of factor graphs is to construct constraints using multi-source sensor data. Its optimization is essentially a nonlinear least squares problem, requiring iterative linearization for solution. Traditional factor graphs mainly include odometry factors, geometric feature factors, visual factors, and loop closure factors. However, geometric or visual constraints are modeled independently. For example, considering only object category consistency or spatial distance without considering correlation can easily lead to mismatches. Therefore, VLA (Virtual Lack of Context) is introduced on top of traditional odometry and loop closure factors. Through scene semantic understanding, the connection relationships between factors are dynamically adjusted, thereby solving problems such as "mismatches in similar environments" and "interference from dynamic obstacles."

[0058] The node insertion in a traditional factor graph is still as follows. Odometry factor: Calculated by the front end to determine the inter-frame odometry relationship, expressed as... ,in Let k represent the robot pose at time k. For the pose transformation between time k and k+1; Loop closure factor: After loop closure is detected in step five, it is calculated based on the final inter-frame transform matrix of the loop closure. The cyclic factor was calculated. ,in Let be the transformation matrix between loop frames k and c.

[0059] However, under the premise of VLA integration, the insertion of factors is different from the traditional method. Instead of inserting factors sequentially with fixed geometric relationships, the factor graph topology is dynamically reconstructed. Factors that meet the conditions are treated as nodes in the factor graph for subsequent processing, reducing redundancy and erroneous constraints.

[0060] 6.1 VLA-driven dynamic reconstruction of factor graphs First, based on the input RGB image sequence and instruction set, semantic keyframes are filtered, and their scores are calculated. Only retain The frames are used as factor graph nodes to reduce redundancy, where The set of semantic categories detected by VLA (such as 'door', 'chair', etc.). This represents the category diversity entropy (the higher the value, the more information the scene contains). For the semantic topology subgraph of the current frame, For global semantic graph The coverage of the subgraph to the global graph (the higher the value, the more representative the keyframes). Secondly, cross-modal factor connectivity prediction is performed. Latent factor connectivity is predicted through semantic relation reasoning, addressing the problem that traditional methods are ineffective in textureless regions. Instead of using traditional factor graph nodes in textureless environments, predicted nodes are used. The formula for predicting connectivity probability is as follows: ,in Factor join between frames i and j (1 indicates a join exists). semantic sequences extracted for VLA For the semantic embedding of the CLIP model, For feature splicing, Activate sigmoid 6.2 VLA Adaptive Noise Estimation and Dynamic Constraints Traditional factor graphs use a predefined covariance matrix Σ (such as odometry noise), while VLA dynamically adjusts Σ through scene complexity assessment, achieving "strict constraints for difficult scenes and loose constraints for easy scenes." The scene difficulty score output by VLA is used to illustrate this. The visual factor noise is dynamically modeled, and the noise formula is: ,in Given the basic covariance matrix, r is the difficulty coefficient. The difficulty level is determined by multiple factors, including texture richness, proportion of dynamic targets, and lighting. Based on the above formula, adaptive noise for different frames is calculated and input into the factor graph nodes to ensure the establishment of more accurate factor graph relationships.

[0061] Based on the above factor relationships, a nonlinear optimization objective function is constructed to jointly optimize the accumulated error:

[0062] in For the set of loopback frames; To meet real-time requirements, a Bayesian tree structure is used to linearize and reorder only the subgraphs affected by the new cyclic factor, enabling incremental global optimization while maintaining global consistency. Iterative updates are performed based on the Levenberg-Marquardt (LM) method. Then, all factor relationships are updated, where J is the Jacobian matrix and r is the residual vector.

[0063] The globally optimized trajectory is used to correct the global point cloud map and simultaneously update the semantic positions and relationships of entities in the scene map. The system maintains a multi-class confusion matrix and continuously improves the confidence of object labels based on the number of observations, ultimately generating a "cognitive map" with high geometric accuracy and deep semantic consistency for use in basic navigation and high-level task navigation.

[0064] The following are examples: Example steps: Step 1: Perform spatiotemporal synchronization and intrinsic / extrinsic parameter calibration on multiple sensors to ensure that sensor data are synchronized in time and that camera intrinsic parameters and sensor extrinsic parameters are accurate, and write them into the parameter file; Step 2: Train a large VLA model based on the actual environment or use a general VLA model, and deploy special semantics and action instructions (such as pillars, doors, windows, corridors, etc.) based on the actual environment. Step 3: The robot is placed in the mapping scene and begins mapping. The robot moves along the scanning route. Step 4: Point cloud distortion correction: First, based on the acceleration and angular velocity of the IMU, all point clouds at other times in the original frame are unified to the pose at the beginning time to obtain the corrected point cloud frame set. Step 5: Front-end odometry calculation: First, the pose is estimated based on IMU data. Then, the camera image is input into the QDepth-VLA large model, outputting a semantic label set and dynamically adjusting the weights according to the semantic confidence. Visual residuals and laser point cloud residuals are calculated simultaneously. The residuals and dynamic weights of VLA prediction, vision and point cloud are fused together to perform residual minimization calculation and obtain the optimized front-end odometry. Step 6: Based on the VLA semantic extraction in Step 5, the QDepth-VLA architecture is adopted to output geometrically enhanced semantic tokens and relation descriptions. Based on this result, a layered situation map is constructed and saved, and the point cloud map is constructed into a four-layer situation map. Step 7: For possible dynamic changes of objects in the scene and uncertainties in complex scenes, the AVA-VLA strategy is adopted to actively update attention. The uncertainty of the target is tracked in real time through belief state. When there are uncertain scenes such as occlusion and dynamic objects, higher visual attention is assigned to these areas to reduce redundancy, improve response speed and accuracy, and these are marked as "instant factors" for corresponding processing in subsequent factor map optimization and global detection. Step 8: Factor Graph Construction: Based on traditional odometry factors and loop closure factors, a VLA-driven fusion strategy is adopted to perform semantic filtering on the current frame, calculate semantic keyframe scores and cross-modal factor connection predictions, and only keyframes that meet the conditions are processed as factor nodes; and adaptive noise estimation and dynamic constraints are performed according to VLA, with its covariance matrix dynamically calculated and updated in real time according to the actual scene, and "soft constraints" are applied to the interference of dynamic objects on pose, bringing in factor graph nodes that meet the conditions to construct a more accurate factor graph relationship; Step 9: Loop closure detection: In addition to using laser point cloud geometric features and visual features for detection, semantic graph structure features of inter-frame VLA modalities are extracted through VLA to construct spatial logical relationships between targets. Based on pre-training and dynamically adjusted weights, combined with geometric features, visual features and VLA semantic features, joint loop closure detection is performed. Frames that meet the conditions are added to the loop closure frame sequence to complete the loop closure search. Step 10: After detecting loop closure, determine whether the scene is a geometrically degraded scene. If not, use the ICP fine registration algorithm to perform geometric alignment on the two frames, obtain the inter-frame transformation matrix, and establish the loop closure factor insertion into the factor graph sequence in Step 8. Step 11: Incremental optimization: After detecting a loop, perform incremental global optimization of the factor graph, linearizing and reordering only the subgraphs affected by the new loop factor; iteratively update based on the Levenberg-Marquardt (LM) method or the Gauss-Newton method, and update all factor relationships; Step 12: Based on the optimized factor graph relationships and trajectories from the previous step, correct and update the global point cloud map and hierarchical situation map, and update the semantic locations and relationships of entities in the scene; Step 13: Repeat steps 4-12; Step 14: Upon receiving the command to end the map scan, save the map and generate a "cognitive map" with high geometric accuracy and deep semantic consistency for use in basic navigation and high-level task navigation.

[0065] Combination Figure 2 and Figure 3 As can be seen, this invention introduces a large VLA model on the basis of traditional multimodal SLAM fusion of point cloud geometry and vision, which elevates perception from the geometric layer to semantic concept alignment. The large VLA model is used to dynamically predict and update dynamic interference filtering, keyframe selection, factor graph relationship construction, noise estimation, and loop closure detection. Real-time requirements are met through incremental optimization, and the misalignment of geometric constraints is corrected through global semantic constraints. This improves the stability of robot SLAM in extreme scenarios such as excessive dynamic interference, high sensor noise, and environmental degradation. In addition to meeting geometric accuracy, the constructed hierarchical situation map can also meet the requirements of high-order navigation tasks, enabling it to be deployed on embodied intelligence and other carriers for subsequent applications.

[0066] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0067] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0068] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the VLA-based embodied robot SLAM methods described in the above embodiments.

[0069] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0070] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0071] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0072] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A SLAM method for embodied robots based on VLA, characterized in that, Includes the following steps, S1. Calibrate the intrinsic parameters of the sensors and the extrinsic parameters among multiple sensors, and perform spatiotemporal synchronization of the data released by the multimodal sensors to ensure that the data are collected at the same time; at the same time, perform distortion correction processing on the radar point cloud based on the IMU data. S2. Based on the VLA large model, semantic and predictive analysis is performed on the distortion-free point cloud and visual data to output a weight map. Data frame noise is filtered according to different weights, and data with high weights is retained. Then, the robot front end odometry pose is estimated by calculating IMU pre-integration, point cloud residual and visual residual. S3. Based on the odometry pose, construct keyframes and local maps; combine VLA large model and visual information to extract semantics from point clouds and abstract them into a hierarchical situation map. In addition to geometric features, extract semantically related labels and semantic relationships, and use VLA large model to complete semantic information and optimize relationships. Save the poses and relationships of objects in the scene together for use in advanced navigation tasks. S4. Through the active attention update mechanism based on belief state, i.e., AVA-VLA, the objects appearing in the scene are dynamically updated in real time. It is determined whether the environmental objects are temporary dynamic objects. The influence of these objects is ignored in the final map and subsequent S6 factor graph optimization. S5. Generate fusion features of keyframes based on laser point cloud features, visual features, VLA semantic features and weight matrix in keyframes. Calculate loop closure similarity using fusion features. When the loop closure similarity based on fusion features meets the threshold, perform fine registration of geometric features to obtain more accurate loop closure factors, and insert them into the factor map in S6. S6 inserts odometer factors based on the front-end odometer of S2 and inserts loop closure factors based on loop closure detection of S5. It also constructs a multivariate factor graph based on the VLA large model by performing semantic keyframe screening, inter-factor connection prediction, and adaptive noise estimation. Furthermore, it adopts incremental global optimization to update the relationships in the factor graph, thereby updating the global map.

2. The embodied robot SLAM method based on VLA according to claim 1, characterized in that: S1 specifically includes, 1.1 Hardware layer synchronization: A synchronization scheme based on PTP or PPS is adopted to synchronize the data of the inertial navigation measurement unit, lidar and camera in time; the IMU performs high-frequency sampling at 300HZ and uses its time as the master clock reference. A unified external trigger signal is output through the hardware synchronization board to control the exposure and sampling time of lidar and camera. The STM32 board sends PWM signals and GPRMC timestamp signals. The timestamp contained in the GPRMC protocol is the IMU master clock, which hard-triggers radar acquisition and assigns the timestamp to the current point cloud frame. At the same time, it sends PWM signals to hard-trigger camera acquisition. While acquiring and publishing data, the radar driver writes the current frame timestamp to shared memory. When the camera driver receives the hard-trigger signal, it performs exposure and data acquisition, and reads the timestamp from shared memory as the time of the current exposure image frame. This completes hard synchronization of multiple sensors. 1.2 Camera intrinsic parameter calibration and dynamic extrinsic parameter calibration: First, a checkerboard calibration board is used to acquire 10-20 sets of images with different poses, sub-pixel corner points are extracted, the correspondence between the world coordinate system and the image coordinate system is established, and the distortion parameters of the camera lens are obtained, thus obtaining the precise intrinsic parameters of the camera. The system defines the robot's body center coordinate system W, the lidar coordinate system L, the camera coordinate system C, and the IMU coordinate system b; The camera-IMU extrinsic parameter calibration is achieved by rapidly rotating and translating the handheld sensor array, and constructing constraint equations using visual feature points and IMU pre-integrated values. The core formula is as follows: Reprojection error: in For IMU pre-integration displacement, For camera projection function, These are the observation values ​​of image feature points; By jointly optimizing the reprojection error of multiple feature points and the IMU pre-integration constraint, the extrinsic parameters from the camera to the IMU are obtained. ,in The extrinsic parameter matrices for the camera and IMU; For the extrinsic parameter calibration of the camera and LiDAR, planar feature points are extracted from the LiDAR point cloud and feature points are extracted from the camera image. Then, each LiDAR point is projected onto the camera pixel coordinate system. ,in K Represents the camera intrinsic parameter matrix; The extrinsic parameter calibration is then transformed into minimizing the point-to-plane distance calculation, the expression of which is: ,in This represents the corresponding point cloud region in the image. For the extrinsic parameter matrices of the camera and radar; 1.3 Laser Point Cloud Motion Distortion Correction: Considering the point cloud scanning characteristics of lidar, the system, during the acquisition of a frame of point cloud data, utilizes the high-frequency angular velocity and acceleration provided by the IMU to compensate each point to its pose at the beginning of the frame; let the laser scanning start time be... The reference coordinate system is For laser points emitted / received at any time during the scan In the carrier coordinate system at that time, the result was obtained through IMU integration. ,time The coordinate transformation from the carrier coordinate system to the reference coordinate system W transforms the current point into the reference coordinate system: By performing transformation processing on all points in the current frame, the distortion-free point cloud frame can be obtained. .

3. The embodied robot SLAM method based on VLA according to claim 2, characterized in that: S2 includes, 2.1 VLA Active Visual Guidance: Based on environmental semantics and task instructions, a "semantically enhanced observation weight map" is generated to guide the selection and weighting of sensor data; Output semantic tokens using the VLA model and the corresponding three-dimensional coordinates Then define VLA semantic confidence. Laser and visual weights are respectively , And satisfy ; 2.2 Multi-sensor prediction: The LiDAR uses VLA semantic weights W_sem to filter dynamic noise points while preserving static semantic features, generating a filtered point cloud. The camera extracts semantically labeled ORB feature points after acquiring images. The inertial navigation system outputs the raw IMU measurements [ω_imu, a_imu], which are then pre-integrated to obtain the relative pose. ; 2.3 Sensor Residual Calculation and Optimization: Update the robot pose using data filtered by VLA weights, and calculate the residuals of each sensor. The residual from the lidar point to the plane is Where N is the number of points-to-faces matched. Let be the normal vector and distance parameters of the i-th plane; Visual luminance residual is ,in For pixel coordinates, For camera projection function, For three-dimensional coordinate points in the world coordinate system, and For translation and rotation transformations from the world coordinate system to the robot coordinate system, and The external parameters for lidar to the camera; Therefore, fusion optimization of pose is equivalent to minimizing residuals: ,in The predicted coordinates of the VLA semantic point in the current pose; The system iteratively updates and alternately processes the geometric residuals of the radar and the photometric residuals of the vision, ultimately obtaining the optimized front-end pose. .

4. The embodied robot SLAM method based on VLA according to claim 3, characterized in that: S3 specifically includes, 3.1 Semantic Feature Extraction: Inputting 3D point cloud P and synchronized RGB data, using the QDepth-VLA architecture, the output is a geometry-enhanced semantic token and relation description, with the following semantic-geometric joint representation: ,in For semantic tags, As the initial confidence level, The centroid of the point cloud in 3D spatial coordinates; The confidence level is corrected by optimizing the residual between the lidar depth and the VLA predicted depth: ,in Predict depth for VLA, For lidar depth, Experience value; 3.2 Architecture Layering: The S-Graphs architecture consists of four layers: geometric point cloud layer, semantic entity layer, logical relationship layer, and global consistency layer. in, Geometric point cloud layer: laser point cloud octree map The resolution is customizable and used as a priori map for positioning and navigation. Semantic entity layer: composed of nodes and edges; Node: Semantic entity extracted by VLA parsing; ; Edges: Geometric relationships between entities, generated through VLA (Variable Area Language) descriptions; The relationship definition rules are implemented through AABB bounding box intersection and union comparison; Logical relation layer: Composed of nodes and edges, it is abstracted based on the second semantic entity layer and a rule relation library is established. Node: A higher-level abstraction of semantic entities. side: Logical reasoning relationships based on rule base Semantic entity layer This indicates that the table and chair are adjacent; at the logical relationship level, it is... , indicating a table and chair set; Global Consistency Layer: Semantic Topology Graph ,node For key semantic targets, edges Constraints on relative distance / direction between targets; 3.3 VLA Semantic Completion and Optimization: Dynamic semantic filtering is performed by labeling dynamic targets using VLA temporal reasoning, removing them from the map; then semantic size prior constraints are applied, given the known categories. Target size Optimize semantic unit size ,in For the observation size, This is a weighting factor.

5. The embodied robot SLAM method based on VLA according to claim 4, characterized in that: S4 specifically includes, 4.1 Belief State Representation: First, the system maintains a neural approximation of the belief state vector to represent the robot's probabilistic understanding of the environmental layout; assuming that there exists in the visual scene... K A potential goal, a state of belief. Defined as target state The posterior probability distribution: ,in For historical observation sequence, For historical attention action sequences; 4.2 Active Attention Action Selection: Based on the principle of minimizing variational free energy, the next attention action is selected. : Simplified to maximizing the expected information gain: ,in Let be the entropy function, representing the uncertainty of the belief state; 4.3 Belief State Update: New Data Observed Then, the belief state is updated using Bayesian filtering: If a particle filter approximation is used, it can be represented as an update of particle weights: ； The system predicts the "motion lifetime" of environmental entities through the VLA model. For objects identified as "temporary dynamics", the system marks them as "transient factors" when building the map. The geometric contribution of these nodes is automatically ignored during global loop closure and factor graph optimization to ensure the long-term effectiveness of the map.

6. The embodied robot SLAM method based on VLA according to claim 5, characterized in that: S5 specifically includes, 5.1 Multimodal feature fusion: First, extract laser point cloud features, visual features and VLA semantic features. Generate fusion features of key frames based on pre-trained weight matrices. Determine and store key frames based on fusion feature attributes and inter-frame distance for use in subsequent loop closure detection. Laser point cloud features: Extract global features of the point cloud and use Scan Context, a circular histogram based on polar coordinates, to convert the 3D point cloud into a two-dimensional matrix. It divides the point cloud into circumferential and circumferential sectors in polar coordinates to form an N×M matrix, where each element represents the maximum height value of the point in the corresponding spatial bucket, thereby encoding the global geometric structure of the scene. Its core attribute is the height distribution matrix of the polar coordinate grid, and its similarity is measured by cosine distance or Hamming distance: in Given two scene Scan Context matrices, N x M dimensional. The trace of the matrix is ​​the sum of the elements on the main diagonal. It is the Frobenius norm; Visual Features: CLIP extracts global visual features and trains on massive image-text pairing datasets, bringing similar images and text closer together in the embedding space while distancing samples from different modalities. This enables CLIP to achieve zero-shot inference, meaning it can classify or understand any visual task without requiring fine-tuning for a specific task. CLIP maps images to 512-dimensional real-valued vectors through contrastive learning. Its core attribute is alignment with the language embedding space, with similarity measured by cosine distance. in, Let be the CLIP embedding vector of image I; VLA semantic features: Extract semantic graph structure features of VLA modalities, construct spatial logical relationships between targets through graph structure encoding, and the spatial modeling relationship is as follows: ,node For key semantic targets, edges Relative distance / direction constraints between targets; where node features are That is, category, center coordinates, size, and edge weights. ,in Indicate whether entities i and j are adjacent; After obtaining multimodal features and feature metrics, loop closure similarity scores are generated separately for each modality, and then a comprehensive decision is made through methods such as voting and weighted summation; the loop closure similarity scores are calculated based on a pre-trained weight matrix, and dynamic weight adjustments are made based on modality reliability; The reliability of loop closure similarity scores in combination with modal attributes: ,in For modality Similarity, laser (L), visual (V), VLA semantic (S); The weights are dynamic and adaptively adjusted based on modal reliability. , ,in For modality The standard deviation of feature similarity; 5.2 Loop Closure Detection and Geometric Verification: After obtaining the fused feature similarity between keyframes, determine whether their similarity meets the loop closure threshold. ,in If the threshold is set, then the loop closure requirement is met, and this frame is included in the loop closure comparison frame range. After obtaining the loop closure candidate frames, the system performs 6-DoF geometric alignment. Based on the system point cloud features, it uses the ICP fine registration algorithm to perform point cloud registration and obtain the inter-frame transformation matrix. This allows for the acquisition of more accurate inter-frame relationships, and the establishment of closure factors is incorporated into factor graph optimization.

7. The embodied robot SLAM method based on VLA according to claim 6, characterized in that: S6 specifically includes, 6.1 VLA-driven dynamic reconstruction of factor graphs First, based on the input RGB image sequence and instruction set, semantic keyframes are filtered, and their scores are calculated. Only retain The frames are used as nodes in the factor graph to reduce redundancy, where, The set of semantic categories detected by VLA; Entropy for category diversity; For the semantic topology subgraph of the current frame, For global semantic graph; The coverage of the subgraph on the global graph; Secondly, cross-modal factor connectivity prediction is performed. Latent factor connectivity is predicted through semantic relation reasoning, using predicted nodes, and the formula for the predicted connectivity probability is as follows: ,in, For the factor connection between frame i and frame j, 1 indicates that a connection exists; The semantic sequence extracted from VLA; For the semantic embedding of the CLIP model, For feature splicing, Activated for sigmoid; 6.2 Scene difficulty score output via VLA The visual factor noise is dynamically modeled, and the noise formula is: ,in, Given the basic covariance matrix, r is the difficulty coefficient. Difficulty level; Based on the above formula, adaptive noise for different frames is calculated and input into the factor graph nodes to ensure the establishment of more accurate factor graph relationships.

8. The embodied robot SLAM method based on VLA according to claim 7, characterized in that: S6 also includes, VLA generates dynamic constraints through temporal semantic reasoning, supports online factor graph optimization, and performs dynamic target tracking based on temporal attention. For dynamic target o, a motion consistency factor is added to model the motion of the dynamic target as a "soft constraint" to avoid it interfering with robot pose estimation. in For historical speed, Predict target velocity for VLA; Based on the above factor relationships, a nonlinear optimization objective function is constructed to jointly optimize the accumulated error: in For the set of loopback frames; By using a Bayesian tree structure, linearization and reordering are performed only on subgraphs affected by new cyclic factors to perform incremental global optimization while maintaining global consistency. Based on the Levenberg-Marquardt method, iterative updates are performed. Then, all factor relationships are updated, where J is the Jacobian matrix and r is the residual vector; The globally optimized trajectory is used to correct the global point cloud map and simultaneously update the semantic positions and relationships of entities in the scene map. The system maintains a multi-class confusion matrix and continuously improves the confidence of object labels based on the number of observations, ultimately generating a "cognitive map" with high geometric accuracy and deep semantic consistency for use in basic navigation and high-level task navigation.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the processor performs the steps of the method as described in any one of claims 1 to 6.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, the processor performs the steps of the method as described in any one of claims 1 to 6.

Images