Multi-modal fused high-precision semantic map construction and centimeter-level positioning navigation method

Abstract

The invention provides a multi-modal fused high-precision semantic map construction and centimeter-level positioning navigation method, which comprises the following steps of: firstly, constructing a'structure-texture-semantics' integrated high-precision map by fusing multi-source data, and then constructing a multi-modal fused high-precision semantic map based on a multi-technology fused centimeter-level positioning technology. The method comprises the following steps: realizing centimeter-level positioning in a complex health-care scene through'high-frequency odometer + low-frequency absolute observation 'double-level data acquisition and in combination with a multi-positioning fusion algorithm, and finally establishing an end-to-end navigation large model: taking multi-source sensing data as input, outputting an executable local path through'coding-fusion-generation' end-to-end process, and finally establishing an end-to-end navigation model; the core logic is combined with a vision-language-action fusion thought of TrackVLA to establish an end-to-end navigation large model. According to the method, a multi-modal semantic map is constructed, a multi-technology positioning scheme is fused, real-time path re-planning and personnel behavior intention prediction are realized, and the method adapts to layouts and people flow rules of different health-care areas.

Description

Technical Field

[0001] This invention relates to the field of navigation technology, specifically a method for constructing high-precision semantic maps and centimeter-level positioning and navigation through multimodal fusion. Background Technology

[0002] In current healthcare and hospital settings, autonomous navigation robots face multiple core challenges: 1. Complexity of environment and interaction: The dynamic and random movement of people in hospital corridors and the rapid movement of people in emergency scenarios require interaction with infrastructure such as semi-automatic doors and elevators. In addition, the layout and flow rules of different areas (ICU and general wards) vary greatly. 2. Deficiencies of existing solutions: Traditional modular navigation solutions (such as positioning based on multi-sensor fusion and pre-planned paths) have problems such as poor module coupling, difficulty in predicting human behavior, and the inability to implement "zero-collision" safety protection. 3. Insufficient human-machine collaboration: Robots can easily interfere with the work of medical staff / patients and lack path planning and intuitive interaction mechanisms that conform to interpersonal etiquette. Summary of the Invention

[0003] To address the problems of existing technologies, this invention provides a method for constructing high-precision semantic maps and centimeter-level positioning and navigation using multimodal fusion. It constructs multimodal semantic maps, integrates multiple positioning technologies, and enables real-time path replanning and prediction of human behavior intentions, adapting to the layout and pedestrian flow rules of different health and wellness areas.

[0004] This invention provides a method for constructing high-precision semantic maps and achieving centimeter-level positioning and navigation through multimodal fusion, comprising the following steps: 1) By fusing multi-source data, a high-precision map is constructed that integrates "structure, texture, and semantics." The specific process is as follows: 1.1) Multi-source data acquisition and preprocessing: Acquire hospital building information model (BIM) data, laser point cloud data, and visual data, and preprocess the data; 1.2) Data registration and structural map construction: Distortion correction is performed on the collected laser point cloud and visual images. The laser point cloud is registered with the geometric framework of the BIM drawings and fused to generate a 3D structured map. The difference in echo intensity of the laser point cloud is used to help distinguish facility types. 1.3) Texture map construction: The constructed map is processed using texture mapping technology and dynamic obstacle culling technology, and then fused to generate a 3D texture map; 1.4) Semantic information annotation and semantic map construction: Identify semantic elements in visual images, associate semantic labels with corresponding point cloud regions, and integrate semantic labels and region rules to generate a semantic map; 1.5) By fusing 3D texture maps and semantic maps, a 3D semantic map containing geometric structure, visual texture, and semantic rules is obtained; 2) Based on centimeter-level positioning technology using multi-technology fusion, centimeter-level positioning in complex health and wellness scenarios is achieved through a two-level data acquisition process of "high-frequency odometry + low-frequency absolute observation" combined with multi-positioning fusion algorithms. The specific process is as follows: 2.1) Multi-source positioning data is acquired at different frequencies through the data acquisition layer, including relative definition through high-frequency odometry and global absolute positioning through low-frequency odometry; 2.2) The data fusion layer optimizes and integrates multiple positioning information, and outputs global pose information with centimeter-level accuracy through Kalman state estimation and constrained pose optimization; 3) Establish an end-to-end navigation model: Using multi-source perception data as input, an executable local path is output through an end-to-end process of "encoding-fusion-generation". The core logic combines the vision-language-action fusion approach of TrackVLA to establish an end-to-end navigation model. The specific process is as follows: 3.1) Multi-source data acquisition and classification input are performed at the input layer. The model receives two types of input data, covering environmental perception and task constraints. 3.2) Feature extraction and unified encoding of multimodal data are performed through an encoding layer; 3.3) Path generation based on pre-trained models is carried out through fusion and generation layers. An end-to-end approach of "multimodal perception-semantic understanding-action generation" is adopted. Through cross-modal fusion of vision and navigation commands, combined with the generation capability of diffusion Transformer, integrated navigation decision-making of "perception-decision-path output" is achieved.

[0005] Further improvements are made, and the data acquisition process described in step 1.1) is as follows: Hospital Building Information Model (BIM) Data Acquisition: By inputting BIM drawings, the hospital's Building Information Model (BIM) data is obtained, and the building structure, including the locations of walls, doors, and elevators, is extracted. The area is divided into different wards, which serve as the basic framework of the map. Laser point cloud acquisition: High-precision three-dimensional point cloud data is obtained by scanning the environment with lidar, and the echo intensity of the point cloud is recorded simultaneously according to different materials; Visual data acquisition: Environmental images are acquired using an RGB-D camera, and texture and depth information of the images are obtained simultaneously.

[0006] Further improvements include: the texture mapping technique described in step 1.3) maps the texture information of the visual image onto the geometric surface of the 3D structured map to restore the visual features of the environment; the dynamic obstacle removal is achieved by dynamically detecting the visual image, removing dynamic objects in the point cloud or image, and retaining the texture information of the static environment.

[0007] Further improvements are made, and the specific methods for semantic information annotation and semantic map construction in step 1.4) are as follows: Point cloud segmentation: Segmenting 3D point clouds according to spatial regions and object types; Semantic recognition: Identify semantic elements through object detection in visual images and associate semantic labels with corresponding point cloud regions; Semantic graph construction: Integrate semantic tags and region rules to generate a semantic graph.

[0008] Further improvements are made to step 2.1), which involves acquiring multi-source positioning data at different frequencies through the data acquisition layer. The specific process is as follows: 2.11) Real-time relative positioning via high-frequency odometry: Real-time point cloud data is collected using 3D LiDAR, and the current point cloud is compared with the pre-constructed 3D semantic map point cloud using a 3D point cloud matching algorithm to calculate the robot's relative pose change and output high-frequency relative positioning data; RGB images and depth information of the environment are collected by an RGBD camera, 3D visual features are extracted, and matched with visual texture features in the semantic map to help correct the cumulative error of the laser odometry. 2.22) Real-time global absolute positioning via low-frequency and high-frequency odometers: Using UWB base stations deployed in the environment, the signal strength of the robot's UWB tags is collected and matched with a pre-built UWB fingerprint database to output low-frequency global absolute position information; signal strength is collected and Bluetooth beacon detection is performed via Bluetooth beacons in the environment to assist in UWB fingerprint positioning correction and supplement the coverage of global positioning.

[0009] Further improvements are made in step 2.2). The Kalman state estimation is based on the relative pose data of laser and vision obtained by the high-frequency odometry. The real-time positioning data is filtered and smoothed by the Kalman filtering algorithm to suppress positioning fluctuations caused by high-frequency noise. The constrained pose optimization uses the global position output by the low-frequency absolute observation as a constraint condition and fuses it with the pose data after Kalman filtering to eliminate the cumulative drift of the high-frequency odometry.

[0010] Further improvements are made, and the two types of input data in step 3.1) include visual perception data and navigation constraint data; the visual perception data is the relative data of a single or multiple frames of RGBD camera, including RGB images and depth information, reflecting the visual characteristics and spatial distance of the real-time environment; the navigation constraint data includes global path semantic information, global path points and the current positioning point.

[0011] Further improvements are made to the feature extraction and encoding process described in step 3.2), which is as follows: 3.21) Use a visual encoder to extract features from single or multiple frames of RGBD data, and convert image texture and depth information into visual feature vectors; 3.22) Using an instruction encoder: Encode the global path semantic information, global path points, and current location point, and transform text rules and spatial coordinates into task constraint feature vectors; 3.23) Utilize a cross-modal projector to perform dimensional alignment and cross-modal fusion of visual and instruction features, outputting a unified multimodal feature representation that matches the "visual-language" feature fusion logic.

[0012] Further improvements, in step 3.3), the specific process of path generation based on the pre-trained model through the fusion and generation layer is as follows: 3.31) Pre-trained visual VLM model: Load a pre-trained visual-language multimodal VLM model and utilize its ability to understand "environmental visual features + task semantic constraints" to complete high-level semantic fusion of multimodal features; 3.32) DiffusionTransformer model: Based on the fused multimodal features, path generation is performed through the DiffusionTransformer model: Based on the current environment dynamics and combined with global path constraints, local path candidates that meet safety and rules are generated step by step; 3.33) Through iterative optimization of the diffusion process, an executable local path is output, including the movement direction, speed, and avoidance strategy.

[0013] The beneficial effects of this invention are as follows: 1. High-precision environmental perception: Construct a multimodal semantic map to accurately identify dynamic obstacles, facility status, and scene rules; 2. Centimeter-level positioning: Integrates multiple positioning technologies to ensure positioning accuracy in complex environments; 3. Dynamic safety navigation: Enables real-time path replanning and prediction of personnel behavior and intentions, meeting the hospital's "zero collision" safety requirements; 4. Scene Adaptability: Adapts to the layout and pedestrian flow rules of different health and wellness areas, improving the acceptance of human-machine collaboration. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the navigation process of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] One specific embodiment of the present invention is as follows: Figure 1 As shown, the technologies employed mainly include multimodal high-precision semantic map construction technology, centimeter-level positioning technology with multi-technology fusion, and an end-to-end navigation brain model.

[0018] I. Specific implementation details of high-precision semantic map construction technology based on multimodality.

[0019] This technology integrates multi-source data (BIM, laser, and visual) to achieve high-precision map construction that combines "structure, texture, and semantics" in a step-by-step manner. The core process is as follows: Step 1: Multi-source data acquisition and preprocessing BIM drawing input: Obtain hospital building information model (BIM) data, extract static geometric information such as building structure (walls, doors, elevator locations) and zoning (ICU / general wards) as the basic framework of the map; Laser point cloud acquisition: High-precision 3D point cloud data is obtained by scanning the environment with lidar, and the echo intensity of the point cloud is recorded simultaneously (to distinguish different materials, such as metal doors and ceramic tile floors). Visual data acquisition: Environmental images are acquired using an RGB-D camera, and texture and depth information of the images are obtained simultaneously.

[0020] Step 2: Data Registration and Structure Map Construction Distortion calibration: Corrects distortion in laser point clouds and visual images to eliminate sensor hardware errors; Point cloud registration technology: Registering laser point clouds with the geometric framework of BIM drawings (such as matching wall outlines based on ICP algorithm) and merging them to generate a 3D structured map (containing accurate spatial geometry). Echo intensity analysis: The difference in echo intensity of laser point clouds is used to help distinguish facility types (e.g., the echo intensity of metal elevator doors is higher than that of wooden doors).

[0021] Step 3: Texture Map Construction Texture mapping technology: Maps the texture information of visual images (such as wall markings and ground patterns) onto the geometric surface of a 3D structured map to restore the visual features of the environment; Dynamic obstacle removal: By detecting dynamic objects (such as pedestrians) in point clouds / images through visual image dynamic detection (such as frame difference method), dynamic objects (such as pedestrians) are removed, while the texture information of the static environment is preserved; The resulting 3D texture map (with geometric structure and realistic visual texture) is generated by fusion.

[0022] Step 4: Semantic Information Annotation and Semantic Map Construction Point cloud segmentation: Segmenting 3D point clouds according to spatial regions and object types (such as "elevator area" and "medicine window"); Semantic recognition: Identify semantic elements (such as "ICU sign" and "No Entry sign") through object detection in visual images (such as the YOLO algorithm) and associate semantic labels with the corresponding point cloud regions; Semantic graph construction: Integrate semantic tags with region rules (such as "ICU area speed limit 0.5m / s") to generate a semantic graph; Finally, the 3D texture map and semantic map are merged to obtain a 3D semantic map (an integrated map containing geometric structure, visual texture, and semantic rules).

[0023] II. Specific implementation details of centimeter-level positioning technology based on multi-technology fusion.

[0024] This technology achieves centimeter-level positioning in complex health and wellness scenarios through a two-tiered data acquisition process combining high-frequency odometer readings and low-frequency absolute observations, along with a multi-positioning fusion algorithm. The core process is as follows: Step 1: Data Acquisition Layer: Acquisition of multi-source positioning data at different frequencies; 1. High-frequency odometer (real-time relative positioning): 3D LiDAR module: Real-time point cloud data is collected using 3D LiDAR. The current point cloud is compared with the pre-constructed 3D semantic map point cloud using 3D point cloud matching algorithms (such as ICP / LOAM) to calculate the robot's relative pose change (position, attitude) and output high-frequency (above 10Hz) relative positioning data. RGBD camera module: The RGB image and depth information of the environment are acquired by an RGBD camera, and 3D visual features (such as ORB feature points) are extracted and matched with visual texture features in the semantic map to help correct the cumulative error of the laser odometry.

[0025] 2. Low-frequency absolute observation (global absolute positioning): UWB fingerprint module: By utilizing UWB base stations deployed in the environment, the signal strength of the robot's UWB tags is collected and matched with a pre-built UWB fingerprint database (which collects UWB signal characteristics at various locations in the environment in advance) to output low-frequency (1-2Hz) global absolute position information. Bluetooth beacon module: By collecting signal strength and detecting Bluetooth beacons in the environment, UWB fingerprint positioning correction is assisted, supplementing the coverage of global positioning.

[0026] Step 2: Data Fusion Layer: Optimization and integration of multiple location information; 1. Kalman State Estimation Based on the relative pose data of high-frequency odometry (laser + vision), the real-time positioning data is filtered and smoothed by Kalman filtering algorithm to suppress positioning fluctuations caused by high-frequency noise.

[0027] 2. Constrained Pose Optimization When the low-frequency absolute observation (UWB + Bluetooth) outputs the global position, it is used as a constraint and fused with the pose data after Kalman filtering (such as graph-based optimization algorithm) to eliminate the cumulative drift of the high-frequency odometry and finally output global pose information with centimeter-level accuracy.

[0028] Summary of core logic By employing a hierarchical strategy of "high-frequency odometer to ensure real-time performance and low-frequency absolute observation to correct drift," combined with a multi-source data fusion algorithm, continuous and stable centimeter-level positioning is achieved in complex health and wellness environments.

[0029] III. Specific implementation details of the "end-to-end navigation big model".

[0030] This model takes multi-source sensor data as input and outputs an executable local path through an end-to-end process of "encoding-fusion-generation". The core logic combines TrackVLA's vision-language-action fusion approach, as detailed below: Step 1: Input Layer: Multi-source Data Acquisition and Classification Input The model receives two types of input data, covering environmental perception and task constraints: Visual perception data: The relative data of a single-frame / multi-frame RGBD camera (including RGB images and depth information) reflects the visual characteristics and spatial distance of the real-time environment; Navigation constraint data: Global path semantic information (such as "ICU area speed limit" and "elevator waiting rules"); Global waypoint (the global path from the start point to the end point of the task); Current location (robot's real-time pose).

[0031] Step 2: Encoding Layer: Feature Encoding of Multimodal Data Following the multimodal coding approach of TrackVLA, feature extraction and unified encoding are performed on the input data: Visual encoder: Feature extraction is performed on single-frame / multi-frame RGBD data (e.g., using a CNN+Transformer structure) to convert image texture and depth information into visual feature vectors; Command encoder: The global path semantic information, global path points, and current location point are encoded (using a language model + spatial encoding structure), and the text rules and spatial coordinates are transformed into task constraint feature vectors. Cross-modal projector: The visual and instruction features are dimensionally aligned and fused across modalities to output a unified multimodal feature representation (matching TrackVLA’s “visual-language” feature fusion logic).

[0032] Step 3: Fusion and Generation Layer: Path Generation Based on Pre-trained Model Pre-trained visual VLM model: Load the pre-trained visual-language multimodal model (VLM) and leverage its ability to understand "environmental visual features + task semantic constraints" to complete high-level semantic fusion of multimodal features (corresponding to the multimodal understanding module of TrackVLA). Diffusion Transformer: Using the fused multimodal features as conditions, path generation is performed through the Diffusion Transformer model: Based on the current environmental dynamics (such as the status of obstacles and facilities) and combined with global path constraints, local path candidates that meet safety and rules are gradually generated. Through iterative optimization of the diffusion process, an executable local path (including action instructions such as movement direction, speed, and avoidance strategy) is output.

[0033] Summary of core logic This model references TrackVLA's end-to-end approach of "multimodal perception - semantic understanding - action generation". By fusing vision and navigation commands across modalities and combining the generative capabilities of the Diffusion Transformer, it achieves integrated navigation decision-making of "perception - decision-making - path output".

[0034] The hardware solution and implementation details of this invention are as follows: I. Implementation details of high-precision semantic map construction technology based on multimodality.

[0035] (I) Hardware Solution BIM data processing hardware: Equipped with a graphics workstation, an Intel Core i9-14900K CPU (24 cores, 32 threads, base frequency 3.2GHz, turbo frequency 5.8GHz), 64GB DDR5 5600MHz memory, and an NVIDIA RTX 4090 24GB graphics card, for efficient parsing and processing of complex geometric data from BIM drawings.

[0036] LiDAR equipment: The Velodyne VLP-16 LiDAR is used, with 16 laser beams, a ranging range of 0.3-100 meters, a ranging accuracy of ±3 cm, an adjustable scanning frequency of 5-20 Hz, and an echo intensity resolution of 16 bits, which meets the requirements for high-precision point cloud acquisition and material differentiation.

[0037] Visual acquisition equipment: Intel RealSense D455 depth camera, RGB resolution 1920×1080, depth resolution 1280×720, depth ranging range 0.1-10 meters, depth accuracy ±2%, frame rate 30fps, simultaneously acquiring texture and depth information.

[0038] Data storage and transmission equipment: Equipped with a 2TB NVMe solid-state drive for temporary storage of collected point cloud and image data, and equipped with a gigabit Ethernet module to support high-speed data transmission to workstations for processing.

[0039] (II) Implementation Details Multi-source data acquisition and preprocessing parameters: BIM data is imported in IFC4.0 format, wall thickness extraction error is ≤5 mm, and area division boundary accuracy is ≤10 cm; when scanning with LiDAR, the scanning frequency is set to 10 Hz and the angular resolution is 0.1°×0.1° to ensure uniform point cloud density; the visual camera and LiDAR are triggered synchronously through hardware, and the time synchronization error is ≤1 millisecond.

[0040] Data registration and structural map construction parameters: distortion calibration adopts Zhang Zhengyou's checkerboard calibration method, camera intrinsic reprojection error ≤ 0.5 pixels, LiDAR distortion correction error ≤ 2 cm; point cloud registration uses an improved ICP algorithm, the iterative convergence threshold is set to 0.001 meters, and the positional error between the registered point cloud and the BIM frame is ≤ 3 cm; the echo intensity differentiation threshold is set to 20%, the echo intensity of metal materials is ≥ 80%, and the echo intensity of wood materials is ≤ 60%.

[0041] Texture map construction parameters: The texture mapping adopts the perspective projection method, the texture sampling interval is 0.5 cm, and the texture distortion rate after mapping is ≤5%; the dynamic obstacle removal adopts the three-frame difference method, the inter-frame pixel change threshold is set to 15%, and the minimum detection area is 100 pixels to ensure accurate removal of dynamic objects such as pedestrians.

[0042] Semantic information annotation and semantic map construction parameters: Point cloud segmentation adopts a region growing algorithm with a growth threshold of 0.05 meters and a minimum cluster number of 500 points; semantic recognition uses the YOLOv8 model with a confidence threshold of 0.7 and an IOU threshold of 0.5, achieving an accuracy of ≥95% for targets such as ICU signs and no-entry signs; the semantic map is stored in RDF format, and the correlation error between region rules and point cloud regions is ≤5 cm.

[0043] II. Implementation details of centimeter-level positioning technology based on multi-technology fusion.

[0044] (I) Hardware Solution High-frequency odometry hardware: It uses the Velodyne VLP-16 LiDAR as its core, paired with an Intel RealSense D455 depth camera, mounted on top of the robot. The distance between the two is fixed at 30 cm, and time synchronization is achieved through a synchronization board. The robot has a built-in industrial control computer with an AMD Ryzen 7 7840U CPU and 32GB of memory, used for real-time processing of odometry data.

[0045] Low-frequency absolute observation hardware: The UWB equipment uses Decawave DW1000 modules, with a base station deployment spacing of ≤15 meters, ranging accuracy of ±10 cm, and a signal frequency of 3.5-6.5 GHz; the Bluetooth beacon uses iBeacon modules, with adjustable transmit power of 0-4 dBm, a signal coverage radius of 10-50 meters, and a deployment density of 1 beacon per 50 square meters to assist in UWB positioning and blind spot filling.

[0046] Location fusion hardware: Equipped with an FPGA chip (Xilinx XC7Z020) to accelerate the operation of Kalman filtering and graph optimization algorithms, ensuring the real-time performance of data fusion with a latency of ≤10 milliseconds.

[0047] (II) Implementation Details Data acquisition layer parameters: For high-frequency odometry, the laser point cloud matching uses the LOAM algorithm at a matching frequency of 15Hz, with a relative pose calculation error ≤2 cm. Visual feature extraction uses the ORB algorithm, extracting ≥1000 feature points per frame, achieving a feature matching accuracy ≥90%, used to correct the cumulative error of the laser odometry, corrected every 5 frames. For low-frequency absolute observation, the UWB fingerprint database is collected at 50 cm intervals, with 100 signal strength measurements taken at each location and the average value calculated; the fingerprint matching similarity threshold is 0.8. The Bluetooth beacon signal strength threshold is set to -70dBm; signals below this threshold are not used for positioning correction.

[0048] Data fusion layer parameters: The noise covariance matrix of the Kalman filtering process is set to diag ([0.01, 0.01, 0.01, 0.001, 0.001, 0.001]), and the measurement noise covariance matrix is ​​dynamically adjusted according to the sensor accuracy. The fluctuation range of the filtered positioning data is ≤1 cm. The constrained pose optimization uses the g2o graph optimization library, with ≤50 iterations and a convergence threshold of 0.005 meters. The global pose accuracy after fusion is ≤5 cm, which meets the centimeter-level positioning requirements.

[0049] III. Implementation details of the "end-to-end navigation big model".

[0050] (I) Hardware Solution Model training hardware: Utilizes an NVIDIA DGX A100 server, equipped with 8 NVIDIA A100 80GB GPUs, 1TB of system memory and 16TB of NVMe storage, supporting distributed training to meet the training needs of large-scale pre-trained models.

[0051] Model inference hardware: The robot is equipped with an NVIDIA Jetson AGX Orin developer kit, 64GB of memory, and 2048-core CUDA cores, which supports real-time model inference with an inference latency of ≤200 milliseconds, ensuring the real-time performance of navigation decisions.

[0052] Data input hardware: The robot uses the Intel RealSense D455 depth camera to collect visual perception data, obtains real-time pose data through the robot's built-in positioning module, and imports navigation constraint data into the model through the robot's control system.

[0053] (II) Implementation Details Input layer parameters: Visual perception data uses 3 consecutive frames of RGBD images as input, with a frame interval of 33 milliseconds to ensure capture of dynamic environmental changes; navigation constraint data uses global pathpoints spaced 1 meter apart, and semantic information is input in JSON format, including more than 20 constraint types such as area speed limits and passage rules, with the current positioning point data update frequency of 10Hz.

[0054] Encoding layer parameters: The visual encoder adopts a ResNet50+Transformer structure, with the CNN part outputting a 7×7 feature map. The Transformer encoder has 6 layers and 8 multi-head attention heads, resulting in a visual feature vector dimension of 512. The instruction encoder adopts a BERT-based language model + 3D convolutional spatial encoding structure, with a text feature dimension of 768 and a spatial coordinate encoding dimension of 256, resulting in a fused instruction feature vector dimension of 512. The cross-modal projector maps visual features and instruction features to the same 512-dimensional space through a fully connected layer, with a projection error ≤0.01.

[0055] Fusion and generation layer parameters: The pre-trained visual VLM model uses BLIP-2, with the bottom layer parameters frozen and only the top fusion layer fine-tuned. The learning rate is 1e-5 and the batch size is 16. The Diffusion Transformer model has 12 layers, 16 multi-head attention heads, 50 diffusion steps, a path generation time resolution of 0.1 seconds, a path point spacing of 5 cm, and the generated local path meets the requirements of obstacle avoidance distance ≥30 cm and speed error ≤0.05 m / s.

[0056] (III) Details of Model Training and Deployment Training data: 100,000 frames of RGBD images were collected from health and wellness scenarios (hospitals, nursing homes), including environmental data of different time periods, different weather conditions, and different personnel flow states; a dataset of 5,000 navigation tasks was constructed, covering various start-end combinations and semantic constraints.

[0057] Training process: First, the visual encoder and instruction encoder are pre-trained, then the cross-modal fusion layer is fine-tuned, and finally the Diffusion Transformer generation layer is trained. The total training time is about 15 days and is completed on the DGX A100 server.

[0058] Model deployment: The trained model is quantized into FP16 format and deployed on Jetson AGX Orin. Inference is accelerated using TensorRT to ensure real-time performance and accuracy. The model achieves a path planning success rate of ≥98% in real-world scenarios.

[0059] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, for the device embodiments, the above descriptions are merely preferred embodiments of the present invention. Since they are fundamentally similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention, without departing from the principle of the present invention, should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for constructing high-precision semantic maps and centimeter-level positioning and navigation through multimodal fusion, characterized in that... Includes the following steps: 1) By fusing multi-source data, a high-precision map is constructed that integrates "structure, texture, and semantics." The specific process is as follows: 1.1) Multi-source data acquisition and preprocessing: Acquire hospital building information model (BIM) data, laser point cloud data, and visual data, and preprocess the data; 1.2) Data registration and structural map construction: Distortion correction is performed on the collected laser point cloud and visual images. The laser point cloud is registered with the geometric framework of the BIM drawings and fused to generate a 3D structured map. The difference in echo intensity of the laser point cloud is used to help distinguish facility types. 1.3) Texture map construction: The constructed map is processed using texture mapping technology and dynamic obstacle culling technology, and then fused to generate a 3D texture map; 1.4) Semantic information annotation and semantic map construction: Identify semantic elements in visual images, associate semantic labels with corresponding point cloud regions, and integrate semantic labels and region rules to generate a semantic map; 1.5) By fusing 3D texture maps and semantic maps, a 3D semantic map containing geometric structure, visual texture, and semantic rules is obtained; 2) Centimeter-level positioning technology based on multi-technology fusion: Through dual-level data acquisition of "high-frequency odometry + low-frequency absolute observation" and combined with multi-positioning fusion algorithms, centimeter-level positioning is achieved in complex health and wellness scenarios. The specific process is as follows: 2.1) Multi-source positioning data is acquired at different frequencies through the data acquisition layer, including relative definition through high-frequency odometry and global absolute positioning through low-frequency odometry; 2.2) The data fusion layer optimizes and integrates multiple positioning information, and outputs global pose information with centimeter-level accuracy through Kalman state estimation and constrained pose optimization; 3) Establish an end-to-end navigation model: Using multi-source perception data as input, an executable local path is output through an end-to-end process of "encoding-fusion-generation". The core logic combines the vision-language-action fusion approach of TrackVLA to establish an end-to-end navigation model. The specific process is as follows: 3.1) Multi-source data acquisition and classification input are performed at the input layer. The model receives two types of input data, covering environmental perception and task constraints. 3.2) Feature extraction and unified encoding of multimodal data are performed through an encoding layer; 3.3) Path generation based on pre-trained models is carried out through fusion and generation layers. An end-to-end approach of "multimodal perception-semantic understanding-action generation" is adopted. Through cross-modal fusion of vision and navigation commands, combined with the generation capability of diffusion Transformer, integrated navigation decision-making of "perception-decision-path output" is achieved.

2. The high-precision semantic map construction and centimeter-level positioning and navigation method based on multimodal fusion according to claim 1, characterized in that: The data acquisition process described in step 1.1) is as follows: Hospital Building Information Model (BIM) Data Acquisition: By inputting BIM drawings, the hospital's Building Information Model (BIM) data is obtained, and the building structure, including the locations of walls, doors, and elevators, is extracted. The area is divided into different wards, which serve as the basic framework of the map. Laser point cloud acquisition: High-precision three-dimensional point cloud data is obtained by scanning the environment with lidar, and the echo intensity of the point cloud is recorded simultaneously according to different materials; Visual data acquisition: Environmental images are acquired using an RGB-D camera, and texture and depth information of the images are obtained simultaneously.

3. The high-precision semantic map construction and centimeter-level positioning and navigation method based on multimodal fusion according to claim 1, characterized in that: Step 1.3) The texture mapping technology described above maps the texture information of a visual image onto the geometric surface of a 3D structured map to restore the visual features of the environment; The dynamic obstacle removal is achieved by dynamically detecting visual images, removing dynamic objects from point clouds or images, and preserving the texture information of the static environment.

4. The high-precision semantic map construction and centimeter-level positioning and navigation method based on multimodal fusion according to claim 1, characterized in that: The specific methods for semantic information annotation and semantic map construction in step 1.4) are as follows: Point cloud segmentation: Segmenting 3D point clouds according to spatial regions and object types; Semantic recognition: Identify semantic elements through object detection in visual images and associate semantic labels with corresponding point cloud regions; Semantic graph construction: Integrate semantic tags and region rules to generate a semantic graph.

5. The high-precision semantic map construction and centimeter-level positioning and navigation method based on multimodal fusion according to claim 1, characterized in that: Step 2.1) describes the acquisition of multi-source positioning data by frequency division through the data acquisition layer. The specific process is as follows: 2.11) Real-time relative positioning via high-frequency odometry: Real-time point cloud data is collected using 3D LiDAR, and the current point cloud is compared with the pre-constructed 3D semantic map point cloud using a 3D point cloud matching algorithm to calculate the robot's relative pose change and output high-frequency relative positioning data; RGB images and depth information of the environment are collected by an RGBD camera, 3D visual features are extracted, and matched with visual texture features in the semantic map to help correct the cumulative error of the laser odometry. 2.22) Real-time global absolute positioning via low-frequency and high-frequency odometers: Using UWB base stations deployed in the environment, the signal strength of the robot's UWB tags is collected and matched with a pre-built UWB fingerprint database to output low-frequency global absolute position information; signal strength is collected and Bluetooth beacon detection is performed via Bluetooth beacons in the environment to assist in UWB fingerprint positioning correction and supplement the coverage of global positioning.

6. The high-precision semantic map construction and centimeter-level positioning and navigation method based on multimodal fusion according to claim 1, characterized in that: Step 2.2) The Kalman state estimation is based on the relative pose data of laser and vision obtained by high-frequency odometry. The real-time positioning data is filtered and smoothed by Kalman filtering algorithm to suppress positioning fluctuations caused by high-frequency noise. The constrained pose optimization takes the global position output of low-frequency absolute observation as a constraint condition and fuses it with the pose data after Kalman filtering to eliminate the cumulative drift of high-frequency odometry.

7. The high-precision semantic map construction and centimeter-level positioning and navigation method based on multimodal fusion according to claim 1, characterized in that: Step 3.1) The two types of input data include visual perception data and navigation constraint data; the visual perception data is the relative data of a single or multiple frames of RGBD camera, including RGB images and depth information, reflecting the visual characteristics and spatial distance of the real-time environment; the navigation constraint data includes global path semantic information, global path points and the current positioning point.

8. The high-precision semantic map construction and centimeter-level positioning and navigation method based on multimodal fusion according to claim 1, characterized in that: The feature extraction and encoding process described in step 3.2) is as follows: 3.21) Use a visual encoder to extract features from single or multiple frames of RGBD data, and convert image texture and depth information into visual feature vectors; 3.22) Using an instruction encoder: Encode the global path semantic information, global path points, and current location point, and transform text rules and spatial coordinates into task constraint feature vectors; 3.23) Utilize a cross-modal projector to perform dimensional alignment and cross-modal fusion of visual and instruction features, outputting a unified multimodal feature representation that matches the "visual-language" feature fusion logic.

9. The high-precision semantic map construction and centimeter-level positioning and navigation method based on multimodal fusion according to claim 1, characterized in that: Step 3.3) describes the specific process of path generation based on the pre-trained model through the fusion and generation layer, as follows: 3.31) Pre-trained visual VLM model: Load a pre-trained visual-language multimodal VLM model and utilize its ability to understand "environmental visual features + task semantic constraints" to complete high-level semantic fusion of multimodal features; 3.32) DiffusionTransformer model: Based on the fused multimodal features, path generation is performed through the DiffusionTransformer model: Based on the current environment dynamics and combined with global path constraints, local path candidates that meet safety and rules are generated step by step; 3.33) Through iterative optimization of the diffusion process, an executable local path is output, including the movement direction, speed, and avoidance strategy.

Images