Robot indoor and outdoor navigation switching method and system based on large language model

Abstract

The invention discloses a robot indoor and outdoor navigation switching method and system based on a large language model, and the method comprises the steps: obtaining multi-dimensional data of an environment where a robot is located through employing a multi-mode sensor, and the multi-dimensional data comprise a visual image, a laser radar point cloud, a GPS signal, and IMU data; the preprocessed image is input into a visual encoder, and key visual features are extracted. And the big language model performs reasoning by using the characteristics, and outputs a scene judgment result in combination with an internal scene knowledge base. In order to enhance the accuracy of scene judgment, the change trend of multi-frame image features is analyzed based on a time sliding window. An extended Kalman filter is adopted in an indoor and outdoor transition area to carry out weighted fusion on multi-mode sensor data, a dual-mode parallel operation mechanism is adopted during navigation mode switching, and SLAM and GPS navigation modes are activated at the same time in the indoor and outdoor transition area. And by dynamically adjusting the weights of the SLAM and the GPS, gradually transitioning to a target navigation mode at a data fusion level, so that smooth switching is realized.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method and system for indoor and outdoor navigation switching of a robot based on a large language model. Background Art

[0002] With the rapid development of artificial intelligence and robotics, robots are increasingly used in industries, services, logistics, etc. Robot navigation technology is a key technology to achieve autonomous mobility of robots, especially to achieve seamless switching between indoor and outdoor navigation in complex environments, which has become a hot topic and difficulty in current research.

[0003] In existing technologies, indoor navigation mainly relies on multi-sensor fusion technology, such as UWB (ultra-wideband) positioning, LiDAR, camera and IMU (inertial measurement unit), combined with SLAM (simultaneous localization and mapping) technology, to achieve autonomous positioning and path planning in indoor environments; while outdoor navigation mainly relies on GNSS (global navigation satellite system) combined with camera and LiDAR and other technologies to achieve accurate global positioning in outdoor environments. However, due to the differences in indoor and outdoor environmental characteristics, existing technologies often experience problems such as reduced positioning accuracy and temporary signal loss when switching between indoor and outdoor navigation.

[0004] For example, when the robot switches from an indoor environment to an outdoor environment, the GPS signal may not be stably acquired, causing the robot to deviate from the preset path; conversely, when entering an indoor environment from an outdoor environment, the robot may lose its position or misjudge due to the limited coverage distance of the Bluetooth beacon signal or mutual interference with the GPS signal. In addition, the environmental characteristics of the indoor and outdoor transition areas are complex, and existing technical solutions that rely on signal strength detection (such as judging indoor and outdoor scenes by the strength of the GPS signal or the change of the UWB signal) are easily interfered, resulting in navigation mode switching failures or frequent misjudgments.

[0005] Some existing multimodal navigation technologies attempt to improve positioning accuracy through sensor fusion. For example, the Chinese patent "CN 117826213 A" proposes a monitoring method based on GNSS / INS (inertial navigation system) and INS / UWB dual-mode tight combination positioning, which uses signal strength to determine whether the carrier is in an indoor or outdoor environment, and uses a federal filter to achieve seamless positioning of indoor and outdoor switching. However, this method has a navigation strategy switching buffer in the transition area, which can easily lead to short-term positioning loss, especially when the Bluetooth beacon signal is similar to the GPS signal. The navigation accuracy is significantly affected. In addition, the coverage distance of the Bluetooth beacon signal in open scenes is limited, which further limits the application scenarios of the system.

[0006] With the rapid development of large language model (LLM) technology, it has demonstrated strong capabilities in natural language processing, visual understanding, and multimodal reasoning. By introducing large language models into the field of robot navigation, by combining multimodal sensor data such as vision, lidar, IMU, GPS, etc., it can realize intelligent judgment and reasoning of indoor and outdoor scenes, and improve the stability and accuracy of navigation switching. However, relying solely on large models for scene judgment and switching still has the problems of slow feedback speed and long processing time. Especially in the absence of prior maps, the judgment efficiency and accuracy of the model still need to be optimized.

[0007] Therefore, there is an urgent need for an indoor and outdoor navigation switching technology that combines multimodal sensors and large language models. Through intelligent scene recognition and progressive navigation strategy switching mechanism, the positioning loss and frequent switching problems in the existing technology can be solved to achieve smooth, efficient and seamless navigation in complex environments. Summary of the invention

[0008] The present application provides a robot indoor and outdoor navigation switching method and system based on a large language model, aiming to solve the problems of positioning loss and frequent switching in the prior art.

[0009] In a first aspect, a robot indoor and outdoor navigation switching method based on a large language model, the method comprising:

[0010] Collecting environmental data using multimodal sensors, the environmental data including visual images, LiDAR point clouds, GPS signal data, and IMU data;

[0011] Performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size;

[0012] Input the preprocessed visual image into the large language model, and generate scene classification results through feature extraction and scene reasoning modules;

[0013] Based on the time sliding window, the changing trend of feature points in multiple frames of visual images is analyzed to dynamically adjust the confidence score of scene classification;

[0014] In the transition area between indoor and outdoor, the visual features, point cloud data, GPS signal data and IMU data are combined and weighted fusion is performed using the extended Kalman filter to generate a comprehensive positioning result.

[0015] According to the scene classification results and comprehensive positioning data, the navigation mode is selected, where: the indoor navigation mode builds an environmental map based on laser SLAM; the outdoor navigation mode combines GPS, vision and IMU to achieve positioning;

[0016] During navigation mode switching, the weights of SLAM and GPS modes are dynamically adjusted through a dual-mode parallel operation mechanism.

[0017] In the above solution, optionally, the multimodal sensor includes:

[0018] Camera, LiDAR, GPS module and IMU module, the camera is used to capture visual image data for scene feature extraction; the LiDAR is used to generate two-dimensional or three-dimensional point cloud data for obstacle detection and local map construction; the GPS module is used to provide global positioning data for path planning and positioning; the IMU module is used to provide short-term posture estimation through an accelerometer and a gyroscope.

[0019] In the above solution, optionally, the preprocessing of the visual image includes:

[0020] Use a two-dimensional Gaussian filter to denoise the image and remove high-frequency noise;

[0021] Dynamically adjust the distribution of pixel values ​​to optimize image brightness and contrast;

[0022] The image resolution was adjusted to 224 × 224 pixels using bilinear interpolation;

[0023] Normalization is done by subtracting the mean from the pixel value of each channel and dividing by the standard deviation.

[0024] In the above solution, optionally, the scene classification process of the large language model includes:

[0025] Use visual encoders to extract scene features of walls, ceilings, windows, trees, and roads in images;

[0026] The feature points extracted from the continuous frames are matched through the feature matching algorithm;

[0027] Generate time series change data of feature points based on matching results;

[0028] Use the multimodal reasoning module in the large language model combined with the scene knowledge base to complete scene classification.

[0029] In the above solution, optionally, the weighted fusion step of the extended Kalman filter includes:

[0030] Use ICP algorithm to process the lidar point cloud data to generate a local environment map;

[0031] Use GPS data to provide global positioning coordinates;

[0032] Provide motion acceleration and angular velocity estimation through IMU data;

[0033] The lidar point cloud data, GPS data and IMU data are predicted and updated, and the weight of each data source is dynamically adjusted to generate a fused positioning result.

[0034] In the above solution, optionally, the dual-mode parallel operation mechanism includes:

[0035] In the transition area, the SLAM mode and the GPS mode are activated, and the output data of the two modes are weighted fused;

[0036] Dynamically monitor LiDAR signal stability, GPS signal strength and IMU posture data;

[0037] Dynamically adjust the weights of SLAM and GPS according to the complexity of the environment, so that SLAM gradually weakens and GPS gradually strengthens, or SLAM gradually strengthens and GPS gradually weakens;

[0038] After the switch is complete, deactivate the navigation mode with the lower weight.

[0039] In the above solution, optionally, the scene analysis of the time sliding window includes:

[0040] Extracting a set of feature points from continuous frame visual images;

[0041] Calculate the matching number of feature points in each frame and the Euclidean distance of feature point changes;

[0042] Determine whether the scene transitions from indoor to outdoor or from outdoor to indoor based on the multi-frame change trend.

[0043] In the above solution, optionally, the parameters for calculating the confidence score include:

[0044] The confidence of single-frame scene classification results, the change trend of multi-frame visual feature points, the coverage and quality of lidar point cloud, and the GPS signal strength and SLAM map construction quality.

[0045] In the above solution, optionally, the feedback mechanism includes:

[0046] Real-time monitoring of the effectiveness of GPS signals;

[0047] Check the error level and coverage completeness of SLAM map construction;

[0048] Quality assessment of reflected signals from LiDAR point cloud data;

[0049] When the data from a certain sensor is unstable, the current navigation mode is maintained and other sensors are requested to provide supplementary data.

[0050] In a second aspect, a robot indoor and outdoor navigation switching system based on a large language model, the system comprising:

[0051] Visual processing module: used to collect environmental data using multimodal sensors, the environmental data including visual images, lidar point clouds, GPS signal data and IMU data;

[0052] Image processing module: used for performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size;

[0053] Large language model module: used to input preprocessed visual images into the large language model, generate scene classification results through feature extraction and scene reasoning modules; based on the time sliding window, analyze the change trend of feature points in multiple frames of visual images, and dynamically adjust the confidence score of scene classification;

[0054] Data fusion module: used to combine visual features, point cloud data, GPS signal data and IMU data in the transition area between indoor and outdoor, and use extended Kalman filter for weighted fusion to generate comprehensive positioning results;

[0055] Navigation switching module: used to select navigation mode according to scene classification results and comprehensive positioning data, where: indoor navigation mode builds environment map based on laser SLAM; outdoor navigation mode combines GPS, vision and IMU to achieve positioning;

[0056] Feedback mechanism module: used to dynamically adjust the weights of SLAM and GPS modes through a dual-mode parallel operation mechanism during navigation mode switching.

[0057] Compared with the prior art, this application has at least the following beneficial effects:

[0058] Based on further analysis and research of the existing technical problems, this application recognizes the problems of positioning loss and frequent switching, and uses multimodal sensors to obtain multi-dimensional data of the robot's environment, including visual images, lidar point clouds, GPS signals, and IMU data. The visual image data is de-noised by a two-dimensional Gaussian filter, and the brightness and contrast are dynamically adjusted according to the pixel distribution. It is then adjusted to 224×224 pixels by a bilinear interpolation method, and finally normalized to adapt it to the input requirements of the large language model. The preprocessed image is input to the visual encoder to extract key visual features. The large language model uses these features for reasoning, and combines the internal scene knowledge base to output the scene judgment result. In order to enhance the accuracy of scene judgment, the changing trend of multi-frame image features is analyzed based on a time sliding window. By calculating the stability and continuity of feature point matching, the confidence score of scene judgment is adjusted. The extended Kalman filter is used in the indoor and outdoor transition area to perform weighted fusion of multimodal sensor data, including the local map constructed by the lidar point cloud, the global positioning provided by GPS, the short-term pose estimation of the IMU, and the results of visual feature extraction. When the navigation mode is switched, a dual-mode parallel operation mechanism is adopted to simultaneously activate the SLAM and GPS navigation modes in the indoor and outdoor transition areas. By dynamically adjusting the weights of SLAM and GPS, the target navigation mode is gradually transitioned to at the data fusion level to achieve smooth switching. Compared with the traditional method that relies on detecting the strength of indoor and outdoor signals for switching, the present invention adopts a scene understanding and judgment technology based on a large language model. Through visual perception, image feature extraction and multi-frame analysis, it intelligently judges whether the robot's current environment is indoors or outdoors. At the same time, a dual-mode parallel transition and multi-frame image trend analysis method based on progressive scene judgment is adopted to avoid frequent erroneous switching caused by environmental interference, single-frame image misjudgment and instantaneous signal fluctuations, ensuring the smooth switching and continuous navigation of the robot in the transition area, and significantly improving the system's anti-interference ability and scene understanding ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 One of the flow charts of the robot indoor and outdoor navigation switching method based on a large language model provided in one embodiment of the present application;

[0060] Figure 2 A second flow chart of a method for switching indoor and outdoor navigation of a robot based on a large language model provided in one embodiment of the present application;

[0061] Figure 3 The third flowchart of the robot indoor and outdoor navigation switching method based on a large language model provided in one embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] In one embodiment, Figure 1 As shown, a robot indoor and outdoor navigation switching method based on a large language model is provided, comprising the following steps:

[0064] Collecting environmental data using multimodal sensors, the environmental data including visual images, LiDAR point clouds, GPS signal data, and IMU data;

[0065] Performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size;

[0066] Input the preprocessed visual image into the large language model, and generate scene classification results through feature extraction and scene reasoning modules;

[0067] Based on the time sliding window, the changing trend of feature points in multiple frames of visual images is analyzed to dynamically adjust the confidence score of scene classification;

[0068] In the transition area between indoor and outdoor, the visual features, point cloud data, GPS signal data and IMU data are combined and weighted fusion is performed using the extended Kalman filter to generate a comprehensive positioning result.

[0069] According to the scene classification results and comprehensive positioning data, the navigation mode is selected, where: the indoor navigation mode builds an environmental map based on laser SLAM; the outdoor navigation mode combines GPS, vision and IMU to achieve positioning;

[0070] During navigation mode switching, the weights of SLAM and GPS modes are dynamically adjusted through a dual-mode parallel operation mechanism.

[0071] This embodiment solves the problems of high misjudgment rate and unstable navigation mode switching when scene judgment is performed solely based on GPS signal strength or Bluetooth beacon signal detection in the background technology by combining multimodal data fusion with a large language model. In the transition area, the frequent switching or misjudgment problems caused by signal fluctuations are effectively reduced through multi-frame image trend analysis and dynamic adjustment of confidence. The multimodal reasoning capability of the large language model is utilized in combination with the scene knowledge base to improve the accuracy of indoor and outdoor feature recognition. Multimodal fusion technology effectively enhances the system's scene judgment capability in complex transition areas. The dual-mode parallel operation mechanism significantly improves the stability of navigation switching in the transition area and avoids positioning interruption. The method of dynamically adjusting the SLAM and GPS weights makes the navigation switching process smoother and is suitable for a variety of complex scenarios.

[0072] In one embodiment, the multimodal sensor comprises:

[0073] Camera, LiDAR, GPS module and IMU module, the camera is used to capture visual image data for scene feature extraction; the LiDAR is used to generate two-dimensional or three-dimensional point cloud data for obstacle detection and local map construction; the GPS module is used to provide global positioning data for path planning and positioning; the IMU module is used to provide short-term posture estimation through an accelerometer and a gyroscope.

[0074] The multimodal sensor system in this embodiment is composed of a camera, a laser radar, a GPS module and an IMU module. The camera is installed on the top or front of the robot to obtain continuous frame environmental images for extracting visual features. The laser radar is installed at the bottom or side of the robot to generate high-precision two-dimensional or three-dimensional point cloud data, supporting obstacle detection and environmental map construction. The GPS module is used for global positioning in outdoor environments and supports large-scale path planning and navigation. The IMU module collects the acceleration and angular velocity of the robot's movement through an accelerometer and a gyroscope for short-term posture estimation to compensate for data loss that may occur in other sensors in complex environments.

[0075] The multimodal sensors work in a synchronous manner and transmit the collected data to the data fusion module in real time through the communication interface. The combination of LiDAR and IMU is used for indoor mapping and motion estimation, and the combination of GPS and camera is used for outdoor global positioning and environmental perception.

[0076] In this embodiment, the functional complementarity of each sensor in a specific scenario significantly enhances the robustness of the system. LiDAR and IMU make up for the lack of GPS signals in indoor scenarios, while GPS and camera achieve efficient path planning in outdoor environments. When the LiDAR is blocked or the GPS signal is weakened, the IMU maintains the robot's short-term positioning capability through motion inference.

[0077] The solution of this embodiment can adapt to different indoor and outdoor environment characteristics, including high-rise areas with weak GPS signals, open indoor areas with limited Bluetooth beacon coverage, and transition areas with drastic light changes.

[0078] The collaboration of multimodal sensors increases data redundancy. Even if a single sensor data fails, other sensors can still maintain navigation continuity.

[0079] In one embodiment, the preprocessing of the visual image includes:

[0080] Use a two-dimensional Gaussian filter to denoise the image and remove high-frequency noise;

[0081] Dynamically adjust the distribution of pixel values ​​to optimize image brightness and contrast;

[0082] The image resolution was adjusted to 224 × 224 pixels using bilinear interpolation;

[0083] Normalization is done by subtracting the mean from the pixel value of each channel and dividing by the standard deviation.

[0084] The visual image preprocessing in this embodiment includes the following specific steps:

[0085] Denoising: The image is smoothed using a two-dimensional Gaussian filter to remove high-frequency noise in the environment while retaining the edge features of the image.

[0086] Brightness and contrast adjustment: By calculating the histogram distribution of image pixel values, the pixel values ​​are dynamically redistributed to enhance the details of dark and bright areas and make the image feature points more obvious.

[0087] Image resizing: Use bilinear interpolation to resize the image resolution to 224×224 pixels to fit the input requirements of the large language model.

[0088] Normalization: Subtract the mean from each pixel value and divide it by the standard deviation to standardize the data and reduce the distribution differences between different images.

[0089] The image after the above processing can retain the environmental feature information to the greatest extent while reducing the interference factors in the data.

[0090] Through denoising and dynamic brightness adjustment, the feature contrast of the image is effectively enhanced, making the feature extraction more accurate. Standardization and scale adjustment ensure the uniformity of the format of the input data of the large language model and reduce the error in the model reasoning process.

[0091] Dynamic brightness adjustment and contrast optimization technology enable the system to adapt to bright, dim or contrasty lighting environments. The preprocessed image data is more uniformly distributed, which can significantly reduce the computational complexity of large language models.

[0092] In one embodiment, the scene classification process of the large language model includes:

[0093] Use visual encoders to extract scene features of walls, ceilings, windows, trees, and roads in images;

[0094] The feature points extracted from the continuous frames are matched through the feature matching algorithm;

[0095] Generate time series change data of feature points based on matching results;

[0096] Use the multimodal reasoning module in the large language model combined with the scene knowledge base to complete scene classification.

[0097] This embodiment significantly improves the judgment accuracy of the model by extracting iconic visual features of indoor and outdoor scenes and combining them with feature point matching results, especially in blurred scenes in transition areas.

[0098] The time series change analysis of feature points further reduces the judgment errors caused by instantaneous environmental changes. The feature point matching algorithm can adapt to the changes in image features during robot movement, which helps to maintain high-precision scene classification in complex dynamic environments. Multimodal reasoning combined with the scene knowledge base enables the model to quickly classify and infer complex environments, reducing classification delays.

[0099] In this embodiment, the weighted fusion step of the extended Kalman filter includes:

[0100] Use ICP algorithm to process the lidar point cloud data to generate a local environment map;

[0101] Use GPS data to provide global positioning coordinates;

[0102] Provide motion acceleration and angular velocity estimation through IMU data;

[0103] The lidar point cloud data, GPS data and IMU data are predicted and updated, and the weight of each data source is dynamically adjusted to generate a fused positioning result.

[0104] In this embodiment, an extended Kalman filter (EKF) is used to fuse multimodal sensor data. The specific steps include:

[0105] Prediction step: Based on the acceleration and angular velocity data provided by the IMU, the motion equation is used to predict the position and posture of the robot. The local map generated by the lidar point cloud is used to predict environmental changes.

[0106] Update step: Use GPS data as a global position reference, compare it with the IMU prediction result and update the position. Combine the visual feature data extracted by the visual encoder, match it with the point cloud in the local map, and update the environment model.

[0107] Dynamic weight adjustment: The weight is adjusted according to the signal strength and credibility of each sensor data, for example:

[0108] When the GPS signal strength is weak, increase the weight of LiDAR and IMU.

[0109] When the LiDAR reflection signal is unstable, increase the weight of the visual features.

[0110] The final fusion result includes the robot's current posture, global position and environment map.

[0111] The prediction-update mechanism of the extended Kalman filter enables the system to generate stable fusion results even when some sensor data is missing or abnormal. The dynamic adjustment of sensor weights ensures navigation accuracy in different environments (such as transition areas or signal interference areas). EKF can process nonlinear sensor data (such as IMU acceleration data) and linear data (such as GPS position data) to achieve efficient fusion of multimodal data.

[0112] In one embodiment, the dual-mode parallel operation mechanism includes:

[0113] In the transition area, the SLAM mode and the GPS mode are activated, and the output data of the two modes are weighted fused;

[0114] Dynamically monitor LiDAR signal stability, GPS signal strength and IMU posture data;

[0115] Dynamically adjust the weights of SLAM and GPS according to the complexity of the environment, so that SLAM gradually weakens and GPS gradually strengthens, or SLAM gradually strengthens and GPS gradually weakens;

[0116] After the switch is complete, deactivate the navigation mode with the lower weight.

[0117] The dual-mode parallel operation of this embodiment effectively avoids navigation interruption caused by mode mutation, especially in transition areas with large signal fluctuations. The weight adjustment mechanism can dynamically adapt to different environmental characteristics to ensure the applicability and accuracy of navigation switching. By integrating the advantages of SLAM and GPS modes, the robustness of the system in complex transition areas is enhanced.

[0118] In one embodiment, the scene analysis of the time sliding window includes:

[0119] Extracting a set of feature points from continuous frame visual images;

[0120] Calculate the matching number of feature points in each frame and the Euclidean distance of feature point changes;

[0121] Determine whether the scene transitions from indoor to outdoor or from outdoor to indoor based on the multi-frame change trend.

[0122] The time sliding window of this embodiment can effectively identify the gradual change of scenes from indoors to outdoors or from outdoors to indoors. The trend analysis method reduces misjudgment caused by abnormal single-frame images and improves the stability of scene judgment.

[0123] In this embodiment, the parameters for calculating the confidence score include:

[0124] The confidence of single-frame scene classification results, the change trend of multi-frame visual feature points, the coverage and quality of lidar point cloud, and the GPS signal strength and SLAM map construction quality.

[0125] Through multi-frame trend analysis and sensor fusion confidence calculation, the judgment bias caused by abnormal data in a single frame is reduced. In the transition area between indoor and outdoor, the score calculation weight is dynamically adjusted to adapt to signal fluctuations and feature changes in complex environments. The comprehensive calculation method of confidence score provides a more reliable decision-making basis for navigation mode switching, avoiding frequent or delayed switching.

[0126] In this embodiment, the feedback mechanism includes:

[0127] Real-time monitoring of the effectiveness of GPS signals;

[0128] Check the error level and coverage completeness of SLAM map construction;

[0129] Quality assessment of reflected signals from LiDAR point cloud data;

[0130] When the data from a certain sensor is unstable, the current navigation mode is maintained and other sensors are requested to provide supplementary data.

[0131] The feedback mechanism of this embodiment can monitor and respond to sensor anomalies in real time to ensure the continuity and stability of navigation. During the navigation mode switching process, the feedback mechanism reduces the positioning error caused by a single sensor anomaly by collecting additional data. By real-time monitoring and dynamically adjusting sensor priorities, the system can maintain high-precision positioning in complex environments.

[0132] In one embodiment, Figure 2 and Figure 3 As shown, the method for seamless switching of indoor and outdoor navigation proposed in this embodiment includes the following steps:

[0133] Step 1: The robot uses GPS combined with camera and IMU for navigation in outdoor areas, and laser SLAM and IMU fusion for navigation indoors.

[0134] Outdoor navigation is a high-precision GPS module integrated on the robot. Under high-precision requirements, combined with RTK-GPS technology, triangulation positioning is performed through satellite signals to provide global location information. By combining RGB cameras, the ORB-SLAM visual SLAM algorithm is used to extract key feature points in the image and construct a sparse three-dimensional point cloud map. When the GPS signal is weak or interfered, relative positioning is performed based on visual recognition features to ensure that the robot can identify landmarks such as roads and buildings in the environment, thereby maintaining navigation stability. The IMU sensor is fused with GPS and camera data through the Kalman filter to ensure that the robot obtains accurate and stable pose estimation during navigation. Through the joint work of RGB cameras and IMUs, the VIO algorithm is used to achieve high-precision pose estimation. Combined with the data of visual and inertial sensors, the images captured by the camera and the IMU sensor data can be used to perform motion inference when GPS fails, maintaining the robot's positioning accuracy. At the same time, an adaptive data fusion strategy is adopted: the system can dynamically adjust the weights of different sensors according to the strength of the GPS signal. When the GPS signal is good, GPS is relied on for positioning first; when the signal is weak or lost, the visual and IMU sensors take over together to ensure positioning continuity.

[0135] Indoor navigation uses Velodyne LiDAR to perform 360-degree environmental scanning, obtain obstacle distance information around the robot, build a two-dimensional map of the environment, use Cartographer for indoor SLAM mapping, and combine the ICP algorithm to accurately calculate the pose by matching the continuously scanned point cloud. The IMU plays a complementary role in laser SLAM, especially when the robot turns, moves quickly, or the LiDAR is blocked. The IMU corrects the short-term pose error in SLAM through inertial data. By fusing LiDAR and IMU data, the robot can maintain higher mapping accuracy in complex dynamic indoor environments. Especially when the LiDAR is blocked or the reflected signal is unstable, the IMU can continue to generate reliable pose information through inertial calculation and dynamically adjust the sensor weight according to different motion states.

[0136] Step 2: The camera continuously captures images of the environment around the robot and extracts key features of the scene through visual perception.

[0137] A binocular camera is used to collect environmental images in real time. The camera is installed on the robot and acquires scene images at a fixed frame rate of 30FPS. Each frame of the image contains rich scene features, such as walls, ceilings, ground, windows, trees, etc., which can provide a basis for subsequent scene judgment.

[0138] Assume that there are two images I1 and I2, which contain extracted feature point sets F1 and F2 respectively, and each feature point fi has a corresponding descriptor vector d(fi). In order to match feature points, we need to calculate the distance (Euclidean distance) between the two descriptors:

[0139] dist(f i ,f j )=||d(f i )-d(f j )||

[0140] Where d(f i ) and d(f j ) are two feature points f i and f j The descriptor vector of , ||·|| represents the Euclidean distance, which is used to measure the similarity between two feature points.

[0141] Step 3: Before performing large model inference, perform basic preprocessing on the input image data to ensure that the image noise is minimized and standardized to meet the input requirements of the large model.

[0142] Image preprocessing is a necessary step before large model reasoning to ensure that the large model can extract effective feature information from the image. Preprocessing includes removing image noise, adjusting brightness and contrast, and standardizing the image data format to match the input requirements of the large model.

[0143] Noise removal: Use a two-dimensional Gaussian filter to smooth the image, reduce the interference of high-frequency noise, and retain important edge features in the image.

[0144] Brightness and contrast adjustment: By redistributing the pixel values ​​of the image, the detail contrast between the dark and bright parts of the image is improved, so that the image features are still clearly visible in high-contrast or low-contrast environments. The contrast is dynamically adjusted by calculating the average brightness value of the image to ensure that the visual features in the image remain consistent under different lighting conditions.

[0145] Image rescaling and standardization: Bilinear interpolation is used to adjust the resolution of the original image to the input size required by the large model (224×224 pixels) to ensure that the size of all input images is consistent. The pixel values ​​of the image are scaled to the range of [0,1]. According to the requirements of the large model, the pixel values ​​of each channel are subtracted from the mean and divided by the standard deviation to ensure that the data has zero mean and unit variance, which helps to eliminate differences between different data and improve the stability and robustness of the model output results.

[0146] Step 4: Based on the multimodal reasoning capability of the large language model, analyze the visual features in the image and determine whether the robot is currently in an indoor or outdoor environment.

[0147] The large model receives the preprocessed image data and analyzes the visual features contained in the image, such as the shape, color, edge, and texture information of the object. After the visual encoder, the pixel information of the image is converted into a feature vector. Then, by analyzing visual features such as walls, ceilings, windows, ground, trees, sky, roads, etc., key features related to indoor and outdoor environments are extracted. Its multimodal cross-attention mechanism combines visual features with the semantic knowledge mastered by the model in the pre-training stage. Based on the results of multimodal reasoning, the model will output a classification result, that is, whether the robot is currently indoors or outdoors. For example, if the image contains "ceiling, windows, lights", the model tends to judge it as an indoor environment. If "sky, trees, roads" appear in the image, the model tends to judge it as an outdoor environment.

[0148] Step 5: The system will make a comprehensive judgment on the location based on the changing trends of multiple frames of images. At the same time, the system will calculate the confidence of each judgment result of the model. The confidence evaluation is used to decide whether to switch the navigation strategy to avoid frequent switching caused by single-frame misjudgment.

[0149] The system collects multiple consecutive frames of images from the camera and passes these data to the large model. After analyzing the visual features of each frame, it performs comprehensive reasoning based on the changing trends of these features. Compared with single-frame image judgment, continuous frame processing enables the model to capture longer-term stable changes in the scene and ensure the accuracy of judgment. For example, as the robot approaches the window, features such as windows and sunlight gradually appear in the image, and the model can gradually confirm that the scene has turned from indoors to outdoors. This reasoning method based on multi-frame images avoids misjudgments caused by single-frame feature ambiguity or specific environmental interference (sudden changes in lighting, etc.).

[0150] After making a scene judgment, the system will calculate a confidence score based on the degree of matching of the current visual features by the large model. If the trend of visual features changes in multiple consecutive frames is consistent, such as gradually identifying outdoor features such as trees, sky, and roads, the confidence will increase, and the model will be more confident that the current environment is outdoor; if the features in the image are unstable or blurred (such as excessive changes in light in a certain frame), the confidence will be lower, and the model may choose to maintain the current navigation mode and wait to collect more data to determine whether to switch. This strategy effectively prevents the robot from switching navigation modes unnecessarily in scene transition areas (doorways moving from indoors to outdoors).

[0151] In addition to real-time scene judgment, the system also captures scene transitions by analyzing image change trends over a longer period of time. For example, when the robot walks from indoors to outdoors, the visual features will gradually transition from the indoor ceiling and walls to the outdoor sky, ground, trees, etc. The system can recognize these gradual scene changes and adjust the confidence threshold when necessary to make switching more flexible. The system can dynamically adjust the confidence threshold based on the complexity of the scene. In complex environments, the threshold can be lowered to speed up switching; in stable environments, the threshold can be increased to ensure stable judgment.

[0152] Step 6: Based on the judgment result of the large model (indoor or outdoor), the system switches the corresponding navigation strategy to ensure that the robot uses the optimal navigation method in different environments.

[0153] The system receives the scene judgment results and confidence scores from the large model, receives the scene judgment results of the large model in real time through the ROS topic, and decides whether to switch the navigation strategy based on the judgment results of the large model according to the preset confidence threshold. If it is judged as "indoor" and the confidence is high, the SLAM fusion navigation strategy is started; if it is judged as "outdoor" and the confidence is high, it is switched to the GPS combined with vision navigation strategy.

[0154] In order to ensure smoothness when switching from one navigation mode to another, a smooth switching mechanism is designed to avoid navigation interruption caused by sudden operations. When the large model judges to switch from indoor to outdoor, the system keeps SLAM and GPS modes running in parallel for a short time. The SLAM mode will gradually weaken its control, and the GPS mode will gradually take over the robot navigation. In the transition stage, the system fuses SLAM and GPS data in a weighted manner, gradually shifting from relying on lidar to relying mainly on GPS data.

[0155] Step 7: In the indoor and outdoor transition area, the system uses progressive scene judgment and mode switching to ensure that the robot can transition smoothly in complex environments.

[0156] In order to avoid frequent switching of navigation strategies in transition areas, the system adopts progressive scene judgment, combining the changing trends of multi-frame images and sensor data to gradually switch navigation modes.

[0157] In the transition area between indoors and outdoors, scene features change rapidly. The system analyzes the feature change trend of multiple frames through a time sliding window to avoid misjudgment caused by single-frame judgment. In this area, the confidence threshold is dynamically adjusted according to the complexity of the environment, allowing the navigation mode to be switched even when the confidence is relatively low. This avoids the problem of frequent switching in the transition area due to confusion of scene features.

[0158] During the switching period, the system can combine the data provided by multiple sensors (lidar, camera, IMU, GPS) and gradually switch scenes by weighted fusion of sensor data. When the robot moves from indoors to outdoors, the extended Kalman filter (EKF) is used to weighted fuse the GPS, IMU and lidar data. When entering the transition area, it still relies on SLAM to build a local map, but gradually increases the weight of GPS data, and finally the navigation is completely controlled by GPS.

[0159] The system monitors the sensor status (GPS signal strength, SLAM map quality, etc.) to ensure the stability of the navigation mode during the switching process. During the entire switching process, the system will feedback information such as signal strength and sensor status through the user interface or internal logs. If the status of a sensor is unstable during the switching process (such as weak GPS signal or large SLAM map error), the system will delay the switch or fall back to the original mode to ensure that the robot can continue to navigate smoothly.

[0160] Step 8: In order to enhance the robustness of the system in complex environments, multimodal data fusion is used to combine the data of multiple sensors such as cameras, lidar, GPS, IMU, etc. to perform joint judgment and positioning. Multimodal fusion can achieve more stable and continuous positioning and environmental perception through the complementarity of different sensors, especially in complex indoor and outdoor environments or transition areas.

[0161] Camera (RGB camera): Provides visual features of the scene for feature recognition, object detection, and large model input. LiDAR (Velodyne LiDAR): Generates a 2D point cloud of the surrounding environment through laser scanning, which is used to build accurate maps and is suitable for obstacle detection and avoidance in indoor environments. GPS module: Provides global positioning information for the robot in outdoor environments, and can perform path planning and positioning in a large outdoor area. IMU: Obtains the acceleration and angular velocity data of the robot through the accelerometer and gyroscope, and provides short-term pose estimation when the sensor signal is weak.

[0162] This embodiment solves the problem of single sensor being easily interfered and inaccurately positioned in complex environments by integrating data from multiple sensors (camera, lidar, GPS, IMU). Especially in the transition area between indoor and outdoor, the present invention adopts progressive scene judgment and realizes the smooth transition of the robot when the environment changes drastically through the coordinated processing of multimodal data.

[0163] The system can automatically switch navigation strategies according to different environments (indoors or outdoors). It not only relies on large models for scene judgment, but also combines the changing trends of multi-frame images to gradually switch navigation modes, avoiding the problem of frequent switching of navigation modes at scene boundaries and improving the robot's intelligent performance in complex environments.

[0164] In this embodiment, a confidence evaluation mechanism based on the large model scene judgment result is designed, which can dynamically adjust the accuracy of scene judgment and the triggering conditions for switching navigation strategies. By setting the confidence threshold, the system instability caused by single-frame judgment errors is avoided, and the system's adaptive ability is enhanced, making the system more robust in scene recognition and navigation mode switching, especially in boundary areas with complex environmental changes, and realizing progressive processing of scene transition areas.

[0165] When the robot enters the indoor-outdoor transition area, the present invention ensures a smooth transition between indoor navigation and outdoor navigation by running the dual modes in parallel. The dual modes are activated simultaneously in a short period of time, and through weighted fusion technology, the weight of one mode is gradually reduced and the weight of the other mode is increased, thereby achieving seamless switching.

[0166] Compared with the traditional method that relies on detecting the strength of indoor and outdoor signals for switching, the present invention adopts scene understanding and judgment technology based on a large language model. Through visual perception, image feature extraction and multi-frame analysis, it intelligently judges whether the robot's current environment is indoors or outdoors. At the same time, it adopts a dual-mode parallel transition and multi-frame image trend analysis method based on progressive scene judgment to avoid the influence of environmental interference, as well as frequent erroneous switching caused by single-frame image misjudgment and instantaneous signal fluctuations, ensuring the robot's smooth switching and continuous navigation in the transition area, and significantly improving the system's anti-interference ability and scene understanding ability.

[0167] With the development of robotics technology, robots are increasingly used in indoor and outdoor navigation, especially in logistics, security, inspection and other scenarios. Robots need to be able to move autonomously and perform tasks in indoor and outdoor environments. However, the differences between indoor and outdoor environments and the complexity of transition areas pose great challenges to robot navigation. Existing robot navigation technology usually relies on switching indoor and outdoor navigation strategies based on changes in signal strength, which has obvious limitations.

[0168] At present, most mobile robots rely on the changes in GPS signal strength or indoor Bluetooth beacon signal strength to judge the robot's current environment when switching between indoor and outdoor navigation strategies. However, signal strength is easily disturbed in complex environments in transition areas, where environmental characteristics are complex and signal strength fluctuates significantly. In places with weak outdoor signals (such as high-rise buildings, tunnels, or densely treed areas), the robot may mistakenly judge that it has entered indoors, thereby switching to an inappropriate navigation mode, resulting in positioning errors; while in places near windows or doorways indoors, signal reflection or attenuation may cause the system to misjudge it as outdoor, thereby frequently switching navigation modes, resulting in unstable navigation. Existing methods that rely on signal strength to switch navigation modes often perform poorly in transition areas, making it difficult to achieve a smooth transition, affecting the continuity of the robot's tasks and the accuracy of navigation. Therefore, how to combine multimodal data, scene feature understanding, dynamic confidence evaluation and other methods to design a more intelligent and stable navigation strategy switching system has become a technical problem that needs to be solved urgently in current robot navigation technology.

[0169] Based on these background issues, this embodiment proposes a new indoor and outdoor navigation mode switching method based on multimodal data fusion and large model scene judgment. Through multi-sensor fusion, scene intelligent recognition, and progressive switching mechanism, the navigation accuracy and stability of the robot in complex environments are improved, overcoming the shortcomings of the existing technology.

[0170] In one embodiment, a robot indoor and outdoor navigation switching system based on a large language model is provided, comprising:

[0171] Visual processing module: used to collect environmental data using multimodal sensors, the environmental data including visual images, lidar point clouds, GPS signal data and IMU data;

[0172] Image processing module: used for performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size;

[0173] Large language model module: used to input preprocessed visual images into the large language model, generate scene classification results through feature extraction and scene reasoning modules; based on the time sliding window, analyze the change trend of feature points in multiple frames of visual images, and dynamically adjust the confidence score of scene classification;

[0174] Data fusion module: used to combine visual features, point cloud data, GPS signal data and IMU data in the transition area between indoor and outdoor, and use the extended Kalman filter for weighted fusion to generate comprehensive positioning results;

[0175] Navigation switching module: used to select navigation mode according to scene classification results and comprehensive positioning data, where: indoor navigation mode builds environment map based on laser SLAM; outdoor navigation mode combines GPS, vision and IMU to achieve positioning;

[0176] Feedback mechanism module: used to dynamically adjust the weights of SLAM and GPS modes through a dual-mode parallel operation mechanism during navigation mode switching.

[0177] The specific implementation content of each module can be found in the above definition of the robot indoor and outdoor navigation switching method based on the large language model, which will not be repeated here.

[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A robot indoor and outdoor navigation switching method based on a large language model, characterized in that: The method comprises: Collecting environmental data using multimodal sensors, the environmental data including visual images, LiDAR point clouds, GPS signal data, and IMU data; Performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size; Input the preprocessed visual image into the large language model, and generate scene classification results through feature extraction and scene reasoning modules; Based on the time sliding window, the changing trend of feature points in multiple frames of visual images is analyzed to dynamically adjust the confidence score of scene classification; In the transition area between indoor and outdoor, the visual features, point cloud data, GPS signal data and IMU data are combined and weighted fusion is performed using the extended Kalman filter to generate a comprehensive positioning result. According to the scene classification results and comprehensive positioning data, the navigation mode is selected, where: the indoor navigation mode builds an environmental map based on laser SLAM; the outdoor navigation mode combines GPS, vision and IMU to achieve positioning; During navigation mode switching, the weights of SLAM and GPS modes are dynamically adjusted through a dual-mode parallel operation mechanism.

2. The navigation switching method according to claim 1, characterized in that: The multimodal sensor comprises: Camera, LiDAR, GPS module and IMU module, the camera is used to capture visual image data for scene feature extraction; the LiDAR is used to generate two-dimensional or three-dimensional point cloud data for obstacle detection and local map construction; the GPS module is used to provide global positioning data for path planning and positioning; the IMU module is used to provide short-term posture estimation through an accelerometer and a gyroscope.

3. The navigation switching method according to claim 1, characterized in that: The preprocessing of the visual image includes: Use a two-dimensional Gaussian filter to denoise the image and remove high-frequency noise; Dynamically adjust the distribution of pixel values ​​to optimize image brightness and contrast; The image resolution was adjusted to 224 × 224 pixels using bilinear interpolation; Normalization is done by subtracting the mean from the pixel value of each channel and dividing by the standard deviation.

4. The navigation switching method according to claim 1, characterized in that: The scene classification process of the large language model includes: Use visual encoders to extract scene features of walls, ceilings, windows, trees, and roads in images; The feature points extracted from the continuous frames are matched through the feature matching algorithm; Generate time series change data of feature points based on matching results; Use the multimodal reasoning module in the large language model combined with the scene knowledge base to complete scene classification.

5. The navigation switching method according to claim 1, characterized in that: The weighted fusion step of the extended Kalman filter includes: Use ICP algorithm to process the lidar point cloud data to generate a local environment map; Use GPS data to provide global positioning coordinates; Provide motion acceleration and angular velocity estimation through IMU data; The lidar point cloud data, GPS data and IMU data are predicted and updated, and the weight of each data source is dynamically adjusted to generate a fused positioning result.

6. The navigation switching method according to claim 1, characterized in that: The dual-mode parallel operation mechanism includes: In the transition area, the SLAM mode and the GPS mode are activated, and the output data of the two modes are weighted fused; Dynamically monitor LiDAR signal stability, GPS signal strength and IMU posture data; Dynamically adjust the weights of SLAM and GPS according to the complexity of the environment, so that SLAM gradually weakens and GPS gradually strengthens, or SLAM gradually strengthens and GPS gradually weakens; After the switch is complete, deactivate the navigation mode with the lower weight.

7. The navigation switching method according to claim 1, characterized in that: The scenario analysis of the time sliding window includes: Extracting a set of feature points from continuous frame visual images; Calculate the matching number of feature points in each frame and the Euclidean distance of feature point changes; Determine whether the scene transitions from indoor to outdoor or from outdoor to indoor based on the multi-frame change trend.

8. The navigation switching method according to claim 1, characterized in that: The parameters for calculating the confidence score include: The confidence of single-frame scene classification results, the change trend of multi-frame visual feature points, the coverage and quality of lidar point cloud, and the GPS signal strength and SLAM map construction quality.

9. The navigation switching method according to claim 1, characterized in that: The method further comprises: a feedback mechanism: Real-time monitoring of the effectiveness of GPS signals; Check the error level and coverage completeness of SLAM map construction; Quality assessment of reflected signals from LiDAR point cloud data; When the data from a certain sensor is unstable, the current navigation mode is maintained and other sensors are requested to provide supplementary data.

10. A robot indoor and outdoor navigation switching system based on a large language model, characterized in that: include: Visual processing module: used to collect environmental data using multimodal sensors, the environmental data including visual images, lidar point clouds, GPS signal data and IMU data; Image processing module: used for performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size; Large language model module: used to input preprocessed visual images into the large language model and generate scene classification results through feature extraction and scene reasoning modules; Based on the time sliding window, the changing trend of feature points in multiple frames of visual images is analyzed to dynamically adjust the confidence score of scene classification; Data fusion module: used to combine visual features, point cloud data, GPS signal data and IMU data in the transition area between indoor and outdoor, and use the extended Kalman filter for weighted fusion to generate comprehensive positioning results; Navigation switching module: used to select navigation mode according to scene classification results and comprehensive positioning data, where: indoor navigation mode builds environment map based on laser SLAM; outdoor navigation mode combines GPS, vision and IMU to achieve positioning; Feedback mechanism module: used to dynamically adjust the weights of SLAM and GPS modes through a dual-mode parallel operation mechanism during navigation mode switching.

Images