A robot indoor-outdoor navigation switching method and system based on a large language model

Abstract

The application discloses a robot indoor and outdoor navigation switching method and system based on a large language model. Multidimensional data of the environment where the robot is located is obtained by using a multi-modal sensor, including visual images, laser radar point clouds, GPS signals, and IMU data. The preprocessed images are input into a visual encoder to extract key visual features. The large language model uses these features for reasoning, combines with an internal scene knowledge base, and outputs a scene judgment result. To enhance the accuracy of scene judgment, the change trend of multi-frame image features is analyzed based on a time sliding window. An extended Kalman filter is used to weight and fuse multi-modal sensor data in the indoor and outdoor transition area. When the navigation mode is switched, a dual-mode parallel running mechanism is used, and SLAM and GPS navigation modes are activated simultaneously in the indoor and outdoor transition area. By dynamically adjusting the weights of SLAM and GPS, the target navigation mode is gradually transitioned at the data fusion level, and smooth switching is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a robot indoor-outdoor navigation switching method and system based on a large language model. BACKGROUND

[0002] With the rapid development of artificial intelligence and robot technology, robots are increasingly widely used in industries, services, logistics and other fields. Robot navigation technology is a key technology for realizing autonomous movement of robots, especially seamless switching navigation indoors and outdoors in complex environments, which has become a hot and difficult point of current research.

[0003] In the prior art, indoor navigation mainly relies on multi-sensor fusion technology, such as UWB (Ultra-Wideband) positioning, laser radar, camera and IMU (Inertial Measurement Unit), combined with SLAM (Simultaneous Localization and Mapping) technology, to realize autonomous positioning and path planning in an indoor environment; while outdoor navigation mainly relies on GNSS (Global Navigation Satellite System) combined with camera and laser radar technology to realize accurate global positioning in an outdoor environment. However, due to the differences in indoor and outdoor environmental characteristics, the existing technology often has problems such as decreased positioning accuracy and temporary loss of signal when switching between indoor and outdoor navigation.

[0004] For example, when a robot switches from an indoor environment to an outdoor environment, GPS signals may not have been stably acquired, causing the robot to deviate from the preset path; conversely, when entering an indoor environment from an outdoor environment, due to the limited coverage distance of the Bluetooth beacon signal or the mutual interference with the GPS signal, the robot may lose positioning or misjudge. In addition, the environment characteristics of the indoor-outdoor transition area are complex, and the existing technical solutions that rely on signal strength detection (such as judging indoor and outdoor scenes by GPS signal strength or UWB signal changes) are easily disturbed, resulting in failure of navigation mode switching or frequent misjudgment.

[0005] Some existing multi-modal navigation technologies attempt to improve positioning accuracy through sensor fusion. For example, Chinese patent “CN 117826213 A” proposes a monitoring method based on GNSS / INS (Inertial Navigation System) and INS / UWB dual-mode tight combination positioning, which uses signal strength to determine whether the carrier is in an indoor or outdoor environment, and realizes seamless positioning in indoor and outdoor switching through a federal filter. However, this method has a navigation strategy switching buffer in the transition area, which easily causes short-term loss of positioning, especially when the Bluetooth beacon signal and the GPS signal are similar in strength, the navigation accuracy is significantly affected. In addition, the coverage distance of the Bluetooth beacon signal is limited in an open scene, further limiting the application scenarios of the system.

[0006] With the rapid development of large language model (LLM) technology, it has shown strong capabilities in natural language processing, visual understanding and multi-modal reasoning. By introducing large language models into the field of robot navigation, combining visual, lidar, IMU, GPS and other multi-modal sensor data, intelligent judgment and reasoning of indoor and outdoor scenes can be achieved, and the stability and accuracy of navigation switching can be improved. However, relying solely on large models for scene judgment and switching still has the problem of slow feedback speed and long processing time, especially in the absence of prior maps, the efficiency and accuracy of the model's judgment still need to be optimized.

[0007] Therefore, there is an urgent need for an indoor and outdoor navigation switching technology that combines multi-modal sensors and large language models, through intelligent scene recognition and gradual navigation strategy switching mechanism, to solve the positioning loss and frequent switching problems in the prior art, and to achieve smooth and efficient seamless navigation in complex environments. SUMMARY

[0008] The present application provides a robot indoor and outdoor navigation switching method and system based on a large language model, aiming to solve the positioning loss and frequent switching problems in the prior art.

[0009] In a first aspect, a robot indoor and outdoor navigation switching method based on a large language model, the method comprising:

[0010] Collecting environmental data using multi-modal sensors, the environmental data including visual images, lidar point clouds, GPS signal data and IMU data;

[0011] Performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size;

[0012] Inputting the preprocessed visual image into a large language model to generate a scene classification result through a feature extraction and scene reasoning module;

[0013] Based on a time sliding window, analyzing the feature point change trend in multiple frames of visual images, and dynamically adjusting the confidence score of scene classification;

[0014] In the indoor and outdoor transition area, combining visual features, point cloud data, GPS signal data and IMU data, using an extended Kalman filter for weighted fusion to generate a comprehensive positioning result;

[0015] According to the scene classification result and the comprehensive positioning data, a navigation mode is selected, wherein: the indoor navigation mode is based on laser SLAM to construct an environmental map; the outdoor navigation mode combines GPS, vision and IMU to realize positioning;

[0016] During navigation mode switching, a dual-mode parallel running mechanism is used to dynamically adjust the weights of SLAM and GPS modes.

[0017] Optionally, in the scheme, the multi-modal sensor comprises:

[0018] a camera for capturing visual image data for scene feature extraction; a lidar for generating two-dimensional or three-dimensional point cloud data for obstacle detection and local map construction; a GPS module for providing global positioning data for path planning and localization; and an IMU module for providing short-term pose estimation through an accelerometer and a gyroscope.

[0019] Optionally, in the scheme, the preprocessing of the visual image comprises:

[0020] image denoising using a two-dimensional Gaussian filter to remove high-frequency noise;

[0021] dynamically adjusting the distribution of pixel values to optimize the brightness and contrast of the image;

[0022] adjusting the image resolution to 224x224 pixels using bilinear interpolation;

[0023] subtracting the mean and dividing by the standard deviation of each channel of pixel values to complete the normalization process.

[0024] Optionally, in the scheme, the scene classification process of the large language model comprises:

[0025] extracting wall, ceiling, window, tree, and road scene features from the image using a visual encoder;

[0026] matching the feature points extracted in consecutive frames through a feature matching algorithm;

[0027] generating time series variation data of the feature points according to the matching results;

[0028] using a multi-modal inference module in the large language model to complete scene classification in combination with a scene knowledge base.

[0029] Optionally, in the scheme, the weighting fusion step of the extended Kalman filter comprises:

[0030] performing ICP algorithm processing on the lidar point cloud data to generate a local environment map;

[0031] using GPS data to provide global positioning coordinates;

[0032] providing motion acceleration and angular velocity estimation through IMU data;

[0033] The laser radar point cloud data, GPS data and IMU data are predicted and updated, the weight of each data source is dynamically adjusted, and a fused positioning result is generated.

[0034] In the above scheme, optionally, the dual-mode parallel running mechanism comprises:

[0035] In the transition area, the SLAM mode and the GPS mode are activated, and the output data of the two modes is weighted and fused;

[0036] The stability of the laser radar signal, the strength of the GPS signal and the IMU pose data are dynamically monitored;

[0037] According to the environmental complexity, the weights of SLAM and GPS are dynamically adjusted, so that SLAM gradually weakens, GPS gradually strengthens, or SLAM gradually strengthens, and GPS gradually weakens;

[0038] After the switching is completed, the navigation mode with lower weight is disabled.

[0039] In the above scheme, optionally, the scene analysis of the time sliding window comprises:

[0040] A set of feature points is extracted from the continuous frame visual image;

[0041] The number of matches of each frame feature point and the Euclidean distance of feature point change are calculated;

[0042] According to the change trend of multiple frames, it is judged whether the scene is from indoor to outdoor or from outdoor to indoor.

[0043] In the above scheme, optionally, the parameters of the confidence score calculation comprise:

[0044] The confidence of single-frame scene classification result, the change trend of multiple-frame visual feature points, the coverage range and quality of laser radar point cloud, and the strength of GPS signal and the quality of SLAM map construction.

[0045] In the above scheme, optionally, the feedback mechanism comprises:

[0046] The effectiveness of the GPS signal is monitored in real time;

[0047] The error level and coverage integrity of SLAM map construction are detected;

[0048] The quality of the reflection signal of the laser radar point cloud data is evaluated;

[0049] When the data of a certain sensor is unstable, the current navigation mode is maintained and other sensors are requested to provide supplementary data.

[0050] In a second aspect, a robot indoor and outdoor navigation switching system based on a large language model, the system comprises:

[0051] Visual processing module: for collecting environmental data using multi-modal sensors, including visual images, lidar point clouds, GPS signal data, and IMU data;

[0052] Image processing module: for noise removal, brightness adjustment, and standardization preprocessing of the visual image data, and adjusting the image resolution to a specified size;

[0053] Large language model module: for inputting the preprocessed visual images into a large language model to generate scene classification results through feature extraction and scene inference modules; based on a time sliding window, analyzing the feature point change trend in multiple frames of visual images, and dynamically adjusting the confidence score of scene classification;

[0054] Data fusion module: for combining visual features, point cloud data, GPS signal data, and IMU data in indoor-outdoor transition areas, using an extended Kalman filter for weighted fusion to generate a comprehensive positioning result;

[0055] Navigation switching module: for selecting a navigation mode according to the scene classification result and the comprehensive positioning data, wherein: the indoor navigation mode is based on laser SLAM to construct an environmental map; the outdoor navigation mode combines GPS, vision, and IMU to realize positioning;

[0056] Feedback mechanism module: for dynamically adjusting the weights of SLAM and GPS modes during navigation mode switching through a dual-mode parallel running mechanism.

[0057] Compared with the prior art, the present application has at least the following beneficial effects:

[0058] The present application is based on further analysis and research of the problems of positioning loss and frequent switching in the prior art. By using multi-modal sensors to obtain multi-dimensional data of the environment where the robot is located, including visual images, laser radar point clouds, GPS signals and IMU data. The visual image data is removed by a two-dimensional Gaussian filter, and the brightness and contrast are dynamically adjusted according to the pixel distribution, and then adjusted to 224x224 pixels by a bilinear interpolation method, and finally normalized to adapt to the input requirements of the large language model. The preprocessed image is input into the visual encoder to extract key visual features. The large language model uses these features for reasoning, combining with the internal scene knowledge base, and outputs the scene judgment result. To enhance the accuracy of scene judgment, the trend of multi-frame image features is analyzed based on a time sliding window. By calculating the stability and continuity of feature point matching, the confidence score of scene judgment is adjusted. In the indoor-outdoor transition area, an extended Kalman filter is used to weight and fuse multi-modal sensor data, including the local map constructed by laser radar point cloud, the global positioning provided by GPS, the short-term pose estimation of IMU and the results of visual feature extraction. When switching navigation modes, a dual-mode parallel running mechanism is adopted, and SLAM and GPS navigation modes are activated simultaneously in the indoor-outdoor transition area. By dynamically adjusting the weights of SLAM and GPS, the system gradually transitions to the target navigation mode at the data fusion level, realizing smooth switching. Compared with the traditional method of switching based on the detection of indoor and outdoor signal strength, the present application uses scene understanding and judgment technology based on large language model, intelligently judges whether the robot is currently in an indoor or outdoor environment through visual perception, image feature extraction and multi-frame analysis, and simultaneously uses a gradual scene judgment-based dual-mode parallel transition and multi-frame image trend analysis method to avoid frequent false switching caused by environmental interference, single-frame image misjudgment and instantaneous signal fluctuations, ensuring smooth switching and continuous navigation of the robot in the transition area, and significantly improving the anti-interference ability and scene understanding ability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 One of the flowcharts of the robot indoor-outdoor navigation switching method based on a large language model provided by an embodiment of the present application;

[0060] Figure 2 One of the flowcharts of the robot indoor-outdoor navigation switching method based on a large language model provided by an embodiment of the present application;

[0061] Figure 3 One of the flowcharts of the robot indoor-outdoor navigation switching method based on a large language model provided by an embodiment of the present application; DETAILED DESCRIPTION

[0062] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0063] In one embodiment, as shown in Figure 1 a large language model-based robot indoor-outdoor navigation switching method is provided, comprising the following steps:

[0064] Collecting environment data using multi-modal sensors, including visual images, lidar point clouds, GPS signal data and IMU data;

[0065] Performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size;

[0066] Inputting the preprocessed visual image into a large language model to generate a scene classification result through a feature extraction and scene reasoning module;

[0067] Based on a time sliding window, analyzing the feature point trend in multiple frames of visual images to dynamically adjust the confidence score of scene classification;

[0068] In the indoor-outdoor transition area, combining visual features, point cloud data, GPS signal data and IMU data, using an extended Kalman filter for weighted fusion to generate a comprehensive positioning result;

[0069] According to the scene classification result and the comprehensive positioning data, selecting a navigation mode, wherein: the indoor navigation mode is based on laser SLAM to construct an environment map; the outdoor navigation mode combines GPS, vision and IMU to realize positioning;

[0070] During navigation mode switching, the weights of SLAM and GPS modes are dynamically adjusted through a dual-mode parallel running mechanism.

[0071] The present embodiment solves the problems of high misjudgment rate and unstable navigation mode switching in the background art by combining multi-modal data fusion with a large language model. In the transition area, through multi-frame image trend analysis and confidence dynamic adjustment, the problem of frequent switching or misjudgment caused by signal fluctuations is effectively reduced. Using the multi-modal reasoning ability of the large language model and combining the scene knowledge base, the accuracy of indoor-outdoor feature recognition is improved. The multi-modal fusion technology effectively enhances the scene judgment ability of the system in complex transition areas. The dual-mode parallel running mechanism significantly improves the stability of navigation switching in the transition area, avoiding positioning interruption. The dynamic adjustment of the weights of SLAM and GPS makes the navigation switching process more stable and suitable for various complex scenarios.

[0072] In one embodiment, the multi-modal sensor comprises:

[0073] a camera for capturing visual image data for scene feature extraction; a lidar for generating two-dimensional or three-dimensional point cloud data for obstacle detection and local map construction; a GPS module for providing global positioning data for path planning and localization; and an IMU module for providing short-term pose estimation through accelerometer and gyroscope.

[0074] The multi-modal sensor system in this embodiment consists of a camera, a lidar, a GPS module, and an IMU module. The camera is mounted on the top or front end of the robot to acquire continuous frame environment images for visual feature extraction. The lidar is mounted on the bottom or side of the robot to generate high-precision two-dimensional or three-dimensional point cloud data, supporting obstacle detection and environment map construction. The GPS module is used for global positioning in outdoor environments, supporting large-scale path planning and navigation. The IMU module collects the acceleration and angular velocity of the robot's motion through accelerometer and gyroscope for short-term pose estimation, compensating for possible data gaps in complex environments.

[0075] The multi-modal sensors work in a synchronous manner, and the collected data is transmitted in real time to the data fusion module through the communication interface. The combination of lidar and IMU is used for indoor mapping and motion estimation, and the combination of GPS and camera is used for outdoor global positioning and environment perception.

[0076] In this embodiment, the complementarity of each sensor in a specific scenario significantly enhances the robustness of the system. The lidar and IMU compensate for the signal loss problem of GPS in indoor scenarios, while GPS and camera achieve efficient path planning in outdoor environments. The IMU maintains the robot's short-term positioning capability through motion inference when the lidar is blocked or the GPS signal is weak.

[0077] This embodiment scheme can adapt to different indoor and outdoor environmental characteristics, including high-rise areas with weak GPS signals, open indoor areas with limited Bluetooth beacon coverage, and transition areas with dramatic changes in light.

[0078] The cooperation of multi-modal sensors increases data redundancy, so even if a single sensor data fails, other sensors can still maintain the continuity of navigation.

[0079] In one embodiment, the preprocessing of the visual image comprises:

[0080] Image denoising using a two-dimensional Gaussian filter to remove high-frequency noise;

[0081] dynamically adjusting the distribution of pixel values to optimize the brightness and contrast of the image;

[0082] Adjusting the image resolution to 224x224 pixels using bilinear interpolation;

[0083] Subtracting the mean and dividing by the standard deviation of each channel's pixel value to complete the normalization process.

[0084] The visual image preprocessing of this embodiment includes the following specific steps:

[0085] De-noising: Smooth the image by a two-dimensional Gaussian filter to remove high-frequency noise in the environment while preserving the edge features of the image.

[0086] Brightness and contrast adjustment: Dynamically redistribute the pixel values by calculating the histogram distribution of the image pixel values to enhance the details of dark and bright parts, making the image feature points more obvious.

[0087] Image size adjustment: Use bilinear interpolation to adjust the image resolution to 224x224 pixels to adapt to the input requirements of large language models.

[0088] Normalization: Subtract the mean and divide by the standard deviation of each pixel value to standardize the data and reduce the distribution differences between different images.

[0089] The image after the above processing can maximize the preservation of environmental feature information while reducing interference factors in the data.

[0090] Through de-noising and dynamic brightness adjustment, the feature contrast of the image is effectively enhanced, making the feature extraction more accurate. Standardization and scale adjustment ensure the uniformity of the input data format of large language models, reducing errors in the model inference process.

[0091] Dynamic brightness adjustment and contrast optimization technology enables the system to adapt to bright, dim, or high-contrast lighting environments. The preprocessed image data is more consistent in distribution, which can significantly reduce the computational complexity of large language models.

[0092] In one embodiment, the scene classification process of the large language model includes:

[0093] Extracting wall, ceiling, window, tree, and road scene features from the image using a visual encoder;

[0094] Matching the feature points extracted in consecutive frames through a feature matching algorithm;

[0095] Generating time series variation data of feature points based on the matching results;

[0096] The multi-modal inference module in the large language model is used in combination with a scene knowledge base to complete scene classification.

[0097] The embodiment extracts the landmark visual features of indoor and outdoor scenes and combines the feature point matching results to significantly improve the judgment accuracy of the model, especially in the ambiguous scenes of the transition area.

[0098] The time series change analysis of the feature points further reduces the judgment errors caused by instantaneous environmental changes. The feature point matching algorithm can adapt to the changes of image features during robot movement, which helps to maintain high-precision scene classification in complex dynamic environments. The multi-modal inference combined with the scene knowledge base enables the model to quickly classify and infer complex environments, reducing classification delay.

[0099] In the embodiment, the weighted fusion step of the extended Kalman filter includes:

[0100] ICP algorithm processing is performed on the laser radar point cloud data to generate a local environment map;

[0101] GPS data is used to provide global positioning coordinates;

[0102] IMU data is used to provide motion acceleration and angular velocity estimation;

[0103] The laser radar point cloud data, GPS data and IMU data are predicted and updated to dynamically adjust the weight of each data source, and a fused positioning result is generated.

[0104] The extended Kalman filter (EKF) in the embodiment is used to fuse multi-modal sensor data, and the specific steps include:

[0105] Prediction step: according to the acceleration and angular velocity data provided by the IMU, the position and attitude of the robot are predicted using the motion equation. The local map generated by the laser radar point cloud is used to predict environmental changes.

[0106] Update step: the GPS data is used as a global position reference to compare and update the position with the IMU prediction result. The visual feature data extracted by the visual encoder is matched with the point cloud in the local map to update the environment model.

[0107] Dynamic weight adjustment: the weight is adjusted according to the signal strength and reliability of each sensor data, for example:

[0108] When the GPS signal strength is weak, the weights of the laser radar and IMU are increased.

[0109] When the laser radar reflection signal is unstable, the weight of the visual feature is increased.

[0110] The final fusion result contains the current pose, global position and environment map of the robot.

[0111] The prediction-update mechanism of the extended Kalman filter enables the system to generate stable fusion results when sensor data is partially missing or abnormal. The dynamic adjustment of sensor weights ensures navigation accuracy in different environments, such as transition areas or signal interference areas. The EKF can process nonlinear sensor data (such as IMU acceleration data) and linear data (such as GPS position data), achieving efficient fusion of multi-modal data.

[0112] In one embodiment, the dual-mode parallel running mechanism includes:

[0113] In the transition area, activate the SLAM mode and the GPS mode, and perform weighted fusion on the output data of the two modes;

[0114] Dynamically monitor the stability of the lidar signal, the strength of the GPS signal and the IMU pose data;

[0115] According to the complexity of the environment, dynamically adjust the weights of SLAM and GPS, so that SLAM gradually weakens, GPS gradually strengthens, or SLAM gradually strengthens, and GPS gradually weakens;

[0116] After switching is completed, disable the navigation mode with lower weight.

[0117] The dual-mode parallel running of this embodiment effectively avoids navigation interruption caused by mode mutation, especially in transition areas with large signal fluctuations. The weight adjustment mechanism can dynamically adapt to different environmental characteristics, ensuring the applicability and accuracy of navigation switching. By fusing the advantages of SLAM and GPS modes, the robustness of the system in complex transition areas is enhanced.

[0118] In one embodiment, the scene analysis of the time sliding window includes:

[0119] Extract a set of feature points from consecutive frames of visual images;

[0120] Calculate the number of matches and the Euclidean distance of feature point changes for each frame of feature points;

[0121] Determine whether the scene transitions from indoor to outdoor or from outdoor to indoor according to the trend of multiple frames.

[0122] The time sliding window of this embodiment can effectively identify the gradual transition process of the scene from indoor to outdoor or from outdoor to indoor. The trend analysis method reduces the false positives caused by single frame image abnormalities, improving the stability of scene judgment.

[0123] In this embodiment, the parameters for calculating the confidence score include:

[0124] The confidence of the single-frame scene classification result, the multi-frame visual feature point change trend, the laser radar point cloud coverage range and quality, and the GPS signal strength and SLAM map construction quality.

[0125] Through multi-frame trend analysis and sensor fusion confidence calculation, the judgment deviation caused by single-frame abnormal data is reduced. In the indoor-outdoor transition area, the scoring calculation weight is dynamically adjusted, which can adapt to the signal fluctuation and feature change of complex environment. The comprehensive calculation method of confidence score provides a more reliable decision basis for navigation mode switching, avoiding frequent switching or delayed switching.

[0126] In the embodiment, the feedback mechanism includes:

[0127] Real-time monitoring of the effectiveness of the GPS signal;

[0128] Detecting the error level and coverage integrity of the SLAM map construction;

[0129] Quality assessment of the reflected signal of the laser radar point cloud data;

[0130] When a certain sensor data is unstable, the current navigation mode is maintained and other sensors are requested to provide supplementary data.

[0131] The feedback mechanism of the embodiment can monitor and respond to sensor abnormal conditions in real time, ensuring the continuity and stability of navigation. During the navigation mode switching process, the feedback mechanism reduces the positioning errors caused by single sensor abnormalities through supplementary data acquisition. By real-time monitoring and dynamic adjustment of sensor priority, the system can maintain high-precision positioning in complex environments.

[0132] In one embodiment, as shown in Figure 2 and Figure 3 The method of the seamless indoor-outdoor navigation switching scheme proposed in the embodiment includes the following steps:

[0133] Step 1: The robot uses the GPS combined with camera and IMU scheme for navigation in outdoor areas, and uses the laser SLAM and IMU fusion scheme in indoor areas.

[0134] Outdoor navigation is integrated with high-precision GPS modules on robots. Under high-precision requirements, RTK-GPS technology is combined to provide global position information through satellite signal triangulation. By combining RGB cameras, ORB-SLAM visual SLAM algorithm is used to extract key feature points in images and construct sparse three-dimensional point cloud maps. When GPS signals are weak or interfered, relative positioning is performed based on visual recognition features to ensure that robots can recognize landmarks such as roads and buildings in the environment, thereby maintaining navigation stability. IMU sensors, GPS, and camera data are fused through Kalman filters to ensure accurate and stable pose estimation during robot navigation. Through the joint work of RGB cameras and IMUs, VIO algorithm is used to achieve high-precision pose estimation, combining visual and inertial sensor data to maintain robot positioning accuracy when GPS fails. At the same time, an adaptive data fusion strategy is adopted: the system can dynamically adjust the weights of different sensors according to the strength of the GPS signal. When the GPS signal is good, the system relies on GPS for positioning; when the signal is weak or lost, visual and IMU sensors take over to ensure continuous positioning.

[0135] Indoor navigation uses Velodyne laser radar for 360-degree environment scanning to obtain obstacle distance information around the robot and construct a two-dimensional map of the environment. Cartographer is used for indoor SLAM mapping, and ICP algorithm is used to match consecutive scanned point clouds for accurate pose calculation. IMU plays a complementary role in laser SLAM, especially when the robot turns, moves quickly, or the laser radar is blocked. IMU corrects short-term pose errors in SLAM through inertial data. By fusing laser radar and IMU data, the robot can maintain higher mapping accuracy in complex dynamic indoor environments. Especially in cases where the laser radar is blocked or the reflected signal is unstable, IMU can continue to generate reliable pose information through inertial calculation, and dynamically adjust the sensor weights according to different motion states.

[0136] Step two: the camera continuously captures the environment images around the robot and extracts the key features of the scene through visual perception.

[0137] Real-time acquisition of environment images is achieved using a binocular camera. The camera is installed on the robot to obtain scene images at a fixed frame rate of 30 FPS. Each frame of image contains rich scene features such as walls, ceilings, floors, windows, trees, etc., which can provide a basis for subsequent scene judgment.

[0138] Assume there are two images I1 and I2, which contain extracted feature point sets F1 and F2 respectively, and each feature point fi has a corresponding descriptor vector d(fi). To perform feature point matching, we need to calculate the distance (Euclidean distance) between two descriptors:

[0139] dist(f i ,f j )=||d(f i )-d(f j )||

[0140] where d(f i ) and d(f j ) are the descriptor vectors of two feature points f i and f j , and ||·|| represents the Euclidean distance, which measures the similarity between two feature points.

[0141] Step three: Before performing large model inference, perform basic preprocessing on the input image data to minimize image noise and standardize it to meet the input requirements of the large model.

[0142] Image preprocessing is a necessary step before large model inference, ensuring that the large model can extract effective feature information from the image. Preprocessing includes removing image noise, adjusting brightness and contrast, and standardizing image data format to match the input requirements of the large model.

[0143] Noise removal: Use a two-dimensional Gaussian filter to smooth the image, reducing high-frequency noise interference while preserving important edge features in the image.

[0144] Brightness and contrast adjustment: By redistributing the pixel values of the image, the contrast between dark and bright parts of the image is improved, making the feature points in the image still clear and visible in high-contrast or low-contrast environments. By calculating the average brightness value of the image, the contrast is dynamically adjusted to ensure that visual features in the image remain consistent under different lighting conditions.

[0145] Image size adjustment and standardization: Use bilinear interpolation to adjust the resolution of the original image to the required input size (224x224 pixels) of the large model, ensuring that all input images have consistent sizes. And scale the pixel values of the image to the range [0, 1], according to the requirements of the large model, subtract the mean value and divide by the standard deviation for each channel, so that the data has zero mean and unit variance, which helps to eliminate differences between different data and improve the stability and robustness of the model output results.

[0146] Step four: Based on the multi-modal reasoning ability of the large language model, analyze the visual features in the image to determine whether the robot is currently in an indoor or outdoor environment.

[0147] The large model receives the pre-processed image data and analyzes the visual features contained in the image, such as the shape, color, edge, and texture information of objects. Through the visual encoder, it converts the pixel information of the image into a feature vector. By analyzing visual features such as walls, ceilings, windows, floors, trees, skies, and roads, it extracts key features related to indoor and outdoor environments. The multi-modal cross-attention mechanism combines visual features with semantic knowledge learned by the model during pre-training. Based on the results of multi-modal reasoning, the model outputs a classification result, i.e., whether the robot is currently indoors or outdoors. For example, if the image contains "ceiling, window, light", the model tends to judge as indoor environment. If the image appears "sky, tree, road", the model tends to judge as outdoor environment.

[0148] Step five: The system will comprehensively determine the location based on the trend of multi-frame image changes. At the same time, the system will calculate the confidence of each judgment result of the model, which is used to determine whether to switch the navigation strategy to avoid frequent switching caused by single-frame misjudgment.

[0149] The system collects multiple consecutive frames of images from the camera and passes these data to the large model. After analyzing the visual features of each frame, it will make comprehensive reasoning based on the trend of these features. Compared with single-frame image judgment, continuous frame processing enables the model to capture long-term stable changes in the scene, ensuring the accuracy of the judgment. For example, as the robot approaches the window, the window, sunlight, and other features in the image gradually appear, and the model can gradually confirm that the scene is changing from indoor to outdoor. This multi-frame image-based reasoning approach avoids misjudgment caused by single-frame feature ambiguity or specific environmental interference (sudden changes in light, etc.).

[0150] After judging the scene, the system will calculate a confidence score based on the degree of matching of the current visual features by the large model. If the visual feature trend of consecutive frames is consistent, such as gradually identifying outdoor features such as trees, sky, and road, the confidence will be higher, and the model will be more confident that it is currently in an outdoor environment; if the features in the image are unstable or ambiguous (such as a frame with a large change in light), the confidence will be lower, and the model may choose to maintain the current navigation mode and wait for more data to determine whether to switch. This strategy effectively prevents the robot from unnecessarily switching navigation modes in the scene transition area (the doorway between indoor and outdoor movement).

[0151] In addition to real-time scene judgment, the system also captures scene transitions by analyzing image trends over a longer period of time. For example, when a robot moves from indoors to outdoors, the visual features gradually transition from indoor ceilings and walls to outdoor sky, ground, trees, etc. The system can recognize these gradual scene changes and adjust the confidence threshold as needed, making the switch more flexible. The system can dynamically adjust the confidence threshold based on the complexity of the scene, lowering the threshold to speed up the switch in complex environments, and raising the threshold to ensure stable judgment in stable environments.

[0152] Step six: According to the judgment result of the large model (indoor or outdoor), the system switches the corresponding navigation strategy to ensure that the robot uses the optimal navigation mode in different environments.

[0153] The system receives the scene judgment result and its confidence score from the large model, and receives the scene judgment result from the large model in real time through the ROS topic. According to the preset confidence threshold, it is determined whether to switch the navigation strategy according to the judgment result of the large model. If it is judged as "indoor" and the confidence is high, the SLAM fusion navigation strategy is started; if it is judged as "outdoor" and the confidence is high, the navigation strategy of GPS combined with vision is switched.

[0154] To ensure the smoothness when switching from one navigation mode to another, a smooth switching mechanism is designed to avoid sudden operations that cause navigation to be interrupted. When the large model judges to switch from indoors to outdoors, the system keeps SLAM and GPS dual-mode running in parallel for a short time. The SLAM mode gradually weakens its control, while the GPS mode gradually takes over the robot navigation. During the transition period, the system fuses SLAM and GPS data through a weighted approach, gradually shifting from relying on lidar to relying on GPS data.

[0155] Step seven: In the indoor-outdoor transition area, the system uses gradual scene judgment and mode switching to ensure smooth transition of the robot in complex environments.

[0156] To avoid frequent switching of navigation strategies in the transition area, the system uses gradual scene judgment, combining the trend of changes in multiple frames of images and sensor data to gradually switch navigation modes.

[0157] In the indoor-outdoor transition area, the scene features change rapidly, and the system analyzes the feature trend of multiple frames of images through a time sliding window to avoid misjudgment caused by single-frame judgment. In this area, the confidence threshold is dynamically adjusted according to the complexity of the environment, allowing the confidence to trigger the switching of navigation modes even in a relatively low situation. Avoid the problem of frequent switching caused by chaotic scene features in the transition area.

[0158] The system can combine the data provided by multiple sensors (laser radar, camera, IMU, GPS) during switching, gradually switching scenes through sensor data weighted fusion. When the robot moves from indoors to outdoors, the GPS, IMU and laser radar data are weighted and fused using an extended Kalman filter (EKF), and the local map is still relied on SLAM to build when just entering the transition area, but gradually increases the weight of GPS data, and finally completely controls navigation by GPS.

[0159] The system ensures the stability of the navigation mode during switching by monitoring sensor states (GPS signal strength, SLAM map quality, etc.). During the entire switching process, the system will feedback information such as signal strength, sensor state, etc. through the user interface or internal log, and if a sensor state is unstable during switching (such as weak GPS signal or large SLAM map error), the system will delay switching or fall back to the original mode to ensure that the robot can continue to navigate smoothly.

[0160] Step eight: In order to enhance the robustness of the system in complex environments, multiple modal data fusion is used to combine the data of multiple sensors such as cameras, laser radars, GPS, IMUs, etc. for joint judgment and positioning. Multi-modal fusion can achieve more stable and continuous positioning and environment perception through the complementarity of different sensors, especially in complex indoor and outdoor environments or transition areas.

[0161] Camera (RGB camera): provides visual features of the scene for feature recognition, object detection and large model input. Laser radar (Velodyne laser radar): generates a two-dimensional point cloud map of the surrounding environment through laser scanning, which is used to build an accurate map and is suitable for obstacle detection and obstacle avoidance in indoor environments. GPS module: provides global positioning information for the robot in outdoor environments, which can be used for path planning and positioning in a large outdoor area. IMU: obtains acceleration and angular velocity data of the robot through an accelerometer and a gyroscope, and provides short-term pose estimation when sensor signals are weak.

[0162] This embodiment solves the defects of single sensor in complex environment, such as easy to be disturbed and inaccurate positioning, by fusing the data of multiple sensors (camera, laser radar, GPS, IMU). Especially in the indoor and outdoor transition area, the invention uses gradual scene judgment and realizes the smooth transition of the robot in the environment with drastic changes through the cooperative processing of multi-modal data.

[0163] The system can automatically switch the navigation strategy according to the different environments (indoor or outdoor), not only relying on large models for scene judgment, but also combining the trend of multiple image changes to gradually switch the navigation mode, avoiding the problem of frequent switching of navigation mode at the scene boundary, and improving the intelligent performance of the robot in complex environments.

[0164] In this embodiment, a confidence evaluation mechanism based on the judgment result of the large model scene is designed, which can dynamically adjust the accuracy of scene judgment and the trigger condition of navigation strategy switching. By setting the confidence threshold, the system instability caused by single-frame judgment error is avoided, and the self-adaptive ability of the system is enhanced, making the system more stable in scene recognition and navigation mode switching, especially in the complex boundary area of environmental change, realizing the gradual processing of the scene transition area.

[0165] When the robot enters the indoor-outdoor transition area, the invention ensures smooth transition of indoor navigation and outdoor navigation through the way of dual-mode parallel running. The dual-mode is activated at the same time in a short time, and through the weighted fusion technology, the weight of one mode is gradually reduced and the weight of the other mode is increased, so as to realize seamless switching.

[0166] Compared with the traditional method of switching depending on the detection of indoor and outdoor signal strength, the invention adopts scene understanding and judgment technology based on large language model, intelligently judges whether the robot is currently in indoor or outdoor environment through visual perception, image feature extraction and multi-frame analysis, and adopts the method of dual-mode parallel transition based on gradual scene judgment and multi-frame image trend analysis, avoiding frequent error switching caused by environmental interference, single-frame image misjudgment and instantaneous signal fluctuation, ensuring smooth switching and continuous navigation of the robot in the transition area, and significantly improving the anti-interference ability and scene understanding ability of the system.

[0167] With the development of robot technology, robots are increasingly widely used in indoor and outdoor navigation, especially in logistics, security, inspection and other scenarios. Robots need to be able to move autonomously and perform tasks in indoor and outdoor environments. However, the difference between indoor and outdoor environments and the complexity of the transition area pose great challenges to robot navigation. Existing robot navigation technology usually relies on switching indoor and outdoor navigation strategies through signal strength changes, which has obvious limitations.

[0168] Currently, most mobile robots rely on the change of GPS signal strength or indoor Bluetooth beacon signal strength to determine the current environment when switching indoor and outdoor navigation strategies. However, signal strength is easily disturbed in complex environments in the transition area, and the environment in the transition area is complex, with obvious signal strength fluctuations. In areas where the outdoor signal is weak (such as high-rise buildings, tunnels, or densely wooded areas), the robot may misjudge that it has entered the indoor environment, thus switching to an inappropriate navigation mode, resulting in positioning errors; while near the window or doorway in the indoor environment, signal reflection or attenuation may cause the system to misjudge that it is outdoor, thus frequently switching navigation modes, causing unstable navigation. The existing method of switching navigation modes based on signal strength often performs poorly in the transition area, making it difficult to achieve smooth transition and affecting the continuity of the robot's task and the accuracy of navigation. Therefore, how to combine multi-modal data, scene feature understanding, dynamic confidence evaluation, and other methods to design a more intelligent and more stable navigation strategy switching system has become a technical problem that needs to be solved in current robot navigation technology.

[0169] The present embodiment is based on these background problems and proposes a new indoor and outdoor navigation mode switching method based on multi-modal data fusion and large model scene judgment. Through multi-sensor fusion, scene intelligent recognition, and gradual switching mechanism, the navigation accuracy and stability of the robot in complex environments are improved, overcoming the shortcomings of the prior art.

[0170] In one embodiment, a robot indoor and outdoor navigation switching system based on a large language model is provided, comprising:

[0171] A visual processing module is used to collect environmental data using multi-modal sensors, including visual images, laser radar point clouds, GPS signal data, and IMU data.

[0172] An image processing module is used to remove noise, adjust brightness, and perform standardization preprocessing on the visual image data, and adjust the image resolution to a specified size.

[0173] A large language model module is used to input the preprocessed visual image into a large language model to generate a scene classification result through a feature extraction and scene reasoning module. Based on a time sliding window, the trend of feature points in multiple frames of visual images is analyzed, and the confidence score of scene classification is dynamically adjusted.

[0174] A data fusion module is used to combine visual features, point cloud data, GPS signal data, and IMU data in the indoor and outdoor transition area, and use an extended Kalman filter for weighted fusion to generate a comprehensive positioning result.

[0175] The navigation switching module is configured to select a navigation mode according to the scene classification result and the integrated positioning data, wherein the indoor navigation mode is based on laser SLAM to construct an environment map, and the outdoor navigation mode is combined with GPS, vision and IMU to realize positioning.

[0176] The feedback mechanism module is configured to dynamically adjust the weights of the SLAM and GPS modes through a dual-mode parallel running mechanism during the navigation mode switching.

[0177] The specific implementation of each module can be referred to the above description of the method for switching indoor and outdoor navigation of a robot based on a large language model, and will not be described here.

[0178] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict, they should be considered as the scope of the present disclosure.

Claims

1. A robot indoor and outdoor navigation switching method based on a large language model, characterized in that: The method comprises: Collecting environmental data using multimodal sensors, including visual images, lidar point clouds, GPS signal data, and IMU data; Performing noise removal, brightness adjustment, and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size; Input the preprocessed visual image into the large language model, and generate scene classification results through feature extraction and scene reasoning modules; Based on a time sliding window, it analyzes the changing trends of feature points in multiple frames of visual images and dynamically adjusts the confidence score of scene classification. In the transition area between indoor and outdoor, the extended Kalman filter is used to perform weighted fusion by combining visual features, point cloud data, GPS signal data and IMU data to generate comprehensive positioning results. According to the scene classification results and comprehensive positioning data, the navigation mode is selected, among which: the indoor navigation mode builds the environment map based on laser SLAM; the outdoor navigation mode combines GPS, vision and IMU to achieve positioning; During navigation mode switching, the weights of SLAM and GPS modes are dynamically adjusted through the dual-mode parallel operation mechanism.

2. The navigation switching method according to claim 1, characterized in that: The multimodal sensor comprises: Camera, LiDAR, GPS module and IMU module. The camera is used to capture visual image data for scene feature extraction; the LiDAR is used to generate two-dimensional or three-dimensional point cloud data for obstacle detection and local map construction; the GPS module is used to provide global positioning data for path planning and positioning; the IMU module is used to provide short-term pose estimation through the accelerometer and gyroscope.

3. The navigation switching method according to claim 1, wherein: The preprocessing of the visual image includes: Use a two-dimensional Gaussian filter to perform image denoising and remove high-frequency noise; Dynamically adjust the distribution of pixel values ​​to optimize image brightness and contrast; The image resolution was adjusted to 224 × 224 pixels using bilinear interpolation; Normalization is done by subtracting the mean from the pixel value of each channel and dividing by the standard deviation.

4. The navigation switching method according to claim 1, wherein: The scene classification process of the large language model includes: Use visual encoders to extract scene features of walls, ceilings, windows, trees, and roads in images; Match the feature points extracted from consecutive frames through feature matching algorithm; Generate time series change data of feature points based on matching results; Use the multimodal reasoning module in the large language model combined with the scene knowledge base to complete scene classification.

5. The navigation switching method according to claim 1, wherein: The weighted fusion step of the extended Kalman filter includes: Use ICP algorithm to process lidar point cloud data to generate a local environment map; Use GPS data to provide global positioning coordinates; Provide motion acceleration and angular velocity estimation through IMU data; The lidar point cloud data, GPS data and IMU data are predicted and updated, and the weight of each data source is dynamically adjusted to generate a fused positioning result.

6. The navigation switching method according to claim 1, wherein: The dual-mode parallel operation mechanism includes: In the transition area, the SLAM mode and the GPS mode are activated, and the output data of the two modes are weightedly fused; Dynamically monitor lidar signal stability, GPS signal strength, and IMU pose data; Dynamically adjust the weights of SLAM and GPS according to the complexity of the environment, so that SLAM gradually weakens and GPS gradually strengthens, or so that SLAM gradually strengthens and GPS gradually weakens; After the switch is complete, disable the lower-weight navigation mode.

7. The navigation switching method according to claim 1, wherein: The scenario analysis of the time sliding window includes: Extracting feature point sets from continuous frame visual images; Calculate the matching number of feature points in each frame and the Euclidean distance of feature point changes; Determine whether the scene transitions from indoor to outdoor or from outdoor to indoor based on the multi-frame change trend.

8. The navigation switching method according to claim 1, wherein: The parameters for the confidence score calculation include: The confidence of single-frame scene classification results, the change trend of multi-frame visual feature points, the coverage and quality of lidar point cloud, and the GPS signal strength and SLAM map construction quality.

9. The navigation switching method according to claim 1, wherein: The method further comprises: a feedback mechanism: Real-time monitoring of the effectiveness of GPS signals; Detect the error level and coverage completeness of SLAM map construction; Perform quality assessment of reflected signals from lidar point cloud data; When the data from a certain sensor is unstable, the current navigation mode is maintained and other sensors are requested to provide supplementary data.

10. A robot indoor and outdoor navigation switching system based on a large language model, characterized in that: include: Visual processing module: used to collect environmental data using multimodal sensors, including visual images, lidar point clouds, GPS signal data, and IMU data; Image processing module: used for performing noise removal, brightness adjustment and standardization preprocessing on the visual image data, and adjusting the image resolution to a specified size; Large language model module: used to input preprocessed visual images into the large language model and generate scene classification results through feature extraction and scene reasoning modules; Based on a time sliding window, it analyzes the changing trends of feature points in multiple frames of visual images and dynamically adjusts the confidence score of scene classification. Data fusion module: used to combine visual features, point cloud data, GPS signal data and IMU data in the transition area between indoor and outdoor, and use extended Kalman filter for weighted fusion to generate comprehensive positioning results; Navigation switching module: used to select navigation mode based on scene classification results and comprehensive positioning data. Indoor navigation mode builds environment map based on laser SLAM; outdoor navigation mode combines GPS, vision and IMU to achieve positioning; Feedback mechanism module: used to dynamically adjust the weights of SLAM and GPS modes through a dual-mode parallel operation mechanism during navigation mode switching.

Images