LLM enhancement-based SLAM loop detection method and application

Abstract

The application belongs to the technical field of loop detection, and particularly relates to a SLAM loop detection method based on LLM enhancement and application. The detection method comprises the following steps: 1, after obtaining multi-source perception data and original spatial coordinates of a current time position point, suspicious historical position points are screened out; if there is more than one suspicious historical position point, step 2 is executed; 2, after the multi-source perception data of the current time position point is preprocessed, a fusion feature matrix is obtained, and then a LLM model is used to obtain semantic description text and semantic embedding vectors of the current time position point based on the fusion feature matrix of the current time; 3, based on the fusion feature matrix, the semantic description text and the semantic embedding vectors, three-way loop determination is performed between the current time position point and the suspicious historical position points, and it is confirmed whether the current time position point and a certain suspicious historical position point constitute a loop. The application can efficiently, accurately and stably perform loop detection.

Description

Technical Field

[0001] This application belongs to the field of loop closure detection technology, and in particular relates to a SLAM loop closure detection method and its application based on LLM enhancement. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology for mobile robots and autonomous driving devices to achieve autonomous navigation. It collects environmental data through multiple sensors and completes self-localization and real-time construction of an environmental map in unknown environments. Loop closure detection, as a key component of the SLAM system, is used to detect whether the carrier (usually a robot) has revisited a previously visited location. By introducing loop closure constraints, it eliminates accumulated localization errors and ensures global map consistency and localization accuracy.

[0003] Existing SLAM loop closure detection methods only support feature matching from a single sensor, such as vision or LiDAR, or use fixed weights for simple feature stitching. While this reduces the computational load during detection, the accuracy of loop closure detection is low. Furthermore, these methods cannot adapt to complex environments with missing sensor data or feature failures, resulting in poor environmental adaptability and extremely unstable accuracy of loop closure detection methods.

[0004] This application content

[0005] The purpose of this application is to overcome the shortcomings of the prior art and provide a SLAM loop closure detection method based on LLM enhancement, which can perform loop closure detection efficiently, accurately and stably.

[0006] To achieve the above objectives, this application adopts the following technical solution: A SLAM loop closure detection method based on LLM enhancement includes the following steps: Step 1: After obtaining the multi-source sensing data and original spatial coordinates of the current location point, filter suspicious historical location points in the first database; if there is more than one suspicious historical location point in the first database, proceed to Step 2. Multi-source sensing data includes visual image data, lidar point cloud data, and IMU inertial measurement data; historical location points whose spatial coordinates are offset from the original location coordinates of the current location point by a distance less than a first offset threshold are recorded as suspicious historical location points. Step 2: After preprocessing the multi-source sensing data of the current location point, a fusion feature matrix is ​​obtained. Then, the LLM model is used to obtain the semantic description text and semantic embedding vector of the current location point based on the fusion feature matrix of the current time. Step 3: Based on the fused feature matrix, semantic description text, and semantic embedding vector, perform a triple loop closure judgment on the current location point and the suspicious historical location point to confirm whether the current location point and a certain suspicious historical location point form a loop. Step 3 also includes the following sub-steps: Step 31: Based on the fused feature matrix and semantic embedding vector, perform the first loop recurrence determination between the current location point and the suspicious historical location points to obtain the initial screening of historical location points; if there is more than one initial screening of historical location points, proceed to step 32. Step 32: Calculate the comprehensive matching score to perform a second loop retracement judgment on the initial screened historical location points in order to filter out potential historical location points; if there is more than one potential historical location point, proceed to step 33. Step 33: Use the RANSAC algorithm to perform a third loop closure determination on potential historical location points to filter out historical loop closure points; the current location point and one historical loop closure point form a loop.

[0007] Preferably, step 2 further includes the following sub-steps: Step 21: After preprocessing and extracting feature vectors from the visual image data collected by all visual sensors, the point cloud data collected by the lidar, and the IMU inertial measurement data collected by the IMU inertial sensor at the current time t, the corresponding visual feature vectors, geometric feature vectors, and inertial feature vectors are obtained respectively. Constructing the multi-source sensing matrix at time t: The matrix formed by the visual feature vectors of all visual sensors corresponding to time t is denoted as the visual feature matrix F. v,t The matrix formed by the geometric eigenvectors of all lidars at the current time t is denoted as the geometric eigenma matrix F. u,t The matrix formed by the inertial eigenvectors of all IMU inertial sensors at the current time t is denoted as the inertial eigenma matrix F. i,t ; Step 22: Based on the multi-source sensing matrices of various types and their corresponding effectiveness coefficients at the current time t, calculate the fusion feature matrix F at the current time. t : F t =ω v,t ·σ(F v,t )+ω u,t ·σ(F l,t )+ω i,t ·σ(F i,t );ω v,t +ω u,t +ω i,t =1; Where, ω v,t ωu,t and ω i,t Let represent the effectiveness coefficients of the visual feature matrix, geometric feature matrix, and inertial feature matrix at the current time t, respectively; σ(·) represents the L2 normalization function. Step 23, use the LLM model based on the fusion feature matrix F at the current time t. t Obtain the semantic description text T at the current time t. t Then, the semantic description text T t After L2 normalization, the semantic embedding vector S is obtained. t : ;in, This represents the text embedding layer mapping function within the LLM model. This represents the L2 norm of a vector.

[0008] Preferably, step 22 further includes the following: monitoring the working status of each type of sensor; if the working status of each type of sensor at the current time t has not changed compared to the previous time t-1, then the validity coefficient of each type of multi-source sensing matrix at the next time t+1 remains consistent with the current time t; the sensor working status includes two types: normal and abnormal.

[0009] Preferably, if the sensor is in working state at the current time t, and this state has changed compared to the previous time t-1, then obtaining the effectiveness coefficients of each type of multi-source sensing matrix at the next time t+1 further includes the following steps: Step 001: Obtain the percentage of visual sensors, LiDAR, and IMU inertial sensors that are operating normally at the current time t, respectively. v,t FW u,t and FW i,t ; Step 002, FW v,t FW u,t and FW i,t After L1 normalization, they are denoted as FW. v,t ´、FW u,t ´ and FW i,t Then calculate the intermediate coefficients ω of the visual feature matrix at the next time step t+1. v,t+1 ´、Intermediate coefficients ω of the geometric characteristic matrix u,t+1 ´ and the intermediate coefficients ω of the inertial characteristic matrix i,t+1 ´: ω v,t+1 =FW v,t ·ω v,t + FW v,t ·FW v,t ´·ω v,t-1 ; ω u,t+1 =FW u,t ·ω u,t +FW u,t ·FW u,t ´·ω u,t-1 ; ω i,t+1 =FW i,t ·ω i,t +FW u,t ·FW i,t ´·ω i,t-1 ; Step 003, the intermediate coefficients ω of the visual feature matrix at the next time step t+1 are... v,t+1 ´、Intermediate coefficients ω of the geometric characteristic matrix u,t+1 ´ and the intermediate coefficients ω of the inertial characteristic matrix i,t+1 After L1 normalization, the effectiveness coefficients ω of the visual feature matrix at the next time step t+1 are obtained. v,t+1 ω, the efficiency coefficient of the geometric feature matrix u,t+1 and the efficiency coefficient ω of the inertial characteristic matrix i,t+1 .

[0010] Preferably, step 31 further includes the following: Step 311, calculate the semantic embedding vector S of the current position point. t Cosine similarity between the semantic embedding vectors of suspicious historical locations and the cosine similarity of suspicious historical locations is retained, and suspicious historical locations with a cosine similarity above the second threshold are retained. Step 312: Obtain the multi-source perception matrix of suspicious historical location points; retain the historical location points that satisfy the pose constraints between the multi-source perception matrix and the current location point multi-source perception matrix, and record them as the initial screening historical location points; Step 32 also includes the following: Based on the fused feature matrix, semantic description text, and semantic embedding vector, calculate the comprehensive matching score between the current location and each initially screened historical location; then output the initially screened historical location points whose comprehensive matching score is greater than a fourth set threshold as potential historical location points; the formula for calculating the comprehensive matching score is as follows: Score(L t ,L H )=α t sim f (F t ,F H )+(1-α t ) sim m (L t ,L H ); simm (L t ,L H )=γ sim s (S t ,S H )+(1-γ) sim T (T t ,T H ); Among them, L t L represents the current position at time t; H This represents a current initial screening historical position; Score(L) t ,L H ) represents the location point L t Compared with the historical location point L of the initial screening H The overall matching score between them; α t The sim represents the environmental coefficient at the current time t. m (L t ,L H ) represents L t With L H The original semantic similarity between them; γ represents the semantic internal fusion weight coefficient, γ∈[0,1]; T H Indicates the initial screening historical location point L H The corresponding semantic description text; S H Represents the semantic description text T H The corresponding semantic embedding vector; sim s (S t ,S H ) represents the semantic embedding vector S t With S H Semantic similarity between them; sim T (T t ,T H ) represents the semantic description text T t With T H The semantic relevance between the texts; F H Indicates the initial screening historical location point L H The fusion feature matrix; sim f (F t ,F H ) represents the fusion feature matrix F t With F H Cosine similarity between them; Step 33 also includes the following sub-steps: Step 331: Extract feature points from the visually corrected image and use the RANSAC algorithm to obtain the visual essence matrix between the current position point and the potential historical position point. Step 332: Calculate the proportion of matching points that satisfy the corresponding visual essence matrix to the total number of feature points extracted from the corresponding visual correction image, and record it as the inlier rate between the current position point and the corresponding potential historical position point. Step 333: If there is only one historical position point and the current position point with an in-point ratio greater than the fifth set threshold, then the corresponding historical position point is a historical loop point; if there are two or more historical position points and the current position point with an in-point ratio greater than the fifth set threshold, then after sorting the in-point ratios in descending order, the historical position point corresponding to the first in-point ratio is determined to be a historical loop point.

[0011] Preferably, if there are two or more historical location points with an in-point rate greater than the fifth set threshold, the in-point rates are sorted in descending order: if the difference between the in-point rate of the first position and the in-point rate of the second position is greater than the safety redundancy difference, the historical location point corresponding to the first in-point rate is determined to be a historical loop point; if the difference between the in-point rate of the first position and the in-point rate of the second position is below the safety redundancy difference, cross-modal secondary verification is performed.

[0012] Preferably, cross-modal secondary verification includes the following: obtaining the original semantic similarity (sim) between the historical position points and the current position point of all inlier rates corresponding to the difference between the current and first inlier rates (below the safety redundancy difference). m (L t ,L H Then sort them in descending order. If the original semantic similarity of the first position is the same as the historical position corresponding to the first inlier rate, then the historical position corresponding to the first inlier rate is determined to be a historical loop point. If the original semantic similarity of the first position is different from the historical position corresponding to the first inlier rate, and the difference between the original semantic similarity of the historical position corresponding to the first position and the original semantic similarity of the historical position corresponding to the first inlier rate is greater than the semantic confidence threshold ΔS, then the historical position corresponding to the original semantic similarity of the first position is determined to be a historical loop point.

[0013] Preferably, in step 33: if there is no inlier rate greater than the fifth set threshold, or if the historical position point corresponding to the first semantic original similarity in the cross-modal secondary verification is below the semantic confidence threshold ΔS, then it is determined that there is no historical loop point in step 33, and an alarm is triggered to the technical personnel. If there are no initial screening historical position points in step 31; or no potential historical position points in step 32; or no historical loop points in step 33, then it is determined that the current position point does not form a loop with any historical position point, and then the process returns to step 1 at the next moment.

[0014] This application also provides an application of an LLM-enhanced SLAM loop closure detection method, including the following: using the LLM-enhanced SLAM loop closure detection method described above, it is determined in real time whether the robot's current position point forms a loop with a certain historical position point. If a loop is formed, the loop closure constraint information is obtained to optimize the robot's positioning drift error and trajectory optimization.

[0015] The beneficial effects of this application are as follows: (1) The loop closure detection method of this application integrates multiple sensor features to obtain a fusion feature matrix of the location point, and uses an LLM model to further obtain semantic description text and semantic embedding vector on the basis of the fusion feature matrix. Finally, based on the fusion feature matrix, semantic description text and semantic embedding vector, a triple loop closure judgment is performed between the current location point and the historical location point to efficiently and accurately determine whether the current location point forms a loop with a certain historical location point. At the same time, the change of the working state of the sensor will automatically cause the change of the validity coefficient and environmental coefficient of this application. This change will automatically adjust the application ratio of different types of multi-source sensing data as the basis for loop closure detection, and improve the stability of the judgment result of this application method.

[0016] (2) The loop closure detection method of this application utilizes multi-source sensing data collected by various types of sensors. Compared with the prior art, this makes the current position data available in the subsequent loop closure detection process more comprehensive, which naturally helps to improve the accuracy of loop closure detection.

[0017] (3) Before performing triple loop closure determination, the loop closure detection method of this application first filters out suspicious historical position points from all historical position points in the first database: This application sets a first offset threshold and uses the offset distance, which has a very small computational cost, to filter out a portion of suspicious historical position points from all historical position points in the first database (historical position points whose spatial coordinates are offset from the original position coordinates of the current position point by more than the first offset threshold cannot form a loop with the current position point). This is the initial screening of historical position points in this application, which also reduces the computational burden for all subsequent steps. If there are no suspicious historical position points in the first database, it can be directly determined that the current position point does not form a loop with any historical position point, so there is no need to calculate the fusion feature matrix of the current position point, let alone perform triple loop closure determination. Therefore, each step of this application will complete the screening of historical position points with the minimum computational cost.

[0018] (4) The fusion feature matrix serves as an important basis for subsequent triple loop closure determination. While utilizing various types of sensor data to improve the accuracy of loop closure determination, the effectiveness coefficients of different types of feature matrices at the next moment are dynamically adjusted, as well as the environmental coefficients in triple loop closure determination are dynamically adjusted. This reduces the interference introduced by sensor data loss and feature failure scenarios in complex environments on the accuracy of loop closure determination, improves the robustness and versatility of the detection method in this application, and realizes adaptive loop closure detection in different environments.

[0019] (5) In this application, the current location point can only be determined to form a loop with a historical location point after a triple loop closure determination based on the fused feature matrix, semantic description text, and semantic embedding vector. In the triple loop closure determination, each loop closure determination is a further screening of historical location points. The number of historical location points retained after each loop closure determination will decrease. Therefore, even though the computational cost of the RANSAC algorithm in the third loop closure determination is significantly larger than that of the first two, the number of historical location points that need to be determined in the third loop closure determination is very small. Therefore, the overall computational cost of loop closure detection in this application at each time point will not be too large, and the overall computational time will not be too long, so it can still ensure efficient loop closure detection. That is, the loop closure detection method of this application significantly reduces the computational complexity while ensuring the rigor and accuracy of detection, greatly improves the real-time performance of the detection results, and meets the real-time requirements of highly dynamic navigation scenarios.

[0020] (6) In this application, the semantic description text and semantic embedding vector of the current location point are obtained by using the LLM model in order to balance the adverse effects of sensor data quality changes on the accuracy of the detection method in scenarios where sensor data is missing or environmental features are weak.

[0021] (7) In the third loop closure determination, this application creatively introduces cross-modal secondary verification, that is, backtracking comparison through the original semantic similarity, which can effectively avoid misjudgment caused by relying solely on the highest inlier rate due to the presence of highly repetitive physical geometric textures in the scene (such as the same load-bearing columns and symmetrical shelves). When the inlier rates are very close, the global judgment of high-dimensional semantics is given priority, breaking through the bottleneck of traditional pure geometric matching. Attached Figure Description

[0022] Figure 1 This is a flowchart of a SLAM loop closure detection method based on LLM enhancement according to this application. Detailed Implementation

[0023] To make the technical solution of this application clearer and more explicit, the application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Solutions derived by those skilled in the art through equivalent substitution and conventional reasoning of the technical features of the technical solution of this application without creative effort all fall within the protection scope of this application.

[0024] like Figure 1 As shown, a flowchart of a SLAM loop closure detection method based on LLM enhancement of this application is presented, including: Step 1: After obtaining the multi-source sensing data and original spatial coordinates of the current location point, filter suspicious historical location points in the first database; if there is more than one suspicious historical location point in the first database, proceed to Step 2. Multi-source sensing data includes visual image data, lidar point cloud data, and IMU inertial measurement data; historical location points whose spatial coordinates are offset from the original location coordinates of the current location point by a distance less than a first offset threshold are recorded as suspicious historical location points.

[0025] Step 2: After preprocessing the multi-source sensing data of the current location point, a fusion feature matrix is ​​obtained. Then, the LLM model is used to obtain the semantic description text and semantic embedding vector of the current location point based on the fusion feature matrix of the current time.

[0026] Step 3: Based on the fusion feature matrix, semantic description text, and semantic embedding vector, perform a triple loop closure judgment on the current location point and the suspicious historical location point to confirm whether the current location point and a certain suspicious historical location point form a loop.

[0027] Optionally, in step 1, if there are no suspicious historical location points in the first database, it is determined that the current location point does not form a loop with any historical location point, and then the process returns to step 1 at the next moment.

[0028] Optionally, for current location points determined not to form a loop with any historical location points, their multi-source sensing data and original spatial coordinates are packaged into spatial information and stored in a second database. Technicians then extract and process the spatial information in the second database according to their work requirements.

[0029] Whether the historical location points corresponding to the spatial information in the second database can be put into the first database is determined by the technical staff based on the work requirements.

[0030] In step 1: Assuming the original spatial coordinates of the current location are (x1, y1, z1), and the spatial coordinates of a historical location are (x2, y2, z2), then the offset distance d between the original spatial coordinates of the current location and the spatial coordinates of that historical location is: .

[0031] The loop closure detection method of this application is usually applied to hardware devices such as sweeping robots or inspection robots. Although the positioning of the hardware device is periodically calibrated, it is not a real-time positioning calibration, so coordinate drift is unavoidable.

[0032] In step 1, it is precisely because the original spatial coordinates of the current position may have undergone coordinate drift that even if the original spatial coordinates of the current position are the same as the spatial coordinates of one (or more) historical positions, it cannot be directly determined that the original spatial coordinates of the current position form a loop with the spatial coordinates of one (or more) historical positions. Therefore, once the original spatial coordinates of the current position are the same as the spatial coordinates of one (or more) historical positions, the original spatial coordinates of the current position in this case must undergo a subsequent triple loop determination to determine whether they truly form a loop with one (or more) historical coordinates.

[0033] On the other hand, if the original spatial coordinates at the current moment are different from the spatial coordinates of any historical location, one possibility is that no coordinate drift has occurred at the current moment, and the original spatial coordinates at the current moment are the true spatial coordinates at the current moment. In this case, the true spatial coordinates at the current moment do not form a loop with any historical spatial coordinates. Another possibility is that coordinate drift has occurred at the current moment, and the original spatial coordinates at the current moment are not the true spatial coordinates at the current moment. This manifests as "the true spatial coordinates at the current moment may still form a loop with the spatial coordinates of one (or more) historical location points, but there is an offset distance between the original spatial coordinates at the current moment and the corresponding one (or more) historical location points." Since the degree of coordinate drift is limited, step 1 limits the subsequent triple loop determination of the original spatial coordinates at the current moment only when the offset distance is below the first offset threshold. Otherwise, it is directly determined that the original spatial coordinates at the current moment do not form a loop with any historical spatial coordinates.

[0034] Starting from step 1, this application performs loop closure detection with minimal computational cost, thereby improving the efficiency of loop closure detection and reducing the consumption of computational resources.

[0035] In step 1, the first offset threshold is preset by the technician based on experience.

[0036] In step 1, technicians can add or remove historical location points from the first data. In this embodiment, real spatial coordinates that have appeared in the past year are used as historical spatial coordinates; at the same time, real coordinates that have appeared more than one year but less than three years ago and whose frequency of occurrence is higher than a set threshold are also used as historical spatial coordinates.

[0037] In step 1, a linear interpolation method based on IMU timestamps is used to unify the time scale among visual image data, LiDAR point cloud data, and IMU inertial measurement data, ensuring that multi-source sensing data can be obtained at the same time. Since the sampling frequency of IMU data is usually much higher than that of visual images or LiDAR point clouds, IMU pre-integration technology or interpolation algorithms are used during timestamp alignment to integrate and aggregate multiple high-frequency IMU data between two frames of visual / LiDAR data, strictly aligning them with the low-frequency visual / LiDAR data at the same timestamp, thus providing the multi-source sensing data output corresponding to the current time.

[0038] Step 2 also includes the following sub-steps: Step 21: After performing the following preprocessing on the multi-source sensing data at the current moment, the multi-source sensing matrix is ​​obtained: For the visual image data acquired by a visual sensor at time t: first, distortion correction is performed to eliminate the physical geometric distortion of the lens; then, grayscale and edge enhancement processing is performed to convert the color image into a single-channel image and highlight the environmental contours to reduce the computational cost of subsequent feature extraction, finally outputting a visually corrected image; then, a deep learning global feature extraction algorithm (such as the NetVLAD algorithm) is used to extract the visual feature vector corresponding to the visual sensor at time t, and the matrix formed by the visual feature vectors of all visual sensors corresponding to time t is denoted as the visual feature matrix F. v,t ; Simultaneously, the lidar point cloud data at the current time t is first denoised using a statistical filtering algorithm to remove outliers; then, voxel downsampling is performed to reduce the data volume; finally, ground points are removed, resulting in effective point cloud data that retains valid environmental geometric features (the effective point cloud data is still in 3D point cloud data format); then, a local geometric feature description algorithm (such as the FPFH algorithm) is used to extract the geometric feature vectors of the effective point cloud data corresponding to the current time t, and the matrix formed by the geometric feature vectors of all lidars corresponding to the current time t is denoted as the geometric feature matrix F. u,t ; Simultaneously, the IMU inertial measurement data at the current time t is first subjected to zero-bias correction to remove static drift error; then temperature error compensation is performed to minimize environmental temperature interference, ultimately outputting continuous position and pose increment data; then, an inertial navigation solution algorithm (such as an integral preprocessing algorithm based on Kalman filtering) is used to process the IMU inertial measurement data and pose increment data to obtain the inertial feature vector of the IMU inertial measurement data corresponding to the current time t. The matrix formed by the inertial feature vectors of all IMU inertial sensors corresponding to the current time t is denoted as the inertial feature matrix F. i,t .

[0039] The inertial feature vector includes angular velocity, acceleration, and pose increment features.

[0040] Step 22: Based on the multi-source sensing matrices of each type and their corresponding effectiveness coefficients at the current time t, calculate the fusion feature matrix F at the current time. t : F t =ω v,t ·σ(F v,t )+ω u,t ·σ(F l,t )+ω i,t ·σ(F i,t );ω v,t +ω u,t +ω i,t =1; Where, ω v,t ω u,t and ω i,t σ(·) represents the effectiveness coefficients of the visual feature matrix, geometric feature matrix, and inertial feature matrix at the current time t, respectively; σ(·) represents the L2 normalization function, which is used to normalize the element values ​​in different types of feature matrices to the same scale and eliminate the differences in feature dimensions.

[0041] Optionally, obtaining the effectiveness coefficients of various types of multi-source sensing matrices at the next time step t+1 also includes the following: The operating status of various types of sensors is monitored. If the operating status of each type of sensor at the current time t remains unchanged compared to the previous time t-1, then the effectiveness coefficients of the multi-source sensing matrices of each type at the next time t+1 will remain consistent with the current time t. Sensor operating status includes two types: normal and abnormal. For any given sensor, whether the operating status changes from abnormal to normal or from normal to abnormal, it constitutes a change in operating status.

[0042] If the sensor is in a working state at the current time t, and this state has changed compared to the previous time t-1, then the following steps are included: Step 001: Obtain the percentage of visual sensors, LiDAR, and IMU inertial sensors that are operating normally at the current time t, respectively. v,t FW u,t and FW i,t ; Step 002, FW v,t FW u,t and FW i,t After L1 normalization, they are denoted as FW. v,t ´、FW u,t ´ and FW i,t Then calculate the intermediate coefficients ω of the visual feature matrix at the next time step t+1. v,t+1 ´、Intermediate coefficients ω of the geometric characteristic matrix u,t+1 ´ and the intermediate coefficients ω of the inertial characteristic matrix i,t+1 ´: ω v,t+1 =FW v,t ·ω v,t + FW v,t ·FW v,t ´·ω v,t-1 ; ω u,t+1 =FW u,t ·ω u,t +FW u,t ·FW u,t ´·ω u,t-1 ; ω i,t+1 =FW i,t ·ω i,t +FW u,t ·FW i,t ´·ω i,t-1 ; Step 003, the intermediate coefficients ω of the visual feature matrix at the next time step t+1 are... v,t+1 ´、Intermediate coefficients ω of the geometric characteristic matrix u,t+1 ´ and the intermediate coefficients ω of the inertial characteristic matrix i,t+1 After L1 normalization, the effectiveness coefficients ω of the visual feature matrix at the next time step t+1 are obtained. v,t+1 ω, the efficiency coefficient of the geometric feature matrix u,t+1 and the efficiency coefficient ω of the inertial characteristic matrix i,t+1 .

[0043] The effectiveness coefficients of various types of multi-source sensing matrices at the initial time t=0 are preset by technicians.

[0044] To facilitate understanding, the following example illustrates this: At the previous moment, the effectiveness coefficients of the visual feature matrix, geometric feature matrix, and inertial feature matrix were 0.4, 0.3, and 0.3, respectively. At the current moment, these coefficients are also 0.4, 0.3, and 0.3, respectively. However, at the current moment, two of the eight visual sensors suddenly stop acquiring image data. All other sensors remain in the same working state as at the previous moment, operating normally. Therefore, at the next moment, the effectiveness coefficients of the visual feature matrix, geometric feature matrix, and inertial feature matrix are 0.3182, 0.3409, and 0.3409, respectively. Compared to the previous moment, the effectiveness coefficient of the visual feature matrix has decreased, while the effectiveness coefficients of the geometric and inertial feature matrices have both increased.

[0045] It's important to note that the matrix effectiveness coefficient is used to quantitatively evaluate the richness and effectiveness of the environmental features captured by this type of sensor at the current moment. Taking visual sensors as an example, a robot typically has multiple visual sensors. Only when all visual sensors are functioning normally can the geometric feature vectors corresponding to all visual sensors be acquired at the same time. In other words, the geometric feature matrix at that moment contains all geometric feature vectors. However, if one or more visual sensors fail, one or more geometric feature vectors will be missing from the geometric feature matrix at that moment. Because each visual sensor has its own focus in data acquisition, even if not all the visual sensors on the robot are currently damaged, the overall effectiveness of the data acquired by this type of sensor is reduced. Therefore, to ensure that the multi-source perception data can better serve the subsequent triple loop closure determination, it is necessary to adjust the proportion of the data acquired by the visual sensors, LiDAR, and IMU inertial sensors in the fusion feature matrix at the next moment, that is, to adjust the effectiveness coefficient of each type of multi-source perception matrix.

[0046] If a visual sensor that malfunctioned in the previous moment is now able to collect data again, then using the method of this application to calculate the matrix effectiveness coefficient for the next moment will result in an improvement in the visual feature matrix effectiveness coefficient compared to the current moment. However, this improvement is not abrupt but rather based on the effectiveness coefficients of the visual feature matrix at both the current and previous moments. Similarly, as can be seen above, the decrease in the matrix effectiveness coefficient is not abrupt either, but rather based on the effectiveness coefficients of the corresponding feature matrices at both the current and previous moments. This ensures that the importance proportion of the corresponding data type in the fused feature matrix caused by changes in the operating state of a certain type of sensor is adjusted in a timely manner, while also being constrained by the effectiveness coefficients of the corresponding matrix at historical moments, preventing abrupt changes.

[0047] Therefore, the adjustment of the effectiveness coefficients of the multi-source sensing matrix of each type according to whether the working state of different types of sensors changes is not only scientific and simple to calculate, but can also effectively ensure that when a single sensor suddenly fails, the application can automatically and reasonably and efficiently improve the weight allocation of other types of effective sensors in the next moment, thus ensuring the smooth progress of the detection method of the application and improving the robustness of the entire detection method of the application.

[0048] Even in extreme cases where all sensors of a certain type fail, this application can still ensure the smooth operation of its detection method by increasing the weight of data collected by other types of sensors, i.e., increasing the effectiveness coefficient of the corresponding matrix.

[0049] Step 23, use the LLM model based on the fusion feature matrix F at the current time t. t Obtain the semantic description text T at the current time t. t Then, the semantic description text T t After L2 normalization, the semantic embedding vector S is obtained. t : ;in, This represents the text embedding layer mapping function within the LLM model. The L2 norm of a vector is used to uniformly map text of different lengths to a fixed-scale metric space.

[0050] In this embodiment, the LLM model is a lightweight large language model with 4-bit / 8-bit quantization (such as the Llama-2 / MiniGPT-4 model that has been pre-trained and fine-tuned on the multimodal SLAM dataset).

[0051] Fusion feature matrix F tAfter being fed into the LLM model: the geometric feature vector corresponding to the current time t is transformed into a depth image through multi-view projection, or the geometric structure information is extracted by a preset extractor and then transformed into a text prompt; the inertial feature vector is transformed into a structured text prompt (such as "current moving speed is x, angular velocity is y"), and after being fused with the visual feature vector, the semantic information of the current environment is jointly analyzed; the analysis result includes, but is not limited to, one or more combinations of scene type, core object distribution, spatial layout and motion state (it is not necessary to force the output of all categories at every time moment, and can be dynamically output according to environmental features).

[0052] The LLM model, based on the fused feature matrix of the current time step and historical time steps, combined with the semantic information extracted above, autonomously generates a semantic description text T in natural language form for the current time step. t (e.g., "Indoor corridor scene, there is a metal shelf on the left, and the moving vehicle moves forward at a constant speed").

[0053] The semantic description text T t Transform into semantic embedding vector S t This is done in the text embedding layer within the LLM model.

[0054] Step 3 also includes the following sub-steps: Step 31: Based on the fused feature matrix and semantic embedding vector, perform the first loop recurrence determination between the current location point and the suspicious historical location points to obtain the initial screening of historical location points; if there is more than one initial screening of historical location points, proceed to step 32. Step 32: Calculate the comprehensive matching score to perform a second loop retracement judgment on the initial screened historical location points in order to filter out potential historical location points; if there is more than one potential historical location point, proceed to step 33. Step 33: Use the RANSAC algorithm to perform a third loop closure determination on potential historical location points to filter out historical loop closure points; the current location point and one historical loop closure point form a loop.

[0055] Optionally, if there are no initial screening historical position points in step 31; or no potential historical position points in step 32; or no historical loop points in step 33, then it is determined that the current position point does not form a loop with any historical position point, and then the process returns to step 1 at the next moment.

[0056] Step 31 also includes the following sub-steps: Step 311, calculate the semantic embedding vector S of the current position point. t Cosine similarity between the semantic embedding vectors of suspicious historical locations and the cosine similarity of suspicious historical locations is retained, and suspicious historical locations with a cosine similarity above the second threshold are retained. Step 312: Obtain the multi-source perception matrix of suspicious historical location points; retain the historical location points that satisfy the pose constraints between the multi-source perception matrix and the current location point multi-source perception matrix, and record them as the initial screening historical location points.

[0057] In this embodiment, the second threshold ranges from 0.6 to 0.7. The second threshold is not set to an extremely high value close to 1 to accommodate the lexical diversity and sentence variation of large language models when generating natural language, and to avoid missed detections of true loops due to synonym substitution.

[0058] Step 32 also includes the following: based on the fusion feature matrix, semantic description text and semantic embedding vector, calculate the comprehensive matching score between the current location point and each of the initial screening historical location points; then output the initial screening historical location points whose comprehensive matching score is greater than the fourth set threshold as potential historical location points.

[0059] Optionally, the comprehensive matching scores are sorted in descending order, and the top k initial screening historical position points with comprehensive matching scores greater than the fourth set threshold are output as potential historical position points.

[0060] If the number of initial screening historical position points with a comprehensive matching score greater than the fourth set threshold is less than k, then all initial screening historical position points with a comprehensive matching score greater than the fourth set threshold will be output as potential historical position points.

[0061] Optionally, the formula for calculating the overall matching score is as follows: Score(L t ,L H )=α t sim f (F t ,F H )+(1-α t ) sim m (L t ,L H ); sim m (L t ,L H )=γ sim s (S t ,S H )+(1-γ) sim T (T t ,T H ); Among them, L t L represents the current position at time t; HThis represents a current initial screening historical position; Score(L) t ,L H ) represents the location point L t Compared with the historical location point L of the initial screening H The overall matching score between them; α t The sim represents the environmental coefficient at the current time t. m (L t ,L H ) represents L t With L H The original semantic similarity between them; γ represents the semantic internal fusion weight coefficient, γ∈[0,1]; T H Indicates the initial screening historical location point L H The corresponding semantic description text; S H Represents the semantic description text T H The corresponding semantic embedding vector; sim s (S t ,S H ) represents the semantic embedding vector S t With S H Semantic similarity between them; sim T (T t ,T H ) represents the semantic description text T t With T H The semantic relevance between the texts; F H Indicates the initial screening historical location point L H The fusion feature matrix; sim f (F t ,F H ) represents the fusion feature matrix F t With F H Cosine similarity between them.

[0062] In this embodiment, the value of γ ranges from 0.7. 0.8.

[0063] Optionally, obtain the environmental coefficient α at the current time t. t It also includes the following: Calculate the fusion feature matrix F at the current time step. t Shannon information entropy H(F) t If H(F) t If the position at time t is above the third information entropy threshold H0, then it is determined that the position is in a normal environment. Let α t =A1; otherwise, it is determined that the current position at time t is in a weak environment, let α t =A2.

[0064] In this embodiment, the value range of H0 is 1.2 to 1.5; the value range of A1 is 0.5. 0.6; the range of values ​​for A2 is 0.2. 0.3.

[0065] (1-α t ) sim m (L t ,L H ) represents the feature matching score for all sensors at the bottom layer, α t sim f (F t ,F H () represents the high-level semantic matching score. It can be seen that the environmental coefficient α... t This is used to adjust the overall weighting of these two factors in the comprehensive matching score. Furthermore, in this application, the environmental coefficient α... t The value of depends on the fusion feature matrix F t Shannon information entropy; and the fused feature matrix F t The acquisition of the data depends on the working status of all the underlying sensors. Therefore, we need to adjust the impact of the working status of the sensors on the overall matching score through an environmental coefficient. When the current environment is determined to be normal, the feature matching scores of all the underlying sensors and the semantic matching scores of the high-level sensors can participate in the calculation of the overall matching score in a balanced manner. However, in a weak environment, the accuracy of the multi-source sensing data collected by the underlying sensors may not be as high. Our loop closure detection needs to enhance the dependence on the semantic description text and semantic embedding vectors obtained by the LLM model. Therefore, we need to actively reduce the weight of the feature matching scores of all the underlying sensors, that is, give up more weight for subsequent semantic matching.

[0066] Step 33 also includes the following sub-steps: Step 331: Extract feature points from the visually corrected image and use the RANSAC algorithm to obtain the visual essence matrix between the current position point and the potential historical position point. Step 332: Calculate the proportion of matching points that satisfy the corresponding visual essence matrix to the total number of feature points extracted from the corresponding visual correction image, and record it as the inlier rate between the current position point and the corresponding potential historical position point. Step 333: If there is only one historical location point and the current location point with an in-point ratio greater than the fifth set threshold, then the corresponding historical location point is a historical loop point, and the historical loop point and the current location point form a loop. If there are two or more historical position points with an in-point ratio greater than the fifth set threshold, then after sorting the in-point ratios in descending order, the historical position point corresponding to the first in-point ratio is determined to be the historical loop point, and the historical loop point and the current position point form a loop.

[0067] Optionally, in step 333: If there are two or more historical location points with an in-point rate greater than the fifth set threshold, then after sorting the in-point rates in descending order: if the difference between the in-point rate of the first position and the in-point rate of the second position is greater than the safety redundancy difference, then the historical location point corresponding to the first in-point rate is determined to be a historical loop point; if the difference between the in-point rate of the first position and the in-point rate of the second position is below the safety redundancy difference, then cross-modal secondary verification is performed.

[0068] Cross-modal secondary verification: Obtain the original semantic similarity (sim) between the historical position points and the current position point of all inlier rates corresponding to the difference between the current and first inlier rates, within the safety redundancy difference. m (L t ,L H Then sort them in descending order. If the original semantic similarity of the first position is the same as the historical position corresponding to the first inlier rate, then the historical position corresponding to the first inlier rate is determined to be a historical loop point. If the original semantic similarity of the first position is different from the historical position corresponding to the first inlier rate, and the difference between the original semantic similarity of the historical position corresponding to the first position and the original semantic similarity of the historical position corresponding to the first inlier rate is greater than the semantic confidence threshold ΔS, then the historical position corresponding to the original semantic similarity of the first position is determined to be a historical loop point.

[0069] In this embodiment, the fifth set threshold is 70%; the safety redundancy difference is 3%; and the semantic confidence threshold ΔS is 0.15.

[0070] This application introduces cross-modal secondary verification, which involves backtracking and comparing based on the original semantic similarity. This effectively avoids misjudgments caused by relying solely on the highest inlier rate due to highly repetitive physical geometric textures in the scene (such as identical load-bearing columns or symmetrical shelves). When inlier rates are highly similar, this application prioritizes the global judgment based on high-dimensional semantics. This mechanism overcomes the bottleneck of traditional pure geometric matching.

[0071] In step 333, if there is no inlier rate greater than the fifth set threshold, or if the difference between the original semantic similarity of the historical position point corresponding to the first semantic similarity in the cross-modal secondary verification and the original semantic similarity of the historical position point corresponding to the first inlier rate is below the semantic confidence threshold ΔS, then it is determined that there is no historical loop closure in step 33, and an alarm is triggered to the technical personnel. This is because the historical position points at step 333 have successfully passed two loop closure checks, indicating that the current position point is extremely similar to these historical position points; however, in step 333, it is still determined that there is no historical loop closure. This indicates that the current position point may have experienced severe positional drift that the sensor cannot process, or that the environment has undergone significant structural changes (such as the original scene being completely demolished and rebuilt). In this case, technical personnel need to intervene for further processing.

[0072] After determining that the current position point and a certain historical position point in the first database form a loop, the current position point and the corresponding historical position point in the first database are merged. That is, the current time is added to the information of the corresponding historical position point in the first database to indicate that the carrier (robot) of the detection method of this application has reached the corresponding historical position point again at the current time.

[0073] Use the spatial coordinates of the corresponding historical location points in the first database as the correct spatial coordinates of the current location point.

[0074] After determining that a loop has formed between the current location and a historical location in the first database, the application proceeds as follows: The correct spatial coordinates and original spatial coordinates of the current location, relative pose transformation, and matching confidence are obtained to generate standardized loop constraint information. This information is then output to the optimization graph (such as a factor graph) in the SLAM backend to optimize the method for correcting the position coordinates (i.e., using relative pose transformation to force the current position caused by accumulated errors back to the correct position space corresponding to the historical valid loop), thereby completely eliminating positioning drift errors and completing trajectory optimization and global map updates.

[0075] The loop closure detection method of this application integrates features from multiple sensors to obtain a fused feature matrix of location points. Based on the fused feature matrix, an LLM model is used to further obtain semantic description text and semantic embedding vectors. Finally, based on the fused feature matrix, semantic description text, and semantic embedding vectors, a triple loop closure determination is performed between the current location point and historical location points to efficiently and accurately determine whether the current location point forms a loop with a certain historical location point. At the same time, changes in the working state of the sensors will automatically cause changes in the effectiveness coefficient and environmental coefficient of this application. These changes will automatically adjust the application ratio of different types of multi-source sensing data used as the basis for loop closure detection, thereby improving the stability of the determination results of this application method.

[0076] According to manual verification, compared with loop closure detection using only visual image data obtained from visual sensors, the loop closure detection accuracy of this application is improved by 27.3%; on a robotic vacuum cleaner with a total of 12 visual sensors, even after turning off 2 visual sensors, the loop closure detection accuracy of this application is still maintained at 95.1%.

[0077] The loop closure detection method of this application utilizes multi-source sensing data collected by various types of sensors. Compared with existing technologies, this makes the current position data available for subsequent loop closure detection more comprehensive, which naturally helps to improve the accuracy of loop closure detection.

[0078] The loop closure detection method of this application first filters out suspicious historical position points from all historical position points in the first database before performing triple loop closure judgment. The reason why this application refers to the spatial coordinates directly obtained from the current position point in step 1 as the "original spatial coordinates" is because the "original spatial coordinates" are not necessarily the correct spatial coordinates of the current position point. The "original spatial coordinates" may be obtained after the correct spatial coordinates have undergone positional drift, which leads to the existence of the following four situations: ① The original spatial coordinates of the current location point coincide with the coordinates of a certain historical location point, but the correct spatial coordinates of the current location point coincide with the coordinates of another historical location point; ②The original spatial coordinates of the current location point coincide with the coordinates of a certain historical location point, but the correct spatial coordinates of the current location point do not coincide with any historical location coordinates; ③The original spatial coordinates of the current location do not coincide with any of the historical location coordinates, but the correct spatial coordinates of the current location coincide with a certain historical location coordinate; ④ The original spatial coordinates of the current location do not coincide with any historical location coordinates, and the correct spatial coordinates of the current location also do not coincide with any historical location coordinates.

[0079] Therefore, we cannot directly determine whether the current location forms a loop with a historical location based on its original spatial coordinates. Furthermore, considering that the degree of coordinate drift is limited, this application sets a first offset threshold and uses the offset distance calculation, which has a very small computational cost, to filter out a subset of suspicious historical location points from all historical location points in the first database. These suspicious historical location points (historical location points whose spatial coordinates are offset from the original coordinates of the current location by more than the first offset threshold cannot form a loop with the current location) represent the initial screening of historical location points in this application, thus reducing the computational burden for all subsequent steps. If no suspicious historical location points exist in the first database, it can be directly determined that the current location does not form a loop with any historical location point, eliminating the need to proceed to step 2. Therefore, each step in this application completes the screening of historical location points with the minimum computational cost.

[0080] Only if more than one suspicious historical location point is selected in step 1 is it necessary to proceed to step 2 to obtain the fusion feature matrix, semantic description text, and semantic embedding vector of the current location point; otherwise, directly obtain the multi-source perception data and original spatial coordinates of the location point information at the next moment and return to step 1.

[0081] The fusion feature matrix serves as a crucial foundation for subsequent triple loop closure determination. While leveraging data from various sensor types to enhance the accuracy of loop closure determination, it continuously adjusts the validity coefficients of different feature matrices at the next time step, as well as the environmental coefficients in the triple loop closure determination. This reduces the interference introduced by sensor data loss and feature failure scenarios in complex environments on the accuracy of loop closure determination, thereby improving the robustness and versatility of the detection method in this application and enabling adaptive loop closure detection under different environments.

[0082] This application determines which historical position the current location forms a loop with by sequentially applying a triple loop closure check based on the fused feature matrix, semantic description text, and semantic embedding vector. In this triple loop closure check, each step further filters the historical position points. The number of retained historical position points decreases after each check. Therefore, even though the computational cost of the RANSAC algorithm in the third loop closure check is significantly higher than the first two, the overall computational cost and time consumption for loop closure detection at each time step are not too high because the number of historical position points requiring the third check is very small, ensuring efficient loop closure detection. In other words, this application's loop closure detection method significantly reduces computational complexity while maintaining detection rigor and accuracy, greatly improving the real-time performance of the detection results and meeting the real-time requirements of highly dynamic navigation scenarios.

[0083] This application uses an LLM model to obtain the semantic description text and semantic embedding vector of the current location point in order to balance the adverse effects of changes in sensor data quality on the accuracy of the detection method in scenarios where sensor data is missing or environmental features are weak.

[0084] The loop closure detection algorithm proposed in this application has extremely high practical application value: it adopts a lightweight LLM model that has been quantized and pruned, which is adapted to the edge computing power constraints of mobile robots and does not require additional high-performance computing hardware; the modular design of each module of the system can be flexibly integrated into existing visual SLAM and laser SLAM systems, with low modification cost, and can be widely used in SLAM systems in fields such as service robots, industrial robots, autonomous driving, and drones.

[0085] This application provides an application of an LLM-enhanced SLAM loop closure detection method, including the following: using the LLM-enhanced SLAM loop closure detection method described above, it is determined in real time whether the robot's current position point forms a loop with a certain historical position point. If a loop is formed, the loop closure constraint information is obtained to optimize the robot's positioning drift error and trajectory optimization.

[0086] Obtaining closure constraint information is a prior art technique and will not be elaborated upon here.

[0087] This application also provides an LLM-enhanced SLAM loop closure detection system, including: Suspicious historical location point filtering module, fusion feature acquisition module, triple loop closure determination module; The suspicious historical location point filtering module is used to obtain the multi-source sensing data and original spatial coordinates of the current location point, filter suspicious historical location points in the first database, and then send the suspicious historical location points and the current location point into the fusion feature acquisition module. The fusion feature acquisition module is used to preprocess the multi-source sensing data of the current location point to obtain the fusion feature matrix. Then, the LLM model is used to obtain the semantic description text and semantic embedding vector of the current location point based on the fusion feature matrix of the current time, and then the data is sent to the triple loop closure determination module. The triple loop closure determination module is used to determine whether the current location point and a certain suspicious historical location point form a loop based on the fused feature matrix, semantic description text and semantic embedding vector. Each module is programmed or configured to perform the steps of an LLM-enhanced SLAM loop closure detection method as described above.

[0088] This application also provides a computer-readable storage medium storing a computer program programmed or configured to perform an LLM-enhanced SLAM loop closure detection method as described above.

[0089] This application also provides a computer program product, including a computer program / instructions that are executed by a processor to implement the steps of the LLM-enhanced SLAM loop closure detection method described above.

[0090] The technologies, shapes, and structures not described in detail in this application are all well-known technologies. It should also be noted that the above are merely preferred embodiments of this application and are not intended to limit the scope of this application. The components or steps in the embodiments of this application can be decomposed and / or recombined, and these decompositions and / or recombinations should be considered as equivalent solutions of this application and should all fall within the protection scope of this application.

Claims

1. A SLAM loop closure detection method based on LLM enhancement, characterized in that, Includes the following steps: Step 1: After obtaining the multi-source sensing data and original spatial coordinates of the current location point, filter suspicious historical location points in the first database; if there is more than one suspicious historical location point in the first database, proceed to Step 2. Multi-source sensing data includes visual image data, lidar point cloud data, and IMU inertial measurement data; historical locations whose offset distance between their spatial coordinates and the original spatial coordinates of the current location is less than a first offset threshold are recorded as suspicious historical locations. Step 2: After preprocessing the multi-source sensing data of the current location point, a fusion feature matrix is ​​obtained. Then, the LLM model is used to obtain the semantic description text and semantic embedding vector of the current location point based on the fusion feature matrix of the current time. Step 3: Based on the fused feature matrix, semantic description text, and semantic embedding vector, perform a triple loop closure judgment on the current location point and the suspicious historical location point to confirm whether the current location point and a certain suspicious historical location point form a loop. Step 3 also includes the following sub-steps: Step 31: Based on the fused feature matrix and semantic embedding vector, perform the first loop recurrence determination between the current location point and the suspicious historical location points to obtain the initial screening of historical location points; if there is more than one initial screening of historical location points, proceed to step 32. Step 32: Calculate the comprehensive matching score to perform a second loop retracement judgment on the initial screened historical location points in order to filter out potential historical location points; if there is more than one potential historical location point, proceed to step 33. Step 33: Use the RANSAC algorithm to perform a third loop closure determination on potential historical location points to filter out historical loop closure points; the current location point and one historical loop closure point form a loop.

2. The SLAM loop closure detection method based on LLM enhancement according to claim 1, characterized in that, Step 2 also includes the following sub-steps: Step 21: After preprocessing and extracting feature vectors from the visual image data collected by all visual sensors, the point cloud data collected by the lidar, and the IMU inertial measurement data collected by the IMU inertial sensor at the current time t, the corresponding visual feature vectors, geometric feature vectors, and inertial feature vectors are obtained respectively. Constructing the multi-source perception matrix of the current time t: the matrix composed of the visual feature vectors of all visual sensors corresponding to the current time t is denoted as a visual feature matrix F v,t ; the matrix composed of the geometric feature vectors of all lidars corresponding to the current time t is denoted as a geometric feature matrix F u,t ; the matrix composed of the inertial feature vectors of all IMU inertial sensors corresponding to the current time t is denoted as an inertial feature matrix F i,t ; Step 22: Based on the multi-source sensing matrices of various types and their corresponding effectiveness coefficients at the current time t, calculate the fusion feature matrix F at the current time. t : F t =ω v,t ·σ(F v,t )+ω u,t ·σ(F l,t )+ω i,t ·σ(F i,t );oh v,t +oh u,t +oh i,t =1; Where, ω v,t ω u,t and ω i,t Let represent the effectiveness coefficients of the visual feature matrix, geometric feature matrix, and inertial feature matrix at the current time t, respectively; σ(·) represents the L2 normalization function. Step 23, use the LLM model based on the fusion feature matrix F at the current time t. t Obtain the semantic description text T at the current time t. t Then, the semantic description text T t After L2 normalization, the semantic embedding vector S is obtained. t : ;in, This represents the text embedding layer mapping function within the LLM model. This represents the L2 norm of a vector.

3. The SLAM loop closure detection method based on LLM enhancement according to claim 2, characterized in that, Step 22 also includes the following: The working status of each type of sensor is monitored. If the working status of each type of sensor at the current time t has not changed compared to the previous time t-1, then the effectiveness coefficient of each type of multi-source sensing matrix at the next time t+1 is consistent with the current time t. The working status of the sensors includes two types: normal and abnormal.

4. The SLAM loop closure detection method based on LLM enhancement according to claim 3, characterized in that, If the sensor is in working state at the current time t, and this state has changed compared to the previous time t-1, then obtaining the effectiveness coefficients of each type of multi-source sensing matrix at the next time t+1 also includes the following steps: Step 001: Obtain the percentage of visual sensors, LiDAR, and IMU inertial sensors that are operating normally at the current time t, respectively. v,t FW u,t and FW i,t ; Step 002, FW v,t FW u,t and FW i,t After L1 normalization, they are denoted as FW. v,t ´、FW u,t ´ and FW i,t Then calculate the intermediate coefficients ω of the visual feature matrix at the next time step t+1. v,t+1 ´、Intermediate coefficients ω of the geometric characteristic matrix u,t+1 ´ and the intermediate coefficients ω of the inertial characteristic matrix i,t+1 ´: oh v,t+1 ´=FW v,t ·oh v,t + FW v,t ·FW v,t ´·ω v,t-1 ; oh u,t+1 ´=FW u,t ·oh u,t +FW u,t ·FW u,t ´·ω u,t-1 ; oh i,t+1 ´=FW i,t ·oh i,t +FW u,t ·FW i,t ´·ω i,t-1 ; Step 003, the intermediate coefficients ω of the visual feature matrix at the next time step t+1 are... v,t+1 ´、Intermediate coefficients ω of the geometric characteristic matrix u,t+1 ´ and the intermediate coefficients ω of the inertial characteristic matrix i,t+1 After L1 normalization, the effectiveness coefficients ω of the visual feature matrix at the next time step t+1 are obtained. v,t+1 ω, the efficiency coefficient of the geometric feature matrix u,t+1 and the efficiency coefficient ω of the inertial characteristic matrix i,t+1 .

5. The SLAM loop closure detection method based on LLM enhancement according to claim 2, characterized in that: Step 31 also includes the following: Step 311, calculate the semantic embedding vector S of the current position point. t Cosine similarity between the semantic embedding vectors of suspicious historical locations and the cosine similarity of suspicious historical locations is retained, and suspicious historical locations with a cosine similarity above the second threshold are retained. Step 312: Obtain the multi-source perception matrix of suspicious historical location points; retain the historical location points that satisfy the pose constraints between the multi-source perception matrix and the current location point multi-source perception matrix, and record them as the initial screening historical location points; Step 32 also includes the following: Based on the fused feature matrix, semantic description text, and semantic embedding vector, calculate the comprehensive matching score between the current location and each initially screened historical location; then output the initially screened historical location points whose comprehensive matching score is greater than a fourth set threshold as potential historical location points; the formula for calculating the comprehensive matching score is as follows: Score(L t ,L H )=α t sim f (F t ,F H )+(1-α t ) sim m (L t ,L H ); sim m (L t ,L H )=γ sim s (S t ,S H )+(1-γ) sim T (T t ,T H ); Among them, L t L represents the current position at time t; H This represents a current initial screening historical position; Score(L) t ,L H ) represents the location point L t Compared with the historical location point L of the initial screening H The overall matching score between them; α t The sim represents the environmental coefficient at the current time t. m (L t ,L H ) represents L t With L H The original semantic similarity between them; γ represents the semantic internal fusion weight coefficient, γ∈[0,1]; T H Indicates the initial screening historical location point L H The corresponding semantic description text; S H Represents the semantic description text T H The corresponding semantic embedding vector; sim s (S t ,S H ) represents the semantic embedding vector S t With S H Semantic similarity between them; sim T (T t ,T H ) represents the semantic description text T t With T H The semantic relevance between the texts; F H Indicates the initial screening historical location point L H The fusion feature matrix; sim f (F t ,F H ) represents the fusion feature matrix F t With F H Cosine similarity between them; Step 33 also includes the following sub-steps: Step 331: Extract feature points from the visually corrected image and use the RANSAC algorithm to obtain the visual essence matrix between the current position point and the potential historical position point. Step 332: Calculate the proportion of matching points that satisfy the corresponding visual essence matrix to the total number of feature points extracted from the corresponding visual correction image, and record it as the inlier rate between the current position point and the corresponding potential historical position point. Step 333: If there is only one potential historical location point with an in-point rate greater than the fifth set threshold, then the corresponding potential historical location point is a historical loop point; if there are two or more potential historical location points with an in-point rate greater than the fifth set threshold, then after sorting the in-point rates in descending order, the potential historical location point corresponding to the first in-point rate is determined to be a historical loop point.

6. The SLAM loop closure detection method based on LLM enhancement according to claim 5, characterized in that: If there are two or more potential historical location points with an in-point ratio greater than the fifth set threshold, then after sorting the in-point ratios in descending order: if the difference between the in-point ratio of the first position and the in-point ratio of the second position is greater than the safety redundancy difference, then the potential historical location point corresponding to the first in-point ratio is determined to be a historical loop point; if the difference between the in-point ratio of the first position and the in-point ratio of the second position is below the safety redundancy difference, then cross-modal secondary verification is performed.

7. The SLAM loop closure detection method based on LLM enhancement according to claim 6, characterized in that, Cross-modal secondary verification includes the following: The semantic primitive similarity (sim) between the potential historical position points and the current position point is obtained by acquiring the difference between the current and first-ranked inlier rates, within the safety redundancy difference. m (L t ,L H After sorting in descending order, if the original semantic similarity of the first position is the same as the potential historical position corresponding to the first inlier rate, then the potential historical position corresponding to the first inlier rate is determined to be a historical loop point; if the original semantic similarity of the first position is different from the potential historical position corresponding to the first inlier rate, and the difference between the potential historical position corresponding to the original semantic similarity of the first position and the potential historical position corresponding to the first inlier rate is greater than the semantic confidence threshold ΔS, then the potential historical position corresponding to the original semantic similarity of the first position is determined to be a historical loop point.

8. The SLAM loop closure detection method based on LLM enhancement according to claim 6, characterized in that: In step 33: If there is no inlier rate greater than the fifth set threshold, or if the potential historical location point corresponding to the first semantic original similarity in the cross-modal secondary verification is below the semantic confidence threshold ΔS, then it is determined that there is no historical loop point in step 33, and an alarm is triggered to the technical personnel. If there are no initial screening historical position points in step 31; or no potential historical position points in step 32; or no historical loop points in step 33, then it is determined that the current position point does not form a loop with any historical position point, and then the process returns to step 1 at the next moment.

9. An application of a SLAM loop closure detection method based on LLM enhancement, characterized in that, Includes the following: using an LLM-enhanced SLAM loop closure detection method as described in any of claims 1-8, to determine in real time whether the robot's current position forms a loop with a certain historical position. If a loop is formed, loop closure constraint information is obtained to optimize the robot's positioning drift error and trajectory.

Images