Visual SLAM loopback detection method, system and device based on structured scene and medium

By combining ORB-SLAM2 and RTAB-MAP, the visual template library is optimized by using convolutional neural networks and discrete Bayesian filters, the problem of low visual SLAM loopback detection efficiency in structured scenarios is solved, and efficient visual template management is achieved. Test.

CN119090946AActive Publication Date: 2024-12-06SUN YAT SEN UNIV

Patent Information

Application Number
CN202411158221.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-06
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

In structured scenarios, traditional visual SLAM loopback detection efficiency is low, resulting in increased storage space usage and extended computing time, and high-performance visual odometer methods are not combined with memory management, resulting in a decrease in detection efficiency.

Method used

Using the combination of ORB-SLAM2's binocular visual odometer and RTAB-MAP, a visual template library is constructed through a convolutional neural network, the image pose node is optimized, the image template that meets the conditions is selected for loopback detection, and a discrete Bayesian filter is used. Perform loop closure detection and optimize the visual template library.

Benefits of technology

It effectively reduces the number of visual templates, improves the computing efficiency and accuracy of loopback detection, reduces the space storage requirements, and improves the stability and robustness of the SLAM system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090946A_ABST
    Figure CN119090946A_ABST
Patent Text Reader

Abstract

The invention discloses a visual SLAM loopback detection method, system and device based on a structured scene and a medium, and the method comprises the steps: carrying out the processing of collected image information through a binocular visual odometer of ORB-SLAM2, and obtaining an image key frame and initial pose data; identifying the image key frame through a convolutional neural network, and constructing a target image visual template library; sending the initial pose data to an RTAB-MAP to perform establishment and transfer of pose nodes so as to optimize an image visual template library; and selecting an image template in the visual template library of the to-be-looped detection image according to the calculated deflection angle of the movement path of the robot relative to the starting point and the distance radius of the movement of the robot obtained based on the binocular visual odometer so as to realize looped detection. The number of visual templates participating in loopback detection can be effectively reduced, and loopback detection efficiency and accuracy are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of power transmission systems, and in particular to a visual SLAM loop detection method, system, equipment and medium based on structured scenes. Background Art

[0002] Structured scenes generally refer to environments with obvious signs, a large number of geometric line and surface features, and relatively simple surrounding environment categories. In structured scenes, visual SLAM algorithms can process image data more efficiently and reduce unnecessary consumption of computing resources.

[0003] However, the efficiency problem of large-scale visual SLAM loop detection in environments with obvious road features such as urban roads has gradually surfaced. As the robot moves over a long period of time and at a large scale, the loop detection of traditional visual SLAM will result in more and more visual templates being stored, occupying more and more storage space, and the calculation time for loop detection will also be longer and longer; the visual templates required for SLAM loop detection will gradually increase with the expansion of the moving range, and the computational efficiency of loop detection brute force matching will decrease exponentially; and the high-performance visual odometer method is not combined with the memory management method of SLAM, resulting in low spatial storage and detection efficiency of visual SLAM loop detection in structured scenes.

[0004] It can be seen that how to perform visual SLAM loop detection on structured scenes to improve detection efficiency has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] The present invention provides a structured scene-based visual SLAM loop detection method, system, device and medium to solve the problems of low spatial storage and computational efficiency in visual SLAM loop detection under structured scenes.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a visual SLAM loop detection method based on structured scenes, comprising:

[0007] The robot's binocular camera collects image information of the target road in real time, and the image information is processed by the binocular visual odometer of ORB-SLAM2 to obtain image key frames and initial pose data;

[0008] The image key frames are recognized and processed by a trained convolutional neural network model to construct a target image visual template library;

[0009] Sending the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be looped;

[0010] Calculate the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtain the distance radius of the robot movement based on the binocular visual odometer under a first preset condition;

[0011] According to the deflection angle and the distance radius, an image template in the visual template library of the image to be looped is selected that meets a second preset condition to perform loop detection.

[0012] Furthermore, the image key frames are recognized and processed by the trained convolutional neural network model to construct a target image visual template library, including:

[0013] Identify and classify the attributes of the road intersection in the image key frame by using the trained convolutional neural network model to obtain a target road intersection attribute image;

[0014] Based on the target road intersection attribute image, a target image visual template library is constructed.

[0015] Furthermore, the sending of the initial pose data to RTAB-MAP to create and transfer image pose nodes to optimize the target image visual template library includes:

[0016] Sending the initial pose data to the short-term memory module of the RTAB-MAP to create a pose node of the corresponding image;

[0017] Corresponding node weights are allocated according to the pose nodes, and image nodes are selected and transferred according to the node weights to obtain image templates that meet the loop conditions in the target image visual template library, so as to optimize the target image visual template library.

[0018] Furthermore, the assigning of corresponding node weights according to the pose nodes, and selecting and transferring image nodes according to the node weights, to obtain an image template that meets the loop condition in the target image visual template library, includes:

[0019] Selecting a target pose node corresponding to a target weight value according to the node weight, and comparing the target pose node with the pose data of the current position image to update the node weight;

[0020] According to the updated node weights, the pose nodes are screened and transferred, and the images corresponding to the screened and transferred pose nodes are used as visual templates of the images to be looped.

[0021] Furthermore, it also includes:

[0022] The RTAB-MAP detects loop closure of the posture node through a discrete Bayesian filter to obtain a loop probability and a detection time;

[0023] Determine whether the loop probability and the detection time exceed preset corresponding thresholds respectively, and obtain a determination result;

[0024] The pose nodes are transferred according to the judgment result to optimize the target image visual template library.

[0025] Furthermore, the calculating of the deflection angle of the robot moving path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtaining the distance radius of the robot moving based on the binocular visual odometer under a first preset condition, includes:

[0026] When the robot moves, a deflection angle between a starting node of the moving path and a pose node of a current position image acquired by the binocular camera is determined, and whether the deflection angle exceeds the angle threshold;

[0027] Under the first preset condition, the ATE error of the robot's moving distance is calculated by the binocular visual odometer of the ORB-SLAM2, and the distance radius is calculated according to the ATE error.

[0028] Further, selecting an image template in the visual template library of the image to be loop detected that meets a second preset condition according to the deflection angle and the distance radius to perform loop detection includes:

[0029] In the first round of screening, when the deflection angle exceeds the angle threshold, the corresponding image is subjected to loop closure detection;

[0030] In the second round of screening, when the loop detection is started, the image visual template within the distance radius in the image visual template library to be loop detected is selected as the template for the loop detection to perform the loop detection.

[0031] Another embodiment of the present invention provides a visual SLAM loop detection system based on structured scenes, comprising:

[0032] The initial data acquisition module is used to collect the image information of the target road in real time through the binocular camera of the robot, process the image information through the binocular visual odometer of ORB-SLAM2, and obtain the image key frame and initial pose data;

[0033] An image template library construction module is used to identify and process the image key frames through a trained convolutional neural network model to construct a target image visual template library;

[0034] An image template library optimization module is used to send the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be looped;

[0035] A calculation module, used to calculate the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtain the distance radius of the robot movement based on the binocular visual odometer under a first preset condition;

[0036] The loop detection module is used to select an image template in the visual template library of the image to be loop detected that meets the second preset condition according to the deflection angle and the distance radius to perform loop detection. When a loop is detected, global optimization is performed to obtain a globally consistent map.

[0037] Another embodiment of the present invention provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the visual SLAM loop detection method based on structured scenes as described above is implemented.

[0038] Yet another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the visual SLAM loop detection method based on structured scenes as described above is implemented.

[0039] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0040] (1) The binocular visual odometry part of ORB-SLAM2 is used to estimate the initial pose of the robot and extract image key frames. The image classification and recognition is performed by constructing a neural convolutional network to realize the detection of road intersections. It can efficiently identify road intersection data in structured scenes and use image data as a visual template for loop detection, effectively reducing the number of visual templates involved in loop detection and improving computational efficiency and accuracy.

[0041] (2) Use RTAB-MAP to optimize the image data, select the visual templates involved in loop closure detection, and formulate a suitable visual template selection strategy. Combining the characteristics of ORB-SLAM2 and RTAB-MAP can improve the efficiency of loop closure detection and reduce space storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1It is a flow chart of a method for visual SLAM loop detection based on structured scenes in one embodiment of the present invention;

[0043] Figure 2 is a topological model diagram of an urban road intersection in one embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a situation where a road intersection exists in a city map in one embodiment of the present invention;

[0045] Figure 4 It is a framework diagram of the overall process of visual SLAM loop detection in one embodiment of the present invention;

[0046] Figure 5 It is a neural network framework diagram of ResNet-34 in one embodiment of the present invention;

[0047] Figure 6 is a graph of Loss and Acc of a ResNet-34 network in one embodiment of the present invention;

[0048] Figure 7 It is a structural diagram of the RTAB-MAP memory management model in one embodiment of the present invention;

[0049] Figure 8 is a schematic diagram of a robot moving path in one embodiment of the present invention; wherein, Figure 8 Includes (a) and (b);

[0050] Fig. 9 It is a structural block diagram of a visual SLAM loop detection system based on a structured scene in one embodiment of the present invention;

[0051] Fig.10 It is a structural block diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0054] In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used herein are only for illustrative purposes, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0055] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0056] An embodiment of the present invention provides a method. For details, see Figure 1 , Figure 1 The flowchart of the method for visual SLAM loop detection based on structured scenes in one embodiment of the present invention is shown, comprising the following steps:

[0057] S1. The robot's binocular camera collects image information of the target road in real time, and the image information is processed by the binocular visual odometer of ORB-SLAM2 to obtain image key frames and initial pose data.

[0058] Visual SLAM (Simultaneous Localization and Mapping) is a technology that uses cameras as sensors to simultaneously locate and map in unknown environments. Visual SLAM uses cameras to capture image information of the surrounding environment, and uses computer vision and machine learning algorithms to process and analyze the images, thereby achieving real-time positioning of mobile platforms such as robots, unmanned vehicles or drones and mapping of the surrounding environment.

[0059] ORB-SLAM2 is a complete SLAM solution based on monocular, binocular and RGB-D cameras. It can realize map reuse, loop detection and re-localization. Whether it is a small handheld device indoors, a drone in a factory environment or a car driving in the city, ORB-SLAM2 can work in real time on a standard CPU. In addition, ORB-SLAM2 includes a lightweight positioning mode that can use visual odometers to track unmapped areas and match feature points.

[0060] Preferably, in this embodiment, a robot (unmanned aerial vehicle, unmanned vehicle) is equipped with a camera device to explore the external road environment. When the robot is driving or flying on the target road, the binocular camera continuously collects image information of the roads around the environment. These images contain rich road intersection information and are the basis for subsequent processing and positioning. ORB-SLAM2 uses the pose estimation method of the visual odometer to calculate the pose change of the camera by matching the feature points between adjacent frames, thereby obtaining the image key frame and the initial pose data of the robot.

[0061] The embodiment of the present invention uses the ORB-SLAM2 binocular visual odometer to estimate the pose, which can quickly calculate the pose change of the camera in scenarios with high real-time requirements. The road image information continuously collected by the robot contains rich road intersection information, such as intersections, T-junctions, and roundabouts. Therefore, the ORB-SLAM2 visual odometer can effectively ensure the accuracy of image acquisition.

[0062] S2. Recognize and process the image key frames through the trained convolutional neural network model to construct a target image visual template library.

[0063] Further, the road intersection attributes of the image in the image key frame are identified and classified by the trained convolutional neural network model to obtain a target road intersection attribute image;

[0064] Based on the target road intersection attribute image, a target image visual template library is constructed.

[0065] Specifically, for the intersection detection problem in the visual SLAM of this embodiment, the core task is to identify whether there is a binary judgment of the intersection in the image, that is, to identify the intersection attributes in the above image key frame. Figure 2 , Figure 2 Shown is a topological model diagram of an urban road intersection in one embodiment of the present invention.

[0066] In this embodiment, it is necessary to detect Figure 2 The "T-shaped" and "cross-shaped" road intersections shown in the models ③④⑤⑥ are shown in the figure. Since deep learning methods have superb capabilities in image classification and recognition, and thanks to the ability of deep convolutional neural networks to automatically extract hierarchical feature representations from raw image data, this embodiment considers using a deep convolutional neural network model to solve the problem of detecting road intersections in structured scenes. For specific road intersections in structured scenes, please refer to Figure 3 , Figure 3 The figure shows a schematic diagram of a situation where a road junction exists in a city map in one embodiment of the present invention. Figure 3 It can be seen from the figure that in structured scenes such as urban roads, most of the locations where loops exist are near road intersections. Therefore, it can be understood that the target road intersection attribute image in this embodiment refers to a road intersection attribute image where loops may exist.

[0067] Specifically, the deep convolutional neural network uses a deep network to achieve image classification and recognition in an end-to-end manner, and has super image classification and recognition capabilities. In an embodiment of the present invention, the selectable visual classification and recognition algorithms include neural networks such as Alexnet, VGG and ResNet series. Among them, ResNet is a deep residual network model that solves the gradient vanishing problem of the deep network by introducing residual connections, thereby realizing a deeper network structure. The classification accuracy of ResNet on the ImageNet data set far exceeds other models, making it a benchmark model in the field of image recognition. At the same time, ResNet is also widely used in other computer vision tasks such as target detection and semantic segmentation.

[0068] Therefore, preferably, for visual SLAM loop detection in structured scenes, this embodiment uses the ResNet-34 network to identify road intersections, and the identified road intersection position data is used as a visual template library for loop detection, that is, the target image visual template library. Unlike Alexnet and VGG, the ResNet-34 network uses a deep residual learning framework to solve the problem of gradient disappearance or explosion, and has good computational efficiency and accuracy performance. Therefore, this embodiment uses ResNet-34 for image road intersection recognition and detection. For details, please refer to Figure 4 , Figure 4FIG. 1 is a diagram showing the overall process of visual SLAM loop detection in one embodiment of the present invention. Figure 4 It can be seen that in this embodiment, the key frame image output by the ORB-SLAM2 algorithm is input into ResNet-34 for recognition and classification.

[0069] Preferably, this embodiment is tested on 22 sequences of image sequence datasets KITTI-00 and KITTI-21 containing road intersections. For details, please refer to Figure 5 and Figure 6 , Figure 5 The figure shows a neural network framework diagram of ResNet-34 in one embodiment of the present invention. Figure 6 Shown is a graph of Loss and Acc of a ResNet-34 network in one embodiment of the present invention.

[0070] It can be understood that after the key frame images are recognized and classified by ResNet-34, the attribute images of the road intersection where loops may exist are obtained. These images constitute the target image visual template library. This embodiment generates an image visual template library by using a deep residual network model to recognize images, which can reduce storage pressure and improve detection efficiency during loop detection.

[0071] The embodiment of the present invention adopts the advanced ResNet-34 algorithm to construct a deep residual network model to identify the attribute features of image road intersections. By taking advantage of the high rate and accuracy of ResNet-34, it is possible to efficiently identify tool path intersection data in structured urban scenes, and use these image data as visual templates for loop detection, that is, the image visual template library to be screened by RTAB-MAP.

[0072] S3. Send the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be loop detected.

[0073] RTAB-Map (Real-time Appearance-based Mapping) is a powerful, open source Simultaneous Localization and Mapping (SLAM) library designed for robotics and unmanned driving. It provides a complete solution for building 3D maps and performing real-time positioning and navigation in unknown environments.

[0074] Research has shown that RTAB-MAP plays a vital role in loop closure detection. It optimizes and enhances the performance of the SLAM system by providing a mechanism for creating, selecting, and transferring nodes to control the number of visual templates for loop closure detection.

[0075] In an embodiment of the present invention, in order to enable the robot to have the ability to operate on a large scale and for a long time, this embodiment uses RTAB-MAP to perform loop detection. RTAB-MAP detects whether the robot has returned to a previously visited location, i.e., a loop, by comparing the similarity between the current environmental observation and the historical observation.

[0076] In RTAB-MAP's memory management, short-term memory (STM), working memory (WM) and long-term memory (LTM) are three key modules. For details, please refer to Figure 7 , Figure 7 Shown is a structural diagram of the RTAB-MAP memory management model in one embodiment of the present invention.

[0077] according to Figure 7 It can be seen that the perception storage module receives the image key frame data collected by the binocular camera and sends the initial pose data to the short-term memory module. Among them, it is worth noting that the perception memory of the perception storage module reduces the data dimension by processing the received image key frame. The purpose of this step is to extract useful features for subsequent loop detection, and then estimate the current pose of the robot based on the image features. Both types of data are passed to the short-term memory of RTAB-MAP.

[0078] Furthermore, this embodiment sends the initial pose data to the short-term memory module of the RTAB-MAP to create a pose node of the corresponding image. Specifically:

[0079] In this embodiment, when the information from the front-end binocular visual odometer and the road intersection is transmitted, the short-term memory module will generate a new node for storing the odometer's posture information, the sensor's raw data, and additional information that is helpful to the subsequent processing module. The nodes output by the short-term memory module need to be looped.

[0080] Corresponding node weights are allocated according to the pose nodes, and image nodes are selected and transferred according to the node weights to obtain image templates that meet the loop conditions in the target image visual template library, so as to optimize the target image visual template library.

[0081] For more information on how to select nodes, please refer to Figure 4The right part. This embodiment assigns weights to nodes generated by short-term memory. When it is necessary to transfer from working memory to long-term memory, the position with the smallest weight is selected. If the weights of multiple positions are the same, the oldest position is selected for transfer. When a node is transferred to long-term memory, the node is deleted from working memory. The node with the highest probability of loop closure is selected for loop detection.

[0082] The following is a detailed description of the processing process of the above nodes:

[0083] In this embodiment, a target pose node corresponding to a target weight value is selected according to the node weight, and the target pose node is compared with the pose data of the current position image to update the node weight; according to the updated node weight, the pose node is screened and transferred, and the image corresponding to the screened and transferred pose node is used as the visual template of the image to be loop detected.

[0084] like Figure 7 As shown, this process requires the coordinated action of RTAB-MAP's short-term memory, working memory, and long-term memory. Specifically:

[0085] The short-term memory updates the most recently created pose nodes through a weight update process, which updates the weights of pose nodes based on the similarity of observing consecutive images in time. If the weight update considers that the new position is similar to the previous position in the short-term memory, they will be merged into the new position and the weight of the new position will be increased. By assigning node weights and updating weights, this embodiment can facilitate the subsequent spatial selection of nodes with larger weights to be stored in the system, thereby streamlining spatial storage.

[0086] After merging, the weight of the new location will be increased to reflect its higher importance and richer information. By merging similar nodes, we can avoid creating a large number of repeated, almost identical locations in the map, thereby reducing the waste of storage space. The merging process is usually accompanied by pose optimization, which helps to reduce pose drift due to noise or errors and improve the accuracy of the map.

[0087] The role of working memory is to detect loop closures between spatial locations and domain detection. During the creation of pose nodes, when the number of nodes stored in short-term memory reaches the node number threshold, the earliest position in short-term memory will be moved to working memory. In subsequent loop detection, map optimization, etc., processing fewer and more accurate nodes can significantly improve the efficiency of the algorithm.

[0088] When the RTAB-MAP system update time or the number of nodes in the working memory exceeds the threshold, some nodes in the working memory are transferred to the long-term memory. After transferring these nodes, some useful information is transferred back to the working memory using a weighted mechanism. The pose nodes in the long-term memory do not participate in loop detection.

[0089] The embodiment of the present invention creates, selects and transfers nodes of the pose information of the image by utilizing the different functions of the short-term memory, working memory and long-term memory of RTAB-MAP, thereby realizing the processing processes of creation, transfer and retrieval of pose nodes, and uses the image corresponding to the screened and transferred pose node as the visual template of the image to be loop detected.

[0090] Preferably, in this embodiment, the node quantity threshold is set by the speed of the robot and the speed of acquiring the position.

[0091] Furthermore, in this embodiment, the RTAB-MAP performs loop closure detection on the pose node through a discrete Bayesian filter to obtain a loop probability and a detection time.

[0092] Specifically, RTAB-MAP estimates the loop closure probability using a discrete Bayesian filter by comparing the new position with the position in working memory. Loop closure is detected if there is a high probability of a loop between the new position and the old position in working memory, i.e., the higher the probability of loop closure, the higher the corresponding probability of loop existence.

[0093] Then, it is determined whether the loop probability and the detection time exceed the preset corresponding thresholds, and a determination result is obtained. The posture node is transferred according to the determination result to optimize the target image visual template library.

[0094] The optimization process of the above target image visual template library is described in detail below:

[0095] In this embodiment, preferably, the size of the working memory should be kept within the limits that the Bayesian filter can handle. There are two key steps for RTAB-MAP to achieve loop detection and control the number of visual templates for loop detection to achieve optimization of the visual template library of the target image:

[0096] The first step is retrieval. Through the estimated loop probability, for the neighboring positions that are not in the working memory and have a high probability of loop closure, the nodes are brought back from the long-term memory to the working memory to increase the number of visual template detections. Although this step seems to increase the number of templates, it is based on the premise of nodes with high loop closure probability, so it can increase the probability of loop closure in the future neighboring position recognition.

[0097] The second step is transfer. If the time spent on processing loop closure detection is greater than the time threshold, the earliest node with the least number of views will be transferred to long-term memory to reduce the number of detections of the visual template. The number of transferred positions depends on the number of positions added to the working memory in the current cycle.

[0098] It is worth noting that this embodiment integrates ORB-SLAM2 into RTAB-MAP. Since the visual odometer in RTAB-MAP is independent of the mapping process, it is necessary to disable the loop detection in ORB-SLAM2, and output the estimated value of its visual odometer to RTAB-MAP to complete subsequent loop detection and global optimization processes, and to deal with issues such as spatial storage.

[0099] S4. Calculate the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtain the distance radius of the robot movement based on the binocular visual odometer under a first preset condition.

[0100] The embodiment of the present invention improves the selection strategy of selecting a visual template for loop closure detection, specifically:

[0101] When the robot moves, the deflection angle between the starting node of the moving path and the pose node of the current position image acquired by the binocular camera is determined to determine whether the deflection angle exceeds the angle threshold. For details, please refer to Figure 8 , Figure 8 FIG. 1 is a schematic diagram of a robot moving path in one embodiment of the present invention. Figure 8 Includes (a) and (b).

[0102] From (a), we can see that the path of the robot moving from point A1 to point B1 is similar to a straight line. As for the path from point A2 to point B2, we can see that the deflection angle from the starting point to the end point, that is, from the starting node to the pose node of the current position image, is less than 360°. In this case, it is impossible for a loop to exist. Therefore, for a route like (a), there is no need to consider whether there is a loop.

[0103] As can be seen from (b), the deflection angle of the path of the robot moving from point A3 to point B3 exceeds a certain threshold. In this case, a loop is likely to occur on the route (b).

[0104] Therefore, the embodiment of the present invention can efficiently determine the image that needs to be loop-closed through the calculation and judgment of the deflection angle, laying a foundation for starting a subsequent efficient and feasible loop-closed detection process.

[0105] Furthermore, in this embodiment, under the first preset condition, the ATE error of the robot's moving distance is calculated by the binocular visual odometer of the ORB-SLAM2, and the distance radius is calculated according to the ATE error.

[0106] It can be understood that ATE (Absolute Trajectory Error) is the absolute trajectory error, which is an important indicator for measuring the performance of the SLAM system. The ATE error represents the deviation between the estimated trajectory and the true trajectory. In this step, it can be understood that the distance radius is calculated in advance before the system runs, that is, the first preset condition is: when there is no loopback in ORB-SLAM2, the distance radius is further calculated based on the segmented ATE accuracy of the KITTI data set.

[0107] Specifically, based on RTAB-MAP, the embodiment of the present invention combines the characteristics of urban road environment and visual binocular SLAM to calculate the segmented ATE accuracy of ORB-SLAM2 in the KITTI dataset when there is no loop, as shown in the following Table 4-1:

[0108] Table 4-1 ORB-SLAM2 segmented ATE accuracy in KITTI dataset without loop closure / m

[0109]

[0110] According to Table 4-1, it can be seen that the robot running distance is related to the ATE error. The embodiment of the present invention uses the ATE accuracy to calculate the distance radius of the visual template based on the visual template library of the image to be looped. That is, the distance radius is determined according to a certain error accuracy, and the visual template within the distance radius needs to be looped. This step is also a more accurate selection of the visual template with potential loops.

[0111] S5. According to the deflection angle and the distance radius, an image template in the visual template library of the image to be loop detected that meets a second preset condition is selected to perform loop detection. When a loop is detected, global optimization is performed to obtain a globally consistent map.

[0112] It is understandable that the purpose of loop detection is to eliminate the cumulative error of the SLAM front-end odometer estimation. The embodiment of the present invention combines ORB-SLAM2 with RTAB-MAP to give full play to the advantages of ORB-SLAM2 front-end visual odometer performance and RTAB-MAP back-end memory management.

[0113] Loop detection is to compare the image acquired by the binocular camera in real time with the image template in the visual template library of the image to be detected, and use the feature matching algorithm to find similar feature points between the image to be detected and the image in the template library to calculate the similarity and perform loop detection. Since the embodiment of the present invention has optimized the template in the visual template library of the image to be detected through the above steps, the efficiency of loop detection is very high.

[0114] In this embodiment, when performing loop detection, a key step is required to formulate a more reasonable strategy for selecting visual templates and select appropriate templates for loop detection. Therefore, it can be understood that in this step, the second preset condition is to select an image visual template that meets the angle threshold and meets the set distance radius. Specifically, in this embodiment, two rounds of screening processes are included:

[0115] In the first round of screening, when the deflection angle exceeds the angle threshold, the corresponding image is used for loop detection.

[0116] In an embodiment of the present invention, it is necessary to determine whether the deflection angle of the robot's moving path meets the preset conditions. Preferably, in order to maintain a higher detection efficiency when the deflection error of the ORB-SLAM2 binocular vision angle is small, the preset angle threshold is 270°. When the deflection angle is less than 270 degrees, it can be determined that the corresponding image visual template does not need to start loop detection. When the yaw angle is greater than 270 degrees, the subsequent loop detection needs to be started for the corresponding image visual template. It can be understood that the first round of screening process is the process of determining whether loop detection is needed.

[0117] In the second round of screening, when the loop detection is started, the image visual template within the distance radius in the image visual template library to be loop detected is selected as the template for the loop detection to perform the loop detection.

[0118] According to the segmented ATE accuracy of ORB-SLAM2 in the KITTI dataset without loop closure shown in Table 4-1, it can be seen that as the running distance increases, the cumulative error of SLAM without loop closure correction will become larger and larger.

[0119] Preferably, according to the characteristics of the intersection of structured urban roads, when the loop detection is started, the distance radius of the visual template is selected by taking 3 times the value of the 5000-meter length ATE error of the ORB-SLAM2 binocular visual odometer in KITTI-00, that is, the template within 21 meters of the stored visual template from the current vehicle is selected as the reference image for loop detection. For specific examples, please refer to Figure 7 It can be understood that the second round of screening process is a process of further screening the visual image template to be looped back based on a specific range.

[0120] It can be understood that in an embodiment of the present invention, an image visual template library is constructed by accurately selecting a visual template strategy for SLAM loop detection. When a loop is detected, global optimization is performed to construct a globally consistent map, which can not only meet real-time requirements but also access the location of the entire map when necessary, thereby speeding up the loop detection and greatly reducing the pressure on memory space.

[0121] In summary, the embodiment of the present invention detects image road intersections through the deep residual neural network ResNet-34, which can extract rich image features and is very effective in distinguishing intersection data in different urban scenes, greatly reducing the number of visual templates, and helping to build a high-quality visual template library; it also combines the estimation performance of the ORB-SLAM2 binocular visual odometer in the KITTI data set when there is no loop and the advantages of RTAB-MAP in back-end optimization and memory management, and can more effectively manage map data and formulate a more reasonable visual template selection strategy to select appropriate templates for loop detection, thereby reducing the system's memory usage and optimizing the consistency of the global map, while improving the stability and robustness of the entire SLAM system.

[0122] An embodiment of the present invention provides a visual SLAM loop detection system based on structured scenes. For details, please refer to Fig. 9 , Fig. 9 Shown is a structural block diagram of a visual SLAM loop detection system based on structured scenes in one embodiment of the present invention.

[0123] The initial data acquisition module M1 is used to collect the image information of the target road in real time through the binocular camera of the robot, process the image information through the binocular visual odometer of ORB-SLAM2, and obtain the image key frame and initial pose data;

[0124] The image template library construction module M2 is used to identify and process the image key frames through a trained convolutional neural network model to construct a target image visual template library;

[0125] An image template library optimization module M3 is used to send the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be looped;

[0126] A calculation module M4 is used to calculate the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtain the distance radius of the robot movement based on the binocular visual odometer under a first preset condition;

[0127] The loop detection module M5 is used to select an image template in the visual template library of the image to be loop detected that meets the second preset condition according to the deflection angle and the distance radius to perform loop detection. When a loop is detected, global optimization is performed to obtain a globally consistent map.

[0128] like Fig.10 As shown, the embodiment of the present invention also provides a computer device, Fig.10 A structural block diagram of a preferred embodiment of a computer device provided by the present invention, the computer device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the visual SLAM loop detection method based on structured scenes as described above when executing the computer program.

[0129] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0130] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The processor is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0131] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory can also be other volatile solid-state storage devices.

[0132] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Fig.10 The structural block diagram is only an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or combine certain components, or different components. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0133] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to perform the steps in the visual SLAM loop detection method based on structured scenes in the above embodiment, for example Figure 1 Steps S1 to S5 described in .

[0134] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A visual SLAM loop detection method based on structured scenes, characterized in that: include: The robot's binocular camera collects image information of the target road in real time, and the image information is processed by the binocular visual odometer of ORB-SLAM2 to obtain image key frames and initial pose data; The image key frames are recognized and processed by a trained convolutional neural network model to construct a target image visual template library; Sending the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be looped; Calculate the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtain the distance radius of the robot movement based on the binocular visual odometer under a first preset condition; According to the deflection angle and the distance radius, an image template in the visual template library of the image to be loop detected that meets the second preset condition is selected to perform loop detection, and when a loop is detected, global optimization is performed to obtain a globally consistent map.

2. The visual SLAM loop detection method based on structured scenes as claimed in claim 1, characterized in that, The image key frames are recognized and processed by the trained convolutional neural network model to construct a target image visual template library, including: Identify and classify the attributes of the road intersection in the image key frame by using the trained convolutional neural network model to obtain a target road intersection attribute image; Based on the target road intersection attribute image, a target image visual template library is constructed.

3. The visual SLAM loop detection method based on structured scenes as claimed in claim 1, characterized in that, The sending of the initial pose data to RTAB-MAP to create and transfer image pose nodes to optimize the target image visual template library includes: Sending the initial pose data to the short-term memory module of the RTAB-MAP to create a pose node of the corresponding image; Corresponding node weights are allocated according to the pose nodes, and image nodes are selected and transferred according to the node weights to obtain image templates that meet the loop conditions in the target image visual template library, so as to optimize the target image visual template library.

4. The visual SLAM loop detection method based on structured scenes as claimed in claim 3, characterized in that, The step of assigning corresponding node weights according to the pose nodes, and selecting and transferring image nodes according to the node weights to obtain an image template that meets the loop condition in the target image visual template library includes: Selecting a target pose node corresponding to a target weight value according to the node weight, and comparing the target pose node with the pose data of the current position image to update the node weight; According to the updated node weights, the pose nodes are screened and transferred, and the images corresponding to the screened and transferred pose nodes are used as visual templates of the images to be looped.

5. The visual SLAM loop detection method based on structured scenes as claimed in claim 3, characterized in that, Also includes: The RTAB-MAP detects loop closure of the posture node through a discrete Bayesian filter to obtain a loop probability and a detection time; Determine whether the loop probability and the detection time exceed preset corresponding thresholds respectively, and obtain a determination result; The pose nodes are transferred according to the judgment result to optimize the target image visual template library.

6. The visual SLAM loop detection method based on structured scenes as claimed in claim 1, characterized in that, The step of calculating the deflection angle of the robot moving path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtaining the distance radius of the robot moving based on the binocular visual odometer under a first preset condition, includes: When the robot moves, a deflection angle between a starting node of the moving path and a pose node of a current position image acquired by the binocular camera is determined, and whether the deflection angle exceeds the angle threshold; Under the first preset condition, the ATE error of the robot's moving distance is calculated by the binocular visual odometer of the ORB-SLAM2, and the distance radius is calculated according to the ATE error.

7. The visual SLAM loop detection method based on structured scenes as claimed in claim 6, characterized in that, The step of selecting, according to the deflection angle and the distance radius, an image template from a visual template library of the image to be loop-closed detected that meets a second preset condition to perform loop-closed detection includes: In the first round of screening, when the deflection angle exceeds the angle threshold, the corresponding image is used for loop detection; In the second round of screening, when the loop detection is started, the image visual template within the distance radius in the image visual template library to be loop detected is selected as the template for the loop detection to perform the loop detection.

8. A visual SLAM loop detection system based on structured scenes, characterized in that: include: The initial data acquisition module is used to collect the image information of the target road in real time through the binocular camera of the robot, process the image information through the binocular visual odometer of ORB-SLAM2, and obtain the image key frame and initial pose data; An image template library construction module is used to identify and process the image key frames through a trained convolutional neural network model to construct a target image visual template library; An image template library optimization module is used to send the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be looped; A calculation module, used to calculate the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtain the distance radius of the robot movement based on the binocular visual odometer under a first preset condition; The loop detection module is used to select an image template in the visual template library of the image to be loop detected that meets the second preset condition according to the deflection angle and the distance radius to perform loop detection. When a loop is detected, global optimization is performed to obtain a globally consistent map.

9. A computer device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the visual SLAM loop detection method based on structured scenes as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the visual SLAM loop detection method based on structured scenes as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Semantic segmentation and visual SLAM tight coupling method for dynamic environment

    CN110827305A

  • Dense binocular SLAM method based on BNN

    CN111998862A

  • AGV map construction method and system

    CN113066152A

  • Local refinement mapping system and method based on SLAM and semantic segmentation

    CN116772820A

  • Lightweight visual SLAM system and VSLAM method in dynamic scene

    CN117274313A

Cited By

  • Vision-based vehicle positioning method and device, storage medium and equipment

    CN119339050A

  • A method, device, storage medium and equipment for vehicle positioning based on vision

    CN119339050B