A structured scene-based visual slam loop detection method, system, device and medium

Abstract

The application discloses a kind of based on structured scene's visual SLAM loop detection method, system, equipment and medium, and its binocular vision odometer of ORB-SLAM2 is handled to the image information collected, obtains image key frame and initial pose data;Image key frame is identified by convolutional neural network, and target image visual template library is constructed;Initial pose data is sent to RTAB-MAP to create and transfer pose node, to optimize image visual template library;The deflection angle of the robot moving path relative to starting point obtained by calculation and the distance radius of the robot movement obtained based on binocular vision odometer are used to select the image template in the image visual template library to be loop detection, to realize loop detection.The application can effectively reduce the number of visual templates participating in loop detection, improve loop detection efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of power transmission systems, and in particular to a structured scene-based visual SLAM loop detection method, system, equipment and medium. Background Art

[0002] Structured scenes generally refer to environments with obvious landmarks, numerous geometric features, and relatively simple surroundings. In structured scenes, visual SLAM algorithms can process image data more efficiently, reducing unnecessary computing resource consumption.

[0003] However, efficiency issues with large-scale visual SLAM loop detection in environments with distinct road features, such as urban roads, have gradually surfaced. As robots move over long distances and over large distances, traditional visual SLAM loop detection requires more and more visual templates to be stored, which in turn occupies more and more storage space and increases the computational time for loop detection. The visual templates required for SLAM loop detection gradually increase as the range of movement expands, and the computational efficiency of brute-force matching for loop detection decreases exponentially. Furthermore, high-performance visual odometry methods have not been integrated with SLAM memory management methods, resulting in low spatial storage and detection efficiency for visual SLAM loop detection in structured scenarios.

[0004] Therefore, how to perform visual SLAM loop detection on structured scenes to improve detection efficiency has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] The present invention provides a structured scene-based visual SLAM loop detection method, system, device and medium to solve the problems of low spatial storage and computational efficiency in visual SLAM loop detection in structured scenes.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a visual SLAM loop detection method based on structured scenes, comprising:

[0007] The robot's binocular camera collects image information of the target road in real time, and the image information is processed by the ORB-SLAM2 binocular visual odometry to obtain image key frames and initial pose data;

[0008] The image key frames are recognized and processed by a trained convolutional neural network model to construct a target image visual template library;

[0009] Sending the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be loop detected;

[0010] calculating a deflection angle of a robot movement path corresponding to the position image pair acquired by the binocular camera relative to a starting point, and acquiring a distance radius of the robot movement based on the binocular vision odometer under a first preset condition;

[0011] selecting an image template in the loop detection image visual template library that meets a second preset condition according to the deflection angle and the distance radius, to perform loop detection.

[0012] Further, the image key frame is identified by the trained convolutional neural network model, and a target image visual template library is constructed, including:

[0013] The road intersection attribute of the image in the image key frame is identified and classified by the trained convolutional neural network model, and a target road intersection attribute image is obtained.

[0014] Based on the target road intersection attribute image, a target image visual template library is constructed.

[0015] Further, the initial pose data is sent to the RTAB-MAP to create and transfer image pose nodes, and the target image visual template library is optimized, including:

[0016] The initial pose data is sent to the short-term memory module of the RTAB-MAP to create the pose nodes of the corresponding images;

[0017] According to the pose nodes, corresponding node weights are assigned, and image nodes are selected and transferred according to the node weights, to obtain image templates in the target image visual template library that meet loop conditions, to optimize the target image visual template library.

[0018] Further, the pose nodes are assigned corresponding node weights, and image nodes are selected and transferred according to the node weights, to obtain image templates in the target image visual template library that meet loop conditions, including:

[0019] According to the node weights, a target pose node corresponding to a target weight value is selected, and the target pose node is compared with the pose data of the current position image, to update the node weights;

[0020] According to the updated node weights, the pose nodes are screened and transferred, and the images corresponding to the screened and transferred pose nodes are used as loop detection image visual templates.

[0021] Further, it further includes:

[0022] The RTAB-MAP performs loop closure detection on the pose node through a discrete Bayesian filter to obtain a loop closure probability and a detection time;

[0023] Determine whether the loop probability and the detection time exceed preset corresponding thresholds respectively, and obtain a determination result;

[0024] The pose nodes are transferred according to the judgment result to optimize the target image visual template library.

[0025] Furthermore, the calculating of the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtaining the distance radius of the robot movement based on the binocular visual odometry under a first preset condition, includes:

[0026] When the robot moves, the deflection angle between the starting node of the moving path and the pose node of the current position image acquired by the binocular camera is determined to determine whether the deflection angle exceeds the angle threshold;

[0027] Under a first preset condition, the ATE error of the robot's moving distance is calculated by the binocular visual odometer of the ORB-SLAM2, and the distance radius is calculated according to the ATE error.

[0028] Furthermore, selecting an image template from the visual template library of the image to be loop detected that meets a second preset condition according to the deflection angle and the distance radius to perform loop detection includes:

[0029] In the first round of screening, when the deflection angle exceeds the angle threshold, the corresponding image is subjected to loop closure detection;

[0030] In the second round of screening, when loop closure detection is started, the image visual template within the distance radius in the image visual template library to be loop closure detected is selected as the template for the loop closure detection to perform loop closure detection.

[0031] Another embodiment of the present invention provides a structured scene-based visual SLAM loop detection system, comprising:

[0032] The initial data acquisition module is used to collect image information of the target road in real time through the robot's binocular camera, process the image information through the binocular visual odometry of ORB-SLAM2, and obtain image key frames and initial pose data;

[0033] An image template library construction module is used to identify and process the image key frames using a trained convolutional neural network model to construct a target image visual template library;

[0034] An image template library optimization module is used to send the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be loop detected;

[0035] a calculation module, configured to calculate a deflection angle of a robot movement path corresponding to a position image acquired by the binocular camera relative to a starting point, and to obtain a distance radius of movement of the robot based on the binocular visual odometry under a first preset condition;

[0036] The loop detection module is used to select an image template from the visual template library of the image to be loop detected that meets the second preset condition based on the deflection angle and the distance radius to perform loop detection. When a loop is detected, global optimization is performed to obtain a globally consistent map.

[0037] Another embodiment of the present invention provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the visual SLAM loop detection method based on structured scenes as described above is implemented.

[0038] Yet another embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the visual SLAM loop detection method based on structured scenes as described above is implemented.

[0039] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0040] (1) The binocular visual odometry part of ORB-SLAM2 is used to estimate the initial pose of the robot and extract image key frames. The image classification and recognition is performed by constructing a neural convolutional network to realize the detection of road intersections. It can efficiently identify road intersection data in structured scenes and use image data as a visual template for loop detection, effectively reducing the number of visual templates involved in loop detection and improving computational efficiency and accuracy.

[0041] (2) Use RTAB-MAP to optimize the image data, select the visual templates involved in loop closure detection, and formulate a suitable visual template selection strategy. Combining the characteristics of ORB-SLAM2 and RTAB-MAP can improve the efficiency of loop closure detection and reduce space storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 11 is a flow chart of a method for visual SLAM loop detection based on structured scenes in one embodiment of the present invention;

[0043] Figure 2 This is a topological model diagram of an urban road intersection in one embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of a situation where a road intersection exists in a city map in one embodiment of the present invention;

[0045] Figure 4 This is a diagram of the overall process framework of visual SLAM loop detection in one embodiment of the present invention;

[0046] Figure 5 1 is a neural network framework diagram of ResNet-34 in one embodiment of the present invention;

[0047] Figure 6 2 is a graph showing the loss and accuracy of a ResNet-34 network in one embodiment of the present invention;

[0048] Figure 7 is a structural diagram of the RTAB-MAP memory management model in one embodiment of the present invention;

[0049] Figure 8 is a schematic diagram of a robot moving path in one embodiment of the present invention; wherein, Figure 8 Includes (a) and (b);

[0050] Figure 9 1 is a block diagram of a structured scene-based visual SLAM loop detection system in one embodiment of the present invention;

[0051] Figure 10 It is a structural block diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0053] In the description of this application, the terms "first," "second," "third," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first," "second," "third," etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0054] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the two components. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are for illustrative purposes only, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0055] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meanings as those commonly understood by those skilled in the art. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Those skilled in the art will understand the specific meanings of the above terms in this application in specific circumstances.

[0056] An embodiment of the present invention provides a method. For details, see Figure 1 , Figure 1 The flowchart of the method for visual SLAM loop detection based on structured scenes in one embodiment of the present invention is shown, which includes the following steps:

[0057] S1. The robot's binocular camera collects image information of the target road in real time, and the ORB-SLAM2 binocular visual odometry processes the image information to obtain image key frames and initial pose data.

[0058] Visual SLAM (Simultaneous Localization and Mapping) is a technology that uses cameras as sensors to simultaneously locate and map an unknown environment. Visual SLAM uses cameras to capture images of the surrounding environment and uses computer vision and machine learning algorithms to process and analyze these images. This enables real-time positioning of mobile platforms such as robots, unmanned vehicles, and drones, as well as mapping of their surroundings.

[0059] ORB-SLAM2 is a complete SLAM solution for monocular, binocular, and RGB-D cameras. It enables map reuse, loop closure detection, and relocalization. Whether used in small indoor handheld devices, drones in factory environments, or cars driving in cities, ORB-SLAM2 can run in real time on standard CPUs. Furthermore, ORB-SLAM2 includes a lightweight localization mode that uses visual odometry to track unmapped areas and match feature points.

[0060] Preferably, in this embodiment, a robot (drone, unmanned vehicle) equipped with a camera device explores the external road environment. While driving or flying on the target road, the binocular camera continuously collects image information of the surrounding road. These images contain rich intersection information and serve as the basis for subsequent processing and positioning. ORB-SLAM2 uses a pose estimation method based on visual odometry to calculate the camera's pose changes by matching feature points between adjacent frames, thereby obtaining image key frames and the robot's initial pose data.

[0061] The embodiment of the present invention uses the ORB-SLAM2 binocular visual odometry to estimate the pose, which can quickly calculate the camera's pose changes in scenarios with high real-time requirements. The road image information continuously collected by the robot contains rich road intersection information, such as intersections, T-junctions, and roundabouts. Therefore, the ORB-SLAM2 visual odometry can effectively ensure the accuracy of image acquisition.

[0062] S2. Recognize and process the image key frames through the trained convolutional neural network model to construct a target image visual template library.

[0063] Furthermore, the road intersection attributes of the image in the image key frame are identified and classified by the trained convolutional neural network model to obtain a target road intersection attribute image;

[0064] Based on the target road intersection attribute image, a target image visual template library is constructed.

[0065] Specifically, for the road intersection detection problem in the visual SLAM of the present embodiment, the core task is to identify whether there is a road intersection in the image, that is, to identify the road intersection attribute in the image key frame. Specifically, please refer to Figure 2 , Figure 2 A city road intersection topology model in one embodiment of the present application is shown.

[0066] In the present embodiment, it is necessary to detect Figure 2 the "T-shaped" and "cross-shaped" road intersections shown in models ③, ④, ⑤, ⑥, etc. in FIG. 1. Since the deep learning method has super strong ability in image classification and recognition, and thanks to the deep convolutional neural network which can automatically extract hierarchical feature representation from raw image data, the present embodiment considers using a deep convolutional neural network model to solve the detection problem of road intersections in structured scenes. For specific structured scene road intersection conditions, please refer to Figure 3 , Figure 3 A schematic diagram of the presence of road intersections in a city map in one embodiment of the present application is shown. As can be seen from Figure 3 in the structured scene of city roads, most of the locations where loops exist are near road intersections, so it can be understood that the target road intersection attribute image in the present embodiment refers to the road intersection attribute image where loops may exist.

[0067] Specifically, the deep convolutional neural network uses a deep network in an end-to-end manner to realize picture classification and recognition, and has super strong image classification and recognition ability. In the embodiments of the present application, the visual classification and recognition algorithm can be selected from the Alexnet, VGG and ResNet series neural networks. Among them, ResNet is a deep residual network model, which solves the gradient vanishing problem of deep network by introducing residual connection, so as to realize a deeper network structure. The classification accuracy of ResNet on the ImageNet dataset is much higher than that of other models, and it has become a benchmark model in the field of image recognition. At the same time, ResNet is also widely used in target detection, semantic segmentation and other computer vision tasks.

[0068] Therefore, preferably, for the visual SLAM loop detection of structured scenes, the present embodiment uses ResNet-34 network to identify road intersections, and the identified road intersection position data is used as a visual template library for loop detection, that is, the target image visual template library. Unlike Alexnet and VGG, ResNet-34 network uses a deep residual learning framework to solve the gradient vanishing or explosion, and has good calculation efficiency and accuracy performance, so the present embodiment uses ResNet-34 for image road intersection recognition and detection. Specifically, please refer to Figure 4 , Figure 4The figure shows the overall process framework of visual SLAM loop detection in one embodiment of the present invention. Figure 4 It can be seen that this embodiment inputs the key frame image output by the ORB-SLAM2 algorithm into ResNet-34 for recognition and classification.

[0069] Preferably, this embodiment is tested on 22 sequences of image sequence datasets KITTI-00 and KITTI-21 containing road intersections. For details, please refer to Figure 5 and Figure 6 , Figure 5 Shown is a neural network framework diagram of ResNet-34 in one embodiment of the present invention, Figure 6 Shown is a graph of the Loss and Acc of the ResNet-34 network in one embodiment of the present invention.

[0070] It can be understood that after using ResNet-34 to identify and classify key frame images, attribute images of road intersections where loops may exist are obtained. These images constitute a target image visual template library. This embodiment generates an image visual template library by using a deep residual network model to identify images, which can reduce storage pressure and improve detection efficiency during loop detection.

[0071] The embodiment of the present invention adopts the advanced ResNet-34 algorithm to construct a deep residual network model to identify the attribute features of image road intersections. By taking advantage of the high speed and accuracy of ResNet-34, it can efficiently identify tool path intersection data in structured urban scenes, and use these image data as visual templates for loop detection, that is, the image visual template library to be screened by RTAB-MAP.

[0072] S3. Send the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain the image visual template library to be loop detected.

[0073] RTAB-Map (Real-time Appearance-based Mapping) is a powerful, open-source Simultaneous Localization and Mapping (SLAM) library designed specifically for robotics and autonomous driving. It provides a complete solution for building 3D maps and performing real-time localization and navigation in unknown environments.

[0074] Research has shown that RTAB-MAP plays a vital role in loop closure detection. It optimizes and enhances the performance of the SLAM system by providing a mechanism for creating, selecting, and transferring nodes to control the number of visual templates for loop closure detection.

[0075] In an embodiment of the present invention, in order to enable the robot to have the ability to operate on a large scale and for a long time, this embodiment uses RTAB-MAP to perform loop detection. RTAB-MAP detects whether the robot has returned to a previously visited location, i.e., a loop, by comparing the similarity between the current environment observation and the historical observation.

[0076] In RTAB-MAP's memory management, short-term memory (STM), working memory (WM), and long-term memory (LTM) are three key modules. For details, please refer to Figure 7 , Figure 7 Shown is a structural diagram of the RTAB-MAP memory management model in one embodiment of the present invention.

[0077] according to Figure 7 As can be seen, the perception storage module receives image keyframe data captured by the binocular camera and sends the initial pose data to the short-term memory module. It is worth noting that the perception memory of the perception storage module processes the received image keyframes to reduce the data dimension. This step aims to extract useful features for subsequent loop closure detection. The robot's current pose is then estimated based on these image features. Both types of data are transferred to the short-term memory of RTAB-MAP.

[0078] Furthermore, this embodiment sends the initial pose data to the short-term memory module of the RTAB-MAP to create a pose node of the corresponding image. Specifically:

[0079] In this embodiment, when information is transmitted from the front-end binocular visual odometry and the road intersection, the short-term memory module generates a new node to store the odometry pose information, the raw sensor data, and additional information that is helpful to subsequent processing modules. The node output by the short-term memory module is subject to loop closure detection.

[0080] Corresponding node weights are assigned according to the pose nodes, and image nodes are selected and transferred according to the node weights to obtain image templates that meet the loop conditions in the target image visual template library, so as to optimize the target image visual template library.

[0081] For more information on how to select nodes, please refer to Figure 4Right side. This embodiment assigns weights to nodes generated in short-term memory. When transferring from working memory to long-term memory, the location with the smallest weight is selected. If multiple locations have the same weight, the oldest location is selected for transfer. When a node is transferred to long-term memory, it is deleted from working memory. This allows loop detection to select the node most likely to have a loop.

[0082] The following is a detailed description of the processing process of the above nodes:

[0083] In this embodiment, a target pose node corresponding to a target weight value is selected according to the node weight, and the target pose node is compared with the pose data of the current position image to update the node weight; according to the updated node weight, the pose node is screened and transferred, and the image corresponding to the screened and transferred pose node is used as the visual template of the image to be loop detected.

[0084] like Figure 7 As shown, this process requires the coordinated action of RTAB-MAP's short-term memory, working memory, and long-term memory. Specifically:

[0085] Short-term memory updates recently created pose nodes through a weight update process, which updates pose node weights based on temporal similarities observed between successive images. If the weight update identifies a new position as similar to a previous position in short-term memory, it merges them into the new position and increases the weight of the new position. By assigning weights to nodes and performing weight updates, this embodiment facilitates subsequent spatial selection of nodes with higher weights for storage in the system, thereby streamlining spatial storage.

[0086] After merging, the weight of the new position will be increased to reflect its higher importance and richer information. By merging similar nodes, we can avoid creating a large number of repeated, almost identical positions in the map, thereby reducing the waste of storage space. The merging process is usually accompanied by pose optimization, which helps to reduce pose drift caused by noise or errors and improve the accuracy of the map.

[0087] Working memory is responsible for detecting loop closures between spatial locations and for domain detection. During pose node creation, when the number of nodes stored in short-term memory reaches a threshold, the oldest positions in short-term memory are moved to working memory. Processing fewer, more accurate nodes significantly improves algorithm efficiency in subsequent loop detection and map optimization processes.

[0088] When the RTAB-MAP system's update time or the number of nodes in working memory exceeds a threshold, some nodes in working memory are transferred to long-term memory. A weighted mechanism is then used to transfer some useful information back to working memory. Pose nodes in long-term memory are not included in loop closure detection.

[0089] The embodiment of the present invention creates, selects and transfers nodes of the pose information of an image by utilizing the different functions of the short-term memory, working memory and long-term memory of RTAB-MAP, thereby realizing the processing processes of creation, transfer and retrieval of pose nodes, and uses the image corresponding to the screened and transferred pose node as the visual template of the image to be loop closure detected.

[0090] Preferably, in this embodiment, the node quantity threshold is set by the speed of the robot and the speed of acquiring the position.

[0091] Furthermore, in this embodiment, the RTAB-MAP is used to detect loop closure of the pose node through a discrete Bayesian filter to obtain a loop probability and a detection time.

[0092] Specifically, RTAB-MAP estimates the loop closure probability by comparing the new location with the locations in working memory using a discrete Bayesian filter. A loop closure is detected if there is a high probability of a loop between the new location and the old location in working memory. In other words, a higher probability of loop closure corresponds to a higher probability of a loop.

[0093] Then, it is determined whether the loop probability and the detection time exceed the preset corresponding thresholds, and a judgment result is obtained. The posture node is transferred according to the judgment result to optimize the target image visual template library.

[0094] The following is a detailed description of the optimization process of the target image visual template library:

[0095] In this embodiment, the size of the working memory should preferably be kept within the limits that the Bayesian filter can handle. There are two key steps for RTAB-MAP to achieve loop detection and control the number of visual templates for loop detection to achieve optimization of the visual template library of the target image:

[0096] The first step is retrieval. Using the estimated loop closure probability, nodes are brought back from long-term memory to working memory for nearby locations that are not in working memory and have a high probability of loop closure. This increases the number of visual templates detected. While this step appears to increase the number of templates, it is based on nodes with high loop closure probabilities, thus increasing the probability of loop closure in future nearby locations.

[0097] The second step is transfer. If the time spent processing loop closure detection exceeds a time threshold, the oldest node with the least number of views is transferred to long-term memory, reducing the number of visual template detections. The number of transferred locations depends on the number of locations added to working memory in the current cycle.

[0098] It is worth noting that this embodiment integrates ORB-SLAM2 into RTAB-MAP. Since the visual odometry in RTAB-MAP is independent of the mapping process, it is necessary to disable the loop detection in ORB-SLAM2. The estimated value of its visual odometry is output to RTAB-MAP to complete subsequent loop detection and global optimization processes, and to handle issues such as spatial storage.

[0099] S4. Calculate the deflection angle of the robot's moving path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtain the distance radius of the robot's movement based on the binocular visual odometer under a first preset condition.

[0100] The embodiment of the present invention improves the selection strategy for selecting visual templates for loop closure detection. Specifically:

[0101] When the robot moves, the deflection angle between the starting node of the moving path and the pose node of the current position image acquired by the binocular camera is used to determine whether the deflection angle exceeds the angle threshold. For details, please refer to Figure 8 , Figure 8 FIG. 1 is a schematic diagram of a robot moving path in one embodiment of the present invention. Figure 8 Includes (a) and (b).

[0102] As can be seen from (a), the path of the robot moving from point A1 to point B1 is similar to a straight line; and the path of the robot moving from point A2 to point B2 shows that the deflection angle between the starting point and the end point, that is, between the starting node and the pose node of the current position image, is less than 360°. In this case, it is impossible for a loop to exist. Therefore, for the route (a), there is no need to consider whether there is a loop.

[0103] As can be seen from (b), the deflection angle of the robot's path from point A3 to point B3 exceeds a certain threshold. In this case, a loop is likely to occur on the route (b).

[0104] Therefore, the embodiment of the present invention can efficiently determine the images that need to be loop-closed detected by calculating and judging the deflection angle, laying the foundation for starting the subsequent efficient and feasible loop-closed detection process.

[0105] Furthermore, in this embodiment, under the first preset condition, the ATE error of the robot's moving distance is calculated by the binocular visual odometry of the ORB-SLAM2, and the distance radius is calculated based on the ATE error.

[0106] As you can understand, ATE (Absolute Trajectory Error) is the absolute trajectory error, which is an important indicator for measuring the performance of SLAM systems. ATE error represents the deviation between the estimated trajectory and the true trajectory. In this step, as you can understand, the distance radius is calculated in advance before the system is run. That is, the first preset condition is: when there is no loop in ORB-SLAM2, the distance radius is further calculated based on the segmented ATE accuracy of the KITTI dataset.

[0107] Specifically, based on RTAB-MAP, the embodiment of the present invention combines the characteristics of urban road environment and visual binocular SLAM to calculate the segmented ATE accuracy of ORB-SLAM2 on the KITTI dataset when there is no loop, as shown in Table 4-1 below:

[0108] Table 4-1 ORB-SLAM2 segmented ATE accuracy in KITTI dataset without loop closure / m

[0109]

[0110] Table 4-1 shows that the robot's travel distance is correlated with the ATE error. This embodiment of the present invention utilizes ATE accuracy to calculate the distance radius of visual templates based on a library of visual templates for loop closure detection. Specifically, a distance radius is determined based on a certain error accuracy, and visual templates within this radius are subject to loop closure detection. This step also allows for more precise selection of visual templates with potential loop closures.

[0111] S5. Select an image template from the visual template library of the image to be loop detected that meets a second preset condition according to the deflection angle and the distance radius to perform loop detection. When a loop is detected, perform global optimization to obtain a globally consistent map.

[0112] It is understandable that the purpose of loop closure detection is to eliminate the cumulative error of the SLAM front-end odometry estimation. The embodiment of the present invention combines ORB-SLAM2 with RTAB-MAP to fully utilize the performance advantages of the ORB-SLAM2 front-end visual odometry and the memory management advantages of the RTAB-MAP back-end.

[0113] Loop closure detection involves comparing the image captured in real time by the binocular camera with the image templates in the visual template library of the image to be looped. A feature matching algorithm is then used to find similar feature points between the image to be detected and the images in the template library, calculating the similarity and performing loop closure detection. Because the embodiments of the present invention have optimized the templates in the visual template library of the image to be looped through the above steps, loop closure detection is highly efficient.

[0114] In this embodiment, when performing loop closure detection, a key step is required to develop a more reasonable visual template selection strategy and select an appropriate template for loop closure detection. Therefore, it can be understood that in this step, the second preset condition is to select an image visual template that meets the angle threshold and falls within the set distance radius. Specifically, this embodiment includes two rounds of screening processes:

[0115] In the first round of screening, when the deflection angle exceeds the angle threshold, the corresponding image is used for loop detection.

[0116] In an embodiment of the present invention, it is necessary to determine whether the deflection angle of the robot's moving path meets the preset conditions. Preferably, in order to maintain higher detection efficiency when the ORB-SLAM2 binocular vision angle deflection error is small, the preset angle threshold is 270°. When the deflection angle is less than 270 degrees, it can be determined that the corresponding image vision template does not need to start loop detection. When the yaw angle is greater than 270 degrees, the subsequent loop detection needs to be started for the corresponding image vision template. It can be understood that the first round of screening process is the process of determining whether loop detection is needed.

[0117] In the second round of screening, when loop closure detection is started, the image visual template within the distance radius in the image visual template library to be loop closure detected is selected as the template for the loop closure detection to perform loop closure detection.

[0118] According to the segmented ATE accuracy of ORB-SLAM2 in the KITTI dataset without loop closure shown in Table 4-1, it can be seen that as the running distance increases, the cumulative error of SLAM without loop closure correction will become larger and larger.

[0119] Preferably, according to the characteristics of the intersection of structured urban roads, when loop detection is enabled, the distance radius of the visual template is selected based on the 3 times value of the 5000-meter length ATE error of the ORB-SLAM2 binocular visual odometer in KITTI-00. That is, the template within 21 meters of the stored visual template from the current vehicle is selected as the reference image for loop detection. For specific examples, please refer to Figure 7 It can be understood that the second round of screening process is a process of further screening the visual image template to be loop detected based on a specific range.

[0120] It can be understood that in an embodiment of the present invention, an image visual template library is constructed by accurately selecting a visual template strategy for SLAM loop detection. When a loop is detected, global optimization is performed to construct a globally consistent map, which can not only meet real-time requirements but also access the location of the entire map when necessary, thereby speeding up the loop detection and greatly reducing the pressure on memory space.

[0121] In summary, the embodiment of the present invention detects image road intersections through the deep residual neural network ResNet-34, which can extract rich image features and is very effective in distinguishing intersection data in different urban scenes. It greatly reduces the number of visual templates and helps to build a high-quality visual template library. It also combines the estimation performance of the binocular visual odometry of ORB-SLAM2 in the KITTI dataset when there is no loop closure and the advantages of RTAB-MAP in back-end optimization and memory management. It can more effectively manage map data and formulate a more reasonable visual template selection strategy to select appropriate templates for loop detection, thereby reducing the system's memory usage and optimizing the consistency of the global map, while improving the stability and robustness of the entire SLAM system.

[0122] An embodiment of the present invention provides a visual SLAM loop detection system based on structured scenes. For details, please refer to Figure 9 , Figure 9 The figure shows a structural block diagram of a visual SLAM loop detection system based on structured scenes in one embodiment of the present invention.

[0123] The initial data acquisition module M1 is used to collect image information of the target road in real time through the robot's binocular camera, process the image information through the binocular visual odometry of ORB-SLAM2, and obtain image key frames and initial pose data;

[0124] An image template library construction module M2 is used to perform recognition processing on the image key frames through a trained convolutional neural network model to construct a target image visual template library;

[0125] An image template library optimization module M3 is used to send the initial pose data to RTAB-MAP to create and transfer image pose nodes, so as to optimize the target image visual template library and obtain a visual template library for the image to be looped;

[0126] a calculation module M4, configured to calculate a deflection angle of the robot's moving path corresponding to the position image acquired by the binocular camera relative to a starting point, and to obtain a distance radius of the robot's movement based on the binocular visual odometry under a first preset condition;

[0127] The loop detection module M5 is used to select an image template from the visual template library of the image to be loop detected that meets the second preset condition according to the deflection angle and the distance radius to perform loop detection. When a loop is detected, global optimization is performed to obtain a globally consistent map.

[0128] like Figure 10 As shown, an embodiment of the present invention further provides a computer device, Figure 10 A structural block diagram of a preferred embodiment of a computer device provided by the present invention, wherein the computer device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the above-mentioned visual SLAM loop detection method based on structured scenes when executing the computer program.

[0129] Preferably, the computer program can be divided into one or more modules / units (e.g., computer program 1, computer program 2, ...), which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0130] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor. The processor is the control center of the terminal device, and uses various interfaces and lines to connect the various parts of the terminal device.

[0131] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application program required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card, etc., or the memory can also be other volatile solid-state storage devices.

[0132] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 10 The structural block diagram is only an example of a terminal device and does not constitute a limitation of the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0133] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to perform the steps in the visual SLAM loop detection method based on structured scenes as described in the above embodiment, for example Figure 1 Steps S1 to S5 described in .

[0134] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A visual SLAM loop detection method based on structured scenes, characterized in that: include: The robot's binocular camera collects image information of the target road in real time, and the image information is processed by the ORB-SLAM2 binocular visual odometry to obtain image key frames and initial pose data; The image key frames are recognized and processed by a trained convolutional neural network model to construct a target image visual template library; The initial pose data is sent to the RTAB-MAP to create and transfer image pose nodes, specifically: the initial pose data is sent to the short-term memory module of the RTAB-MAP to create corresponding image pose nodes; corresponding node weights are assigned according to the pose nodes, and the target pose node corresponding to the target weight value is selected according to the node weight, and the target pose node is compared with the pose data of the current position image to update the node weight; the pose nodes are screened and transferred according to the updated node weights, and the images corresponding to the screened and transferred pose nodes are used as the image visual templates to be detected by loop closure, and the image templates that meet the loop closure conditions in the target image visual template library are obtained; wherein, the RTAB-MAP performs loop closure detection on the pose nodes through a discrete Bayesian filter to obtain the loop closure probability and detection time; it is judged whether the loop closure probability and the detection time exceed the preset corresponding thresholds respectively to obtain a judgment result; the pose nodes are transferred according to the judgment result to optimize the target image visual template library to obtain the image visual template library to be detected by loop closure; Calculating the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtaining the distance radius of the robot movement based on the binocular visual odometry under a first preset condition; According to the deflection angle and the distance radius, an image template in the visual template library of the image to be loop detected that meets the second preset condition is selected to perform loop detection. When a loop is detected, global optimization is performed to obtain a globally consistent map.

2. The visual SLAM loop detection method based on structured scenes as claimed in claim 1, wherein The image key frames are recognized and processed by the trained convolutional neural network model to construct a target image visual template library, including: Identifying and classifying the attributes of the intersection in the image key frame using a trained convolutional neural network model to obtain an attribute image of the target intersection; Based on the target road intersection attribute image, a target image visual template library is constructed.

3. The visual SLAM loop detection method based on structured scenes as claimed in claim 1, wherein The calculating of the deflection angle of the robot movement path corresponding to the position image acquired by the binocular camera relative to the starting point, and obtaining the distance radius of the robot movement based on the binocular visual odometer under a first preset condition, includes: When the robot moves, the deflection angle between the starting node of the moving path and the pose node of the current position image acquired by the binocular camera is determined to determine whether the deflection angle exceeds an angle threshold; Under a first preset condition, the ATE error of the robot's moving distance is calculated by the binocular visual odometer of the ORB-SLAM2, and the distance radius is calculated according to the ATE error.

4. The visual SLAM loop detection method based on structured scenes as claimed in claim 3, wherein The step of selecting, based on the deflection angle and the distance radius, an image template from a visual template library of the image to be loop detected that meets a second preset condition to perform loop detection includes: In the first round of screening, when the deflection angle exceeds the angle threshold, the corresponding image is used for loop closure detection; In the second round of screening, when loop closure detection is started, the image visual template within the distance radius in the image visual template library to be loop closure detected is selected as the template for the loop closure detection to perform loop closure detection.

5. A visual SLAM loop detection system based on structured scenes, characterized in that: include: The initial data acquisition module is used to collect image information of the target road in real time through the robot's binocular camera, process the image information through the binocular visual odometry of ORB-SLAM2, and obtain image key frames and initial pose data; An image template library construction module is used to identify and process the image key frames using a trained convolutional neural network model to construct a target image visual template library; An image template library optimization module is used to send the initial pose data to the RTAB-MAP to create and transfer image pose nodes, specifically: sending the initial pose data to the short-term memory module of the RTAB-MAP to create corresponding image pose nodes; assigning corresponding node weights according to the pose nodes, and selecting target pose nodes corresponding to target weight values ​​according to the node weights, and comparing the target pose nodes with the pose data of the current position image to update the node weights; screening and transferring the pose nodes according to the updated node weights, using the images corresponding to the screened and transferred pose nodes as the image visual templates to be detected by loop closure, and obtaining image templates that meet the loop closure conditions in the target image visual template library; wherein the RTAB-MAP performs loop closure detection on the pose nodes through a discrete Bayesian filter to obtain a loop closure probability and a detection time; respectively judging whether the loop closure probability and the detection time exceed preset corresponding thresholds to obtain a judgment result; transferring the pose nodes according to the judgment result to optimize the target image visual template library to obtain an image visual template library to be detected by loop closure; a calculation module, configured to calculate a deflection angle of a robot movement path corresponding to a position image acquired by the binocular camera relative to a starting point, and to obtain a distance radius of movement of the robot based on the binocular visual odometry under a first preset condition; The loop detection module is used to select an image template from the visual template library of the image to be loop detected that meets the second preset condition based on the deflection angle and the distance radius to perform loop detection. When a loop is detected, global optimization is performed to obtain a globally consistent map.

6. A computer device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the visual SLAM loop detection method based on structured scenes according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the structured scene-based visual SLAM loop detection method according to any one of claims 1 to 4 is implemented.

Images