Low-light image enhancement methods, and visual simultaneous localization and mapping methods for low-light environments.
By enhancing the brightness of low-light images and extracting ORB feature points, combined with Bundle Adjustment optimization and Spanning Tree optimization, the localization accuracy and robustness issues of visual SLAM in low-light environments are solved, achieving robust visual simultaneous localization and mapping.
Patent Information
- Application Number
- CN202311693663.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-11
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-12-11
AI Technical Summary
In low-light environments, the positioning accuracy of visual SLAM methods is greatly reduced, making it difficult to meet the requirements for outdoor and nighttime operations. Furthermore, the lack of active lighting indoors increases operating costs and wastes resources.
A lightweight convolutional neural network is used to enhance the brightness of low-light images. Combined with ORB feature point extraction and Bundle Adjustment optimization, and using spanning tree optimization for closed-loop correction, the robustness and real-time performance of localization and mapping are improved.
Robust visual synchronous localization and mapping were achieved under low-light conditions, improving positioning accuracy and robustness, reducing the running time of closed-loop correction, and enhancing the real-time performance and tracking stability of the system.
Smart Images

Figure CN117474773B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a low-light image enhancement method and a visual synchronous localization and mapping method for low-light environments, belonging to the fields of mobile robots and computer vision. Background Technology
[0002] Simultaneous localization and mapping (SLAM) is a core technology in fields such as robotics and autonomous driving. Its purpose is to enable autonomous mobile hardware platforms to recognize their environment and estimate their position within it.
[0003] Various sensors can be used in SLAM tasks, such as LiDAR, cameras, and ultrasonic sensors. In vision-based methods, localization and mapping use only the camera as the information source, without using LiDAR or other active sensors. Compared to SLAM methods using other sensors, camera-only vision SLAM methods have the lowest cost. More importantly, the visual images acquired by the camera can provide richer semantic and textural information. Semantic information includes the position, orientation, color, texture, shape, and specific attributes of objects in the environment. This information can help robots improve their accurate perception and adaptation to unknown and complex environments, enabling them to perform more complex tasks. For example, semantic vision SLAM improves the accuracy and robustness of localization by leveraging semantic constraints. In mapping tasks, semantic vision SLAM provides rich object information to construct different types of semantic maps, such as pixel-level maps and object-level maps. When autonomous mobile robots need to interact with the real environment, understand their surroundings, and associate them with human commands, the semantic information provided by vision methods is indispensable. However, using only cameras as sensors means relying heavily on the quality of the acquired images. Due to the unpredictable and unpredictable lighting conditions in outdoor environments, and the significant reduction in positioning accuracy in low light, this limitation is one of the main obstacles to applying visual SLAM algorithms to outdoor mobile robots. Therefore, the reliability of such algorithms is insufficient for outdoor and nighttime operations. Similarly, using visual algorithms in indoor conditions is challenging because fully automated, unattended work environments often lack active lighting, as insufficient illumination would significantly increase operating costs and waste resources.
[0004] In view of this, the present invention is hereby proposed. Summary of the Invention
[0005] This invention provides a low-light image enhancement method and a visual synchronous localization and mapping method for low-light environments. On the one hand, it can be used to enhance video image frames acquired in real time under low light conditions. On the other hand, it can be optimized based on the enhanced image to complete visual synchronous localization and mapping.
[0006] The technical solution of this invention is:
[0007] According to a first aspect of the present invention, a low-light image enhancement method is provided, comprising: publishing a video acquired in real time under low light conditions as a ROS low-light image message based on the ROS robot operating system; converting the low-light image into the HSV color space; using the V channel and desired brightness of the low-light image as input to a lightweight convolutional neural network, and estimating the global brightness adjustment V channel histogram coefficients using a high-order curve adjustment method in a five-layer multilayer perceptron; concatenating the histogram coefficients and feeding them into a convolutional layer with a local receptive field to obtain an enhanced V channel; replacing the original V channel of the low-light image with the enhanced V channel and performing RGB color space conversion to obtain the enhanced image.
[0008] According to a second aspect of the present invention, a low-light image enhancement system is provided, comprising: a processing module for publishing video acquired in real time under low light conditions as a ROS low-light image message based on the ROS robot operating system; a conversion module for converting the low-light image into an HSV color space; and an acquisition module for taking the V channel and desired brightness of the low-light image as input to a lightweight convolutional neural network, estimating the global brightness adjustment V channel histogram coefficients using a high-order curve adjustment method in a five-layer multilayer perceptron; the histogram coefficients are concatenated and fed into a convolutional layer with a local receptive field to obtain an enhanced V channel; the enhanced V channel replaces the original V channel of the low-light image and is converted to RGB color space to obtain an enhanced image.
[0009] According to a third aspect of the present invention, a method for visual synchronous localization and mapping in low-light environments is provided, comprising: extracting ORB feature points from an enhanced image, performing feature matching between adjacent frames, and triangulating the matched feature points to generate 3D map points; estimating the initial pose using epipolar geometry for the first and second frames and initializing the second frame image as a reference keyframe, creating an initial map based on the estimated initial pose and 3D map points of the first and second frames, and optimizing the estimated initial pose for the first time using Bundle Adjustment; based on the initial map, inputting a new image frame each time, performing feature matching between adjacent frames, and triangulating the matched feature points to generate 3D map points; estimating the initial pose of the current frame image using the PnP method, expanding the map using the currently obtained 3D map points and the estimated initial pose to generate a local map; further optimizing the estimated initial pose of the current frame in the local map using Bundle Adjustment for the first time; determining whether to create a new keyframe for the current frame based on keyframe creation conditions; and, if the current frame is a keyframe, performing keyframe filtering and elimination, determining the number of co-view keyframes, and then using Bundle Adjustment... The adjustment performs a second optimization and updates the spanning tree; after detecting a loop closure candidate keyframe, loop closure correction is performed; after loop closure correction, the spanning tree is used for Bundle Adjustment optimization.
[0010] The method of determining whether to create a new keyframe based on the keyframe creation conditions is as follows: the method is to determine whether to create a new keyframe based on whether the 3D map points of the current frame reach a preset value, the degree of overlap with the previous keyframe, and whether there is a preset parallax with the previous keyframe. If all three conditions are met, the current frame is created as a keyframe; otherwise, it is not created.
[0011] The process of filtering and removing keyframes and determining the number of co-observed keyframes, followed by a second optimization using Bundle Adjustment and updating the spanning tree, is as follows: When the current frame is a keyframe, the following checks are performed: If the number of keyframes exceeds three, keyframes are filtered and removed: If at least a preset proportion of 3D map points in the local map are observed by more than three keyframes, only the three keyframes with the highest co-observation relationship and their corresponding 3D map points are retained; If the number of co-observed keyframes reaches n or more, the current keyframe is paired with each of the n-1 co-observed keyframes (excluding the current keyframe) for reprojection feature matching to obtain matching feature point pairs; After matching, the matching feature point pairs are triangulated to generate new first 3D map points; If new 3D map points exist, the new first 3D map points, the 3D map points of the original keyframes in the local map, and their corresponding frames' first optimized poses are combined using Bundle Adjustment. The adjustment process simultaneously performs a second optimization and updates the spanning tree; otherwise, it uses Bundle Adjustment to simultaneously perform a second optimization on the 3D map points of the keyframes in the original local map and the poses of their corresponding frames after the first optimization, and updates the spanning tree.
[0012] A reverse matching scheme is introduced to eliminate mismatched feature point pairs.
[0013] The reverse matching specifically involves: when a feature point of a keyframe is reprojected into any co-view keyframe, descriptor matching is performed in the area near the projection to find matching features, and then the feature point is projected back to the original keyframe. Only when the error between the projected point and the original feature point does not exceed a preset pixel is it selected as a matching feature point pair; otherwise, it is discarded.
[0014] After detecting a candidate keyframe for loop closure, loop closure correction is performed. After loop closure correction, a spanning tree is used for Bundle Adjustment optimization. Specifically, after detecting a candidate keyframe for loop closure in the keyframes before the current frame, a preset number of co-visible keyframes with the highest co-visibility with the current frame and the candidate keyframe for loop closure are used for pairwise feature matching, triangulation to generate new second 3D map points, and the new second 3D map points and the result of the second optimization are used in loop closure correction. After loop closure correction, a spanning tree is used for Bundle Adjustment optimization.
[0015] The specific process of creating the spanning tree is as follows: following the order in which keyframes were created, from back to front, the spanning tree is constructed by retaining only the vertices and edges with the highest degree of shared visibility between any two points with any vertex as the subject, without any repetition. In the spanning tree, the keyframe with the highest degree of shared visibility among the most recently created and retained keyframes is taken as the parent keyframe, and the most recently created and retained keyframe is taken as its child keyframe. If the most recently created and retained keyframe is updated, the parent-child relationship needs to be updated.
[0016] According to a fourth aspect of the present invention, a visual synchronous localization and mapping system for low-light environments is provided, comprising: a first optimization module, configured to extract ORB feature points from an enhanced image, perform feature matching between adjacent frames, and triangulate the matched feature points to generate 3D map points; for the first and second frames of images, an initial pose is estimated using epipolar geometry and the second frame image is initialized as a reference keyframe; an initial map is created based on the estimated initial pose of the first and second frames and the 3D map points of the first and second frames, and the estimated initial pose is optimized for the first time using Bundle Adjustment; based on the initial map, a new image is input each time, feature matching is performed between adjacent frames, and the matched feature points are triangulated to generate 3D map points; the initial pose of the current frame image is estimated using the PnP method, and the map is expanded using the currently obtained 3D map points and the estimated initial pose to generate a local map; and the initial pose of the current frame in the local map is then optimized using Bundle Adjustment. The first optimization is performed by Bundle Adjustment; the first judgment module is used to determine whether a new keyframe should be created for the current frame based on the keyframe creation conditions; if the current frame is a keyframe, keyframe filtering and elimination are performed, the number of shared keyframes is judged, and then Bundle Adjustment is used for the second optimization and the spanning tree is updated; the second judgment module is used to perform loop closure correction after detecting a loop closure candidate keyframe; after loop closure correction, Bundle Adjustment is used for optimization using the spanning tree.
[0017] The beneficial effects of this invention are:
[0018] 1. This invention uses a lightweight convolutional neural network based on the ROS system, which enables the neural network to enhance the brightness of the input video in real time while eliminating noise and correcting color errors. The neural network is embedded as a preprocessing module in the synchronous localization and mapping algorithm, which enables the synchronous localization and mapping algorithm to work robustly in low-light environments.
[0019] 2. This invention uses spanning trees for Bundle Adjustment optimization after loop closure correction, significantly reducing the runtime of the loop closure thread. Using only spanning tree Bundle Adjustment, compared to global Bundle Adjustment optimization after loop closure, greatly improves the algorithm's real-time performance, while the resulting loss in accuracy is acceptable. This invention employs keyframe filtering and map point supplementation strategies, greatly enhancing the tracking robustness of the synchronous localization and mapping system in low-light environments.
[0020] 3. By adopting the technical solution described above, the present invention has higher accuracy compared with the most mainstream visual synchronous localization and mapping method ORB-SLAM3. Attached Figure Description
[0021] Figure 1 This is a flowchart of an embodiment of the present invention;
[0022] Figure 2 These are the predicted trajectories of the present invention and ORB-SLAM3 on the TUM and EuRoC datasets;
[0023] Figure 3 This is a comparison chart of the error curves of the embodiments of the present invention and ORB-SLAM3;
[0024] Figure 4 This describes the actual operation of the embodiment of the present invention and ORB-SLAM3 in a low-light environment;
[0025] Figure 5 This is a diagram of a spanning tree. Detailed Implementation
[0026] The invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited to the description.
[0027] Example 1: As Figure 1-5As shown, according to one aspect of the present invention, a low-light image enhancement method is provided, characterized by comprising: S1, publishing video acquired in real time under low light conditions as a ROS low-light image message based on the ROS robot operating system; S2, converting the low-light image into the HSV color space; using the V channel and desired brightness of the low-light image as input to a lightweight convolutional neural network, estimating the global brightness adjustment V channel histogram coefficients using a high-order curve adjustment method in a five-layer multilayer perceptron; the histogram coefficients are concatenated and fed into a convolutional layer with a local receptive field to obtain the enhanced V channel, the convolutional layer being able to extract local information and generate high-frequency details, replacing the original V channel of the low-light image with the enhanced V channel and performing RGB color space conversion to obtain the enhanced image. The lightweight convolutional neural network comprises a five-layer multilayer perceptron and six convolutional layers; specifically:
[0028] The brightness of the input low-light image is represented using the V channel in the HSV color space. Global information is extracted from the histogram of the V channel of the low-light image, and the histogram coefficients are represented as high-order curve coefficients.
[0029]
[0030] Iteratively adjust the V channel of the low-light image to obtain the histogram coefficients after global brightness adjustment, using an iterative formula:
[0031]
[0032] After obtaining the histogram coefficients, they are further enhanced by connecting them to intermediate layers of the convolutional layers in the network. The final output image is an image that has undergone brightness and contrast enhancement, noise reduction, and color correction. The entire network is trained end-to-end, meaning that all components are jointly trained to optimize the overall performance of the network. The enhanced image is then published as a new ROS message.
[0033] Among them, X k Represents the histogram coefficients obtained in the k-th iteration. Let V represent the V channel of image I in the HSV color space during the k-th iteration. Indicates obtaining an image The histogram operation is described by F(), which represents the global feature extraction layer in FLW, implemented using a five-layer multilayer perceptron. The feature extraction layer can be easily modified to include additional variables to control the degree of image enhancement, such as adding the desired average brightness value μ. In embodiments of the invention, the desired brightness μ is generated using a random function, taking values in the range [0.2, 0.8].
[0034] The total loss function of lightweight convolutional neural networks includes commonly used L1 and SSIM loss functions, as well as color loss, brightness loss, and structure loss. All of these loss functions are combined to form the total loss function.
[0035] According to a second aspect of the present invention, a low-light image enhancement system is provided, comprising: a processing module for publishing video acquired in real time under low light conditions as a ROS low-light image message based on the ROS robot operating system; a conversion module for converting the low-light image into an HSV color space; and an acquisition module for taking the V channel and desired brightness of the low-light image as input to a lightweight convolutional neural network, estimating the global brightness adjustment V channel histogram coefficients using a high-order curve adjustment method in a five-layer multilayer perceptron; the histogram coefficients are concatenated and fed into a convolutional layer with a local receptive field to obtain an enhanced V channel; the enhanced V channel replaces the original V channel of the low-light image and is converted to RGB color space to obtain an enhanced image.
[0036] According to a third aspect of the present invention, a method for visual simultaneous localization and mapping in low-light environments is provided, comprising:
[0037] S3. Perform ORB feature point extraction on the enhanced image frames, perform feature matching between adjacent frames, and triangulate the matched feature points to generate 3D map points. For the first and second frames, use epipolar geometry to estimate the initial pose and initialize the second frame as a reference keyframe. Based on the estimated initial pose of the first and second frames and the 3D map points of the first and second frames, create an initial map and use Bundle Adjustment to optimize the estimated initial pose for the first time.
[0038] S4. Based on the initialized map, each time a new image frame is input, feature matching is performed between adjacent frames. The matched feature points are triangulated to generate 3D map points. The PnP method is used to perform initial pose estimation on the current frame image. The map is then expanded using the obtained 3D map points and the estimated initial pose to generate a local map. The initial pose estimated for the current frame in the local map is then optimized for the first time using Bundle Adjustment. For the first and second frames, epipolar geometry is used for pose estimation, while for images other than the first and second frames, PnP (Perspective-n-Point) is used for pose estimation. Combining the two methods can avoid the inability to estimate the pose of the first and second frames and make the estimation of subsequent poses faster and more accurate.
[0039] S5. The step of determining whether to create a new keyframe based on the keyframe creation conditions specifically involves: determining whether to create a new keyframe based on whether the number of 3D map points in the current frame reaches a preset value, the degree of overlap with the previous keyframe, and whether there is a preset disparity with the previous keyframe. If all three criteria are met, the current frame is created as a keyframe; otherwise, it is not created. In an embodiment of the present invention, if the total number of map points in the current frame is greater than 50, and the number of 3D map points with co-view relationships with the previous keyframe is less than a threshold (less than 85% of the total number of map points in the previous keyframe), or if significant disparity is detected (the average rotation of ORB feature points exceeds 15 pixels), then the current frame is designated as a new keyframe.
[0040] To improve the tracking robustness and real-time performance of synchronous localization and mapping in low-light environments, a judgment is made based on the condition that the current frame is a keyframe: if the number of keyframes exceeds three, keyframes are filtered and discarded; if at least 95% of the 3D map points in the local map are observed by more than three keyframes, only the three keyframes with the highest co-view relationship and their corresponding 3D map points are retained; if the number of co-view keyframes reaches n or more (in the example of this invention, n can be set to 20), the current keyframe is reprojected and matched pairwise with each of the n-1 co-view keyframes with the highest co-view relationship (excluding the current keyframe) to obtain matching feature point pairs; in this step... The process introduces a reverse matching scheme to eliminate mismatched features. Specifically, when a feature point of a keyframe is reprojected into any co-view keyframe, descriptor matching is performed in the vicinity of the projection area to find matching features. Then, the feature point is projected back to the original keyframe. Only when the error between the projected point and the original feature point is no more than one pixel is it selected as a matching feature point pair; otherwise, it is discarded. After matching, the matching feature point pairs are triangulated to generate new first 3D map points. If new 3D map points exist, the new first 3D map points, the 3D map points of the original keyframes in the local map, and their corresponding frames' first optimized poses are simultaneously optimized a second time using Bundle Adjustment, and the spanning tree is updated. Otherwise, the 3D map points of the original keyframes in the local map and their corresponding frames' first optimized poses are simultaneously optimized a second time using Bundle Adjustment, and the spanning tree is updated.
[0041] S6. Similar to S5, after detecting a loop closure candidate keyframe in the keyframes before the current frame, pairwise feature matching is performed using the 10 co-visibility keyframes with the current frame and the loop closure candidate keyframes. Triangulation is then used to generate new second 3D map points. These new second 3D map points, along with the results of the second optimization, are used in loop closure correction. After loop closure correction, a spanning tree is used for Bundle Adjustment optimization. By adopting the above strategy, this embodiment of the invention can greatly supplement the feature points matched during pose estimation, thereby significantly improving the stability and robustness of tracking.
[0042] This invention uses iBoW-LCD for loop closure candidate keyframe detection. The proposed method utilizes an incremental bag-of-words (BoW) scheme based on binary descriptors to retrieve similar images seen previously, avoiding any vocabulary training phase typically required by the classic BoW model. Furthermore, iBoW-LCD ensures that the bag-of-words model is always applicable to the current environment, which can further improve the accuracy of loop closure detection.
[0043] After pose estimation and optimization are completed, the image frame enters the loop closure detection thread. In existing synchronous localization and mapping systems, after prolonged operation, the accumulation of too many keyframes and map points leads to a significant decrease in system real-time performance, and may even cause system crashes, when global Bundle Adjustment optimization is performed after loop closure correction. Therefore, in this embodiment of the invention, after loop closure correction, only the spanning tree is used for Bundle Adjustment optimization.
[0044] The spanning tree is a concept relative to the shared view and the essential graph. The shared view refers to all keyframes created and retained by the synchronous localization and mapping method up to the current point as vertices. If two keyframes satisfy a shared view relationship (i.e., they jointly observe the same 3D map point), they are connected by an edge, and the weight of the edge is the number of shared 3D map points. This forms an undirected weighted graph for graph optimization. For any vertex, the essential graph is based on the shared view, retaining only edges with a shared view relationship greater than a preset threshold, and retaining at least one edge. According to the order in which the keyframes were created, from back to front, the spanning tree is constructed based on the shared view, retaining only the vertices and edges with the highest non-repeating shared view relationship between any given vertex (for example, taking the third vertex, removing already retained edges with the third vertex). After connecting the edges, the remaining edge with the highest weight connected to the third vertex is taken as the highest non-repeating common view relationship. In the spanning tree, the most recently created and retained keyframe with the highest common view is taken as the parent keyframe, and the most recently created and retained keyframe is taken as its child keyframe. If the most recently created and retained keyframe is updated (if the current keyframe is filtered and removed in step S5, then the most recently created and retained keyframe will be updated), then the parent-child relationship needs to be updated (i.e., according to the order in which the retained keyframes were created, from back to front, the spanning tree retains only the vertices and edges with the highest non-repeating common view relationship between any two vertices as the main body, based on the common view, and then the most recently created and retained keyframe with the highest common view is taken as the parent keyframe, and the most recently created and retained keyframe is taken as its child keyframe). This strategy greatly reduces the running time of the closed-loop thread. Since the optimization using the spanning tree Bundle Adjustment may still take a few seconds to execute, newly added keyframes and map points must also be updated to maintain trajectory consistency. Using only spanning tree bundle adjustment, compared to using global bundle adjustment after loop closure, can significantly improve the algorithm's real-time performance, while the resulting loss in accuracy is acceptable. For example... Figure 5 The diagram shown is an exemplary diagram of a spanning tree.
[0045] S7. By continuously repeating S1-S6, the estimated pose of the camera can be estimated in real time. Also based on the ROS operating system, the estimated pose and sparse point cloud map can be published as ROS topics in real time.
[0046] According to a fourth aspect of the present invention, a visual synchronous localization and mapping system for low-light environments is provided, comprising: a first optimization module, configured to extract ORB feature points from an enhanced image, perform feature matching between adjacent frames, and triangulate the matched feature points to generate 3D map points; for the first and second frames of images, an initial pose is estimated using epipolar geometry and the second frame image is initialized as a reference keyframe; an initial map is created based on the estimated initial pose of the first and second frames and the 3D map points of the first and second frames, and the estimated initial pose is optimized for the first time using Bundle Adjustment; based on the initial map, a new image is input each time, feature matching is performed between adjacent frames, and the matched feature points are triangulated to generate 3D map points; the initial pose of the current frame image is estimated using the PnP method, and the map is expanded using the currently obtained 3D map points and the estimated initial pose to generate a local map; and the initial pose of the current frame in the local map is then optimized using Bundle Adjustment. The first optimization is performed by Bundle Adjustment; the first judgment module is used to determine whether a new keyframe should be created for the current frame based on the keyframe creation conditions; if the current frame is a keyframe, keyframe filtering and elimination are performed, the number of shared keyframes is judged, and then Bundle Adjustment is used for the second optimization and the spanning tree is updated; the second judgment module is used to perform loop closure correction after detecting a loop closure candidate keyframe; after loop closure correction, Bundle Adjustment is used for optimization using the spanning tree.
[0047] Experiments were conducted using the technical solution described above, demonstrating that the present invention achieves higher accuracy compared to the most mainstream visual simultaneous localization and mapping method, ORB-SLAM3. As shown in Tables 1 and 2, the method in this embodiment was compared with ORB-SLAM3 on eight video sequences from two public datasets, TUM and EuRoC, and the method in this embodiment exhibits better performance.
[0048] Table 1
[0049]
[0050] Table 2
[0051]
[0052] In Tables 1 and 2, Max ape, Mean ape, and rmse represent the maximum trajectory error, average trajectory error, and root mean square trajectory error, respectively. As shown in Tables 1 and 2, except for the tum_freiburg1_xyz video sequence where the mean absolute trajectory error of ORB-SLAM3 is slightly better than that of this embodiment, and the EuRoC_MH01 and EuRoC_MH03 video sequences where the relative trajectory error of ORB-SLAM3 is slightly better than that of this embodiment, in other video sequences, the maximum trajectory error, average trajectory error, and root mean square error of this embodiment are all better than those of ORB-SLAM3.
[0053] like Figure 2 (a) A comparison of absolute predicted trajectories and true trajectories on three different video sequences in the embodiments of the present invention, using two datasets in TUM and one dataset in EuRoC. The dashed lines represent the true pose trajectories of the datasets, and the solid lines represent the absolute predicted trajectories of the embodiments of the present invention. Figure 2 (b) shows the absolute predicted trajectory and the true trajectory of ORB-SLAM3 on the same three video sequences. It can be seen that the absolute predicted trajectory of the embodiment of the present invention is more similar to the true trajectory.
[0054] Figure 3 (a) shows the error curves of the three directions x, y, z and the three angles yaw, pitch and roll on the Freiburg2-xyz dataset of the present invention. The dashed line represents the true value and the solid line represents the present invention. Figure 3 (b) shows the error curves of ORB-SLAM3 on the same dataset. It can be seen that this embodiment has smaller errors in the x, y, z directions and in the yaw, pitch, and roll angles.
[0055] Figure 4 (a) shows the operation of the embodiments of the present invention in a real low-light environment. Figure 4 (b) shows the performance of ORB-SLAM3 in the same scene. As can be seen, this embodiment visibly enhances the brightness of the image used for visual synchronous localization and mapping, as well as the number of feature points that can be extracted.
[0056] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A method for visual simultaneous localization and mapping in low-light environments, characterized in that, include: ORB feature point extraction is performed on the enhanced image, feature matching is performed between adjacent frames, and the matched feature points are triangulated to generate 3D map points. The initial pose is estimated using epipolar geometry for the first and second frames, and the second frame is initialized as a reference keyframe. An initial map is created based on the estimated initial pose of the first and second frames and the 3D map points of the first and second frames, and the estimated initial pose is optimized for the first time using Bundle Adjustment. Based on the initial map, each time a new image frame is input, feature matching is performed between adjacent frames, and the matched feature points are triangulated to generate 3D map points. The PnP method is used to perform initial pose estimation on the current frame image, and the map is expanded using the currently obtained 3D map points and the estimated initial pose to generate a local map. Then, the initial pose estimated for the current frame in the local map is optimized for the first time using Bundle Adjustment. Based on the keyframe creation conditions, determine whether to create a new keyframe for the current frame; if the current frame is a keyframe, perform keyframe filtering and removal, determine the number of shared keyframes, and then use Bundle Adjustment for a second optimization and update the spanning tree. After detecting a candidate keyframe for loop closure, loop closure correction is performed; after loop closure correction, a spanning tree is used for BundleAdjustment optimization.
2. The visual synchronous localization and mapping method for low-light environments according to claim 1, characterized in that, The acquisition of the enhanced image specifically involves: Based on the ROS robot operating system, real-time video captured under low light conditions is published as a ROS low light image message. Convert low-light images to the HSV color space; The V channel of the low-light image and the desired brightness are used as inputs to a lightweight convolutional neural network. The high-order curve adjustment method in a five-layer multilayer perceptron is used to estimate the global brightness adjustment V channel histogram coefficients. The histogram coefficients are concatenated and fed into a convolutional layer with a local receptive field to obtain the enhanced V channel. The enhanced V channel replaces the original V channel of the low-light image and is converted to RGB color space to obtain the enhanced image.
3. The visual synchronous localization and mapping method for low-light environments according to claim 1, characterized in that, The method of determining whether to create a new keyframe based on the keyframe creation conditions is as follows: the method is to determine whether to create a new keyframe based on whether the 3D map points of the current frame reach a preset value, the degree of overlap with the previous keyframe, and whether there is a preset parallax with the previous keyframe. If all three conditions are met, the current frame is created as a keyframe; otherwise, it is not created.
4. The visual synchronous localization and mapping method for low-light environments according to claim 1, characterized in that, The process of filtering and removing keyframes and determining the number of shared keyframes, followed by a second optimization using Bundle Adjustment and updating the spanning tree, specifically involves: If the current frame is a keyframe, the following steps are taken: If the number of keyframes exceeds three, keyframes are filtered and removed: If at least a preset proportion of 3D map points in the local map are observed by more than three keyframes, only the three keyframes with the highest co-observation relationship and their corresponding 3D map points are retained; If the number of co-observation keyframes reaches n or more, the current keyframe is reprojected and matched pairwise with each of the n-1 co-observation keyframes (excluding the current keyframe) with the highest co-observation relationship to obtain matching feature point pairs; After matching, the matching feature point pairs are triangulated to generate a new first 3D map point; If a new 3D map point exists, the new first 3D map point, the 3D map points of the original keyframes in the local map, and their corresponding frames' first optimized poses are simultaneously optimized a second time using BundleAdjustment; and the spanning tree is updated; otherwise, the 3D map points of the original keyframes in the local map and their corresponding frames' first optimized poses are simultaneously optimized a second time using BundleAdjustment; and the spanning tree is updated.
5. The visual synchronous localization and mapping method for low-light environments according to claim 4, characterized in that, A reverse matching scheme is introduced to eliminate mismatched feature point pairs.
6. The visual synchronous localization and mapping method for low-light environments according to claim 5, characterized in that, The reverse matching specifically involves: when a feature point of a keyframe is reprojected into any co-view keyframe, descriptor matching is performed in the area near the projection to find matching features, and then the feature point is projected back to the original keyframe. Only when the error between the projected point and the original feature point does not exceed a preset pixel is it selected as a matching feature point pair; otherwise, it is discarded.
7. The visual synchronous localization and mapping method for low-light environments according to claim 1, characterized in that, After detecting a candidate keyframe for loop closure, loop closure correction is performed. After loop closure correction, a spanning tree is used for Bundle Adjustment optimization. Specifically, after detecting a candidate keyframe for loop closure in the keyframes before the current frame, a preset number of co-visible keyframes with the highest co-visibility with the current frame and the candidate keyframe for loop closure are used for pairwise feature matching, triangulation to generate new second 3D map points, and the new second 3D map points and the result of the second optimization are used in loop closure correction. After loop closure correction, a spanning tree is used for Bundle Adjustment optimization.
8. The visual synchronous localization and mapping method for low-light environments according to claim 1, characterized in that, The specific process of creating the spanning tree is as follows: following the order in which keyframes were created, from back to front, the spanning tree is constructed by retaining only the vertices and edges with the highest degree of shared visibility between any two points with any vertex as the subject, without any repetition. In the spanning tree, the keyframe with the highest degree of shared visibility among the most recently created and retained keyframes is taken as the parent keyframe, and the most recently created and retained keyframe is taken as its child keyframe. If the most recently created and retained keyframe is updated, the parent-child relationship needs to be updated.
9. A visual synchronous localization and mapping system for low-light environments, characterized in that, include: The first optimization module is used to extract ORB feature points from the enhanced image, perform feature matching between adjacent frames, and triangulate the matched feature points to generate three-dimensional map points. For the first and second frames, the initial pose is estimated using epipolar geometry, and the second frame is initialized as a reference keyframe. An initial map is created based on the estimated initial pose and 3D map points of the first and second frames, and Bundle Adjustment is used to optimize the estimated initial pose for the first time. Based on the initialized map, a new image is input each time, and feature matching is performed between adjacent frames. The matched feature points are triangulated to generate 3D map points. The PnP method is used to estimate the initial pose of the current frame image. The map is expanded using the obtained 3D map points and the estimated initial pose to generate a local map. The initial pose estimated for the current frame in the local map is then optimized for the first time using Bundle Adjustment. The first judgment module is used to determine whether to create a new keyframe based on the keyframe creation conditions; if the current frame is a keyframe, it performs keyframe filtering and removal, judges the number of shared keyframes, and then uses BundleAdjustment to perform a second optimization and update the spanning tree. The second judgment module is used to perform loop closure correction after detecting a loop closure candidate keyframe; after loop closure correction, the spanning tree is used for Bundle Adjustment optimization.
Citation Information
Patent Citations
Rapid lightweight low-illumination image enhancement method and system based on supervised learning
CN116363009A
