ORB-SLAM2 map construction method fused with united histogram equalization

Abstract

The invention discloses an ORB-SLAM2 map construction method based on fusion joint histogram equalization. The ORB-SLAM2 map construction method based on fusion joint histogram equalization comprises the following steps: S10, carrying out image acquisition and fusion joint histogram equalization enhancement preprocessing; s20, ORB feature extraction and matching are carried out; s30, tracking and positioning; and S40, mapping and optimization are carried out. Aiming at the problem of tracking degradation caused by difficulty in feature point extraction of ORB-SLAM2 under a low-light condition in the prior art, the method is combined with an image enhancement module, and can be applied to scenes such as underground mine exploration robots and night patrol unmanned aerial vehicles; the requirements of the robot for real-time positioning and map construction under the complex illumination conditions of low illumination, high dynamic range and the like are met, and the environmental adaptability and operation efficiency of the SLAM system are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of map construction, and particularly relates to an ORB-SLAM2 map construction method fusing joint histogram equalization. BACKGROUND

[0002] Simultaneous Localization and Mapping (SLAM) technology is a core technology for realizing that a self-moving device (such as a robot, a drone, an autonomous vehicle) "perceives an environment, locates itself and constructs a map" in an unknown environment, and the performance of the technology directly determines the self-running capability of the device in a complex scene. With the development of computer vision technology, visual SLAM based on a visual sensor (monocular, stereo, RGB-D camera) has become a mainstream direction in the field of SLAM due to the advantages of low cost and rich environmental information acquisition, and ORB-SLAM2, as a mature open-source framework supporting multiple sensors, is widely used in robot navigation, augmented reality, environment monitoring and other scenes due to the advantages of strong real-time performance, high positioning accuracy, loop detection and repositioning capability.

[0003] The core workflow of ORB-SLAM2 depends on front-end image feature extraction and matching: by extracting ORB features (based on FAST corner detection and BRIEF descriptor optimization) with rotation invariance and scale invariance, the association between different frames of images is established, and then camera pose estimation and three-dimensional map construction are realized. However, the traditional ORB-SLAM2 has significant technical defects in actual application, especially in complex lighting scenes, and the specific problems are as follows: 1. Feature extraction is unstable under complex lighting conditions such as low light or high dynamic range. The extraction quality of ORB features is highly dependent on the contrast and edge definition of the image: in a low-light environment (such as night, underground mine, indoor shadow area), the overall image gray value is low, and the gray difference between the target and the background is blurred, making it difficult for the FAST corner detection algorithm to capture effective feature points, and the number of feature points decreases sharply; in a high dynamic range scene (such as a backlight environment, an indoor-outdoor transition area with alternating light and dark), the image is prone to local overexposure or underexposure, the details in the overexposed area are lost, and the features in the underexposed area are masked by noise, further leading to uneven distribution of ORB feature points and a decrease in the number of effective matching pairs, directly affecting the stability of the tracking module of ORB-SLAM2, and easily causing "tracking loss".

[0004] 2. Traditional image enhancement algorithms are difficult to adapt to the robustness and real-time requirements of SLAM. To improve image quality under complex lighting conditions, existing technologies often use image enhancement algorithms such as histogram equalization (HE) and adaptive histogram equalization (CLAHE) as preprocessing steps for SLAM. However, these algorithms have obvious limitations: traditional HE algorithms only adjust the global grayscale distribution of the image without considering pixel neighborhood information, which easily amplifies image noise (especially in low-light scenes, noise enhancement will further interfere with ORB feature descriptor matching, leading to an increase in matching error rate); although CLAHE achieves local enhancement through block processing, it has "block effect" (the enhanced image shows obvious region boundaries), and requires manual adjustment of block size and contrast limit parameters, resulting in poor parameter adaptability and difficulty in dealing with changing lighting scenes; some deep learning-based image enhancement algorithms (such as generative adversarial networks GAN) can achieve high-quality enhancement, but their computational complexity is high (the time for a single enhancement usually exceeds 100ms), far exceeding the real-time requirements of ORB-SLAM2 (the processing time for each frame must be less than 50ms to meet a frame rate of more than 20fps), and cannot be applied to real-time SLAM systems.

[0005] 3. The application gap of the Joint Histogram Equalization (JHE) algorithm. Existing technologies have proposed Joint Histogram Equalization (JHE) image enhancement algorithms. Their core advantage lies in constructing a two-dimensional joint histogram based on the "original image - neighborhood average image," and achieving gray-level mapping through the cumulative distribution function (CDF). This not only utilizes pixel neighborhood information to suppress noise but also adaptively widens the image's dynamic range, showing particularly significant enhancement effects on images with narrow dynamic ranges and low contrast. However, currently, the JHE algorithm is only applied to the quality optimization of static images (such as visual enhancement of medical images). No technical solution has yet been found to combine it with SLAM systems to address the robustness issue of SLAM under complex lighting conditions. How to adapt the lightweight nature and enhancement effects of JHE to the real-time performance and positioning accuracy requirements of ORB-SLAM2 has become a pressing technical problem in this field.

[0006] In summary, the performance deficiencies of the existing ORB-SLAM2 system in complex lighting scenarios, as well as the insufficient compatibility between traditional image enhancement algorithms and SLAM systems, have given rise to a technical solution that needs to balance "image enhancement effect, real-time performance, and SLAM positioning accuracy." Deeply integrating the JHE algorithm with ORB-SLAM2 is a key innovative direction for solving the above-mentioned technical pain points. Summary of the Invention

[0007] In view of this, the present invention proposes an ORB-SLAM2 map construction method that integrates joint histogram equalization, comprising the following steps: S10, Image acquisition and fusion combined with histogram equalization enhancement preprocessing; S20, ORB feature extraction and matching; S30, tracking and localization; S40, mapping and optimization.

[0008] Preferably, the S10 specifically comprises the following steps: S11, the sensor collects raw image data, and the size and color channel information of the image are obtained; S12, performing fusion joint histogram equalization enhancement processing on the raw image.

[0009] Preferably, the S12 specifically comprises the following steps: S121, if it is a color image, it is converted from the BGR color space to the YUV color space, and the luminance component and the chrominance component are separated; if it is a grayscale image, the original image is directly used as the luminance component; S122, calculating a 3x3 neighborhood average luminance matrix for the luminance component, and constructing a raw luminance-average luminance two-dimensional joint histogram; S123, calculating the cumulative distribution function of the joint histogram, and mapping the cumulative distribution function to the 0-255 gray scale range through linear stretching to generate the enhanced luminance component; S124, recombining the enhanced luminance component and the original chrominance component, converting back to the BGR color space, and obtaining the enhanced image.

[0010] Preferably, the S2 performs ORB feature extraction and matching on the image obtained after the enhancement of the S10, and specifically comprises the following steps: S21, extracting ORB features from the output enhanced image, the ORB features being constructed based on FAST corner detection and BRIEF descriptor, including key point position, direction and descriptor; S22, using a combination of the bag-of-words model and brute-force matching to realize feature matching of the current frame and the reference key frame map points, and generating a matching pair set.

[0011] Preferably, the S30 is based on the ORB feature points extracted and matched by the S20, and specifically comprises the following steps: S31, estimating the initial pose of the current frame based on the matching pairs of S22 through the PNP algorithm; S32, optimizing the pose by bundle adjustment to minimize the re-projection error and obtain the accurate real-time pose of the camera; S33, if the tracking fails, the failure conditions include that the number of matching pairs is lower than the threshold, triggering the relocalization mechanism, searching the historical key frame based on the bag-of-words model, and re-matching and optimizing the pose.

[0012] Preferably, the S40 is used to build a map and optimize the pose and map points calculated according to the S30, and the specific steps include the following steps. S41, for the newly inserted key frame, three-dimensional map points are generated by triangulation, redundant map points are removed, and local bundle adjustment is performed to optimize the local map and key frame pose. S42, loop detection, loop candidate frames are detected based on the bag-of-words model, and after the loop is confirmed through geometric verification, pose graph optimization and global bundle adjustment are performed to eliminate cumulative drift and ensure global consistency of the map.

[0013] Compared with the prior art, the JHE enhancement module is embedded in the front end of the ORB-SLAM2, the brightness component of the input image is subjected to joint histogram equalization processing, the chroma information is retained, the enhanced image is input into the core module of the ORB-SLAM2 to complete feature extraction, positioning and mapping, the stability of the features under complex illumination is improved, and the real-time performance of the system is ensured. The accuracy, stability and robustness under low light and high dynamic range conditions are improved. The present application can be applied to underground mine detection robots, night patrol unmanned aerial vehicles and other scenes, and meets the needs of robots for real-time positioning and map construction under complex illumination conditions such as low light and high dynamic range, and improves the environmental adaptability and operation efficiency of the SLAM system. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to make the purpose, technical scheme and beneficial effects of the present application clearer, the present application provides the following drawings for illustration: Figure 1 A framework flowchart of the ORB-SLAM2 map construction method fused with joint histogram equalization is provided for the present application. Figure 2 A specific flowchart of the ORB-SLAM2 map construction method fused with joint histogram equalization is provided for the present application. Figure 3 A difference value diagram of the trajectory and the true value in the xyz three directions obtained by running the freiburg2 pioneer slam sequence in the TUM RGB-D dataset is provided for the present application. Figure 4 A difference value diagram of the trajectory and the true value in the xyz three directions obtained by running the freiburg2 pioneer slam sequence in the TUM RGB-D dataset is provided for the present application.

[0015] Figure 5 A box line comparison diagram of the trajectories obtained by running the freiburg2 pioneer slam sequence in the TUM RGB-D dataset is provided for the present application. DETAILED DESCRIPTION

[0016] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0017] ORB-SLAM2 is a widely used open source framework in the current visual SLAM field, and becomes the mainstream technical solution in robot navigation, augmented reality and other scenes, with the compatibility of monocular, stereo and RGB-D multi-sensor, and the core capabilities of real-time positioning, loop detection and repositioning. However, in the complex lighting scene of low light and high dynamic range, the image contrast is insufficient, which leads to the sharp reduction of the number of extracted ORB feature points and the increase of matching error rate, thereby causing the significant deficiencies of tracking loss and positioning accuracy decline. Moreover, the traditional image enhancement algorithm is difficult to balance between “enhancement effect” and “SLAM real-time performance”. In order to solve the above-mentioned deficiencies, the present application embeds a lightweight joint histogram equalization (JHE) enhancement module in the image input link of ORB-SLAM2. First, the original image is converted to YUV color space to separate the luminance component (Y channel) and the chrominance component (U, V channel), and only the luminance component is subjected to JHE processing. The average luminance matrix is generated by 3x3 box filtering, the “original luminance-average luminance” two-dimensional joint histogram is constructed, the cumulative distribution function (CDF) is calculated and linearly stretched to the 0-255 gray scale range, which not only suppresses noise and improves contrast by using neighborhood information, but also avoids color distortion. Then, the enhanced luminance component and the original chrominance component are recombined into a BGR image, which is input into the core module of ORB-SLAM2 for ORB feature extraction, matching, pose estimation and map construction. This scheme improves the number of extracted ORB feature points under complex lighting without changing the core logic of ORB-SLAM2, and the single-frame processing time of the JHE module is <10ms, ensuring that the overall frame rate of the system is >20fps, which balances robustness and real-time performance.

[0018] Referring to Figure 1 , Figure 2 , the fusion joint histogram equalization ORB-SLAM2 map construction method of the present application, in general, includes 4 steps: image acquisition and JHE preprocessing; ORB feature extraction and matching; tracking and positioning; mapping and optimization. Specifically, the steps include: S10, image acquisition and JHE enhancement preprocessing; S20, ORB feature extraction and matching; S30, tracking and positioning; S40, mapping and optimization; wherein, S10 further includes: S11, the sensor acquires original image data (supports monocular, stereo and RGB-D cameras), and obtains the size (width x height) and color channel information of the image; S12, performing JHE enhancement processing on the original image; further comprising: S121, if it is a color image, converting it from BGR color space to YUV color space, separating the luminance component (Y channel) and the chrominance component (U, V channel); if it is a grayscale image, directly taking the original image as the luminance component; S122, calculating a 3x3 neighborhood average luminance matrix for the luminance component, and constructing a "original luminance-average luminance" two-dimensional joint histogram; S123, calculating the cumulative distribution function (CDF) of the joint histogram, mapping the CDF to the 0-255 grayscale range through linear stretching, and generating an enhanced luminance component; S124, recombining the enhanced luminance component (Y) and the original chrominance component (U, V) to convert back to BGR color space, and obtaining an enhanced image; S20 performing ORB feature extraction and matching on the image obtained after enhancement in S10, which further comprises: S21, extracting ORB features (including key point position, direction, and descriptor) from the enhanced image output by S10, which are constructed based on FAST corner detection and BRIEF descriptor and have rotation and scale invariance; S22, using a combination of the bag-of-words model (DBoW2) and brute-force matching to realize feature matching between the current frame and reference keyframes / map points, and generating a matching pair set; S30, based on the ORB feature points extracted and matched in S20, performing tracking and positioning, which further comprises: S31, based on the matching pairs in step S22, estimating the initial pose of the current frame through the PNP (Perspective-n-Point) algorithm; S32, using Motion-only BA (bundle adjustment) to optimize the pose and minimize the reprojection error to obtain the accurate real-time pose of the camera; S33, if tracking fails (e.g., the number of matching pairs is below a threshold), triggering a relocalization mechanism: searching for historical keyframes based on the bag-of-words model, re-matching and optimizing the pose; S40, based on the pose and map points calculated in S30, performing mapping and optimization, which further comprises: S41, for newly inserted keyframes, generating three-dimensional map points through triangulation, removing redundant map points (e.g., points with excessive reprojection error), and performing local BA optimization on the local map and keyframe pose; S42, loop detection: detecting loop candidate frames based on the bag-of-words model, confirming the loop after geometric verification, performing pose graph optimization and global BA, eliminating cumulative drift, and ensuring global consistency of the map.

[0019] The input data used in the present application adopts image and depth data in the TUM RGB-D dataset and image data in the EuRoC dataset. The TUM RGB-D dataset is a visual dataset with depth information taken by a handheld RGB-D camera or a robot-mounted RGB-D camera. The EuRoC dataset is a visual dataset taken by a micro unmanned aerial vehicle. The simulation experiment platform in this paper is an Intel(R) Core(TM) i7-14700HX (2.10 GHz) computer with 64G memory. The present application is run on 16 sequences in the TUM RGB-D dataset and 7 sequences in the EuRoC dataset, and the running results are compared with those of the classic ORB-SLAM2 system to observe the improvement effect.

[0020] In order to verify the technical achievements of the present application, the effectiveness of the present application will be compared with that of the algorithm: In the comparison between the present method and the classic ORB-SLAM2 system, the final comparison result comes from the data obtained by running five times for each sequence. In the difficult-to-track sequence, tracking is difficult to extract feature points due to low light conditions and high dynamic range, so that tracking and positioning are inaccurate or even lost; while in the easy-to-track sequence, initialization is easy and tracking can be maintained all the time, so that the tracking and positioning accuracy and robustness are good. First, the root mean square error (RMSE) and standard deviation (S.D.) of the absolute trajectory obtained by running on the TUM-RGBD dataset are taken as indexes to compare the improvement of accuracy and stability before and after the fusion of the joint histogram equalization. Then the same indexes are used to quantitatively measure the difference between the running results of the present application and the classic ORB-SLAM2 system on the EuRoC dataset.

[0021] As shown in Tables 1, 2, Figure 3 , Figure 4 , Figure 5 the experimental data results of the present application and the classic ORB-SLAM2 system are compared. Table 1 is a comparison of the root mean square error and standard deviation of the absolute trajectory of the present application and the classic ORB-SLAM2 system on the TUM RGB-D dataset; Table 2 is a comparison of the root mean square error and standard deviation of the absolute trajectory of the present application and the classic ORB-SLAM2 system on the EuRoC dataset; Figure 3 is a comparison chart of the running trajectory of the present application on the freiburg2 pioneer slam sequence in the TUM RGB-D dataset and the true value; Figure 4 is a comparison chart of the running trajectory of the classic ORB-SLAM2 on the freiburg2 pioneer slam sequence in the TUM RGB-D dataset and the true value; Figure 5The box plot of the running results of the present application and the classic ORB-SLAM2 on the freiburg2 pioneer slam sequence of the TUM RGB-D dataset.

[0022] Table 1 Comparison of the root mean square error (RMSE) and the standard deviation (S.D.) of the absolute trajectory of the present application and ORB-SLAM2 on the TUM RGB-D dataset

[0023] Table 2 Comparison of the root mean square error (RMSE) and the standard deviation (S.D.) of the absolute trajectory of the present application and ORB-SLAM2 on the EuRoC dataset

[0024] The above Table 1 and Table 2 list the data comparison of the root mean square error and the standard deviation of the absolute trajectory of the present application system and the classic ORB-SLAM2 system, which can intuitively show the different effects of the present application on different sequences.

[0025] The above Table 1 and Table 2 list the data comparison of the root mean square error and the standard deviation of the present application and the classic ORB-SLAM2 algorithm, which can intuitively show the different effects of the present application on different sequences. In the difficult-to-track sequence, that is, the sequence with low light conditions and high dynamic range, the tracking accuracy is the highest, and the stability and robustness are the best. On the easy-to-track sequence, the improvement effect of tracking accuracy, stability and robustness is limited. Through the comparison of the root mean square error and the standard deviation, the improvement effect of the present application in accuracy and stability can be reflected. Figure 3 、 Figure 4 The present application and the classic ORB-SLAM2 system respectively generate the map trajectory in the xyz direction on the freiburg2 pioneer slam sequence of the TUM RGB-D dataset, and the blue line is the true value and the green line is the running trajectory. Figure 5 The box plot comparison of the present application and the classic ORB-SLAM2 system on the freiburg2 pioneer slam sequence is shown. In summary, the present application realizes the improvement effect of accuracy, stability and robustness in the difficult-to-track sequence with low light conditions and high dynamic range by fusing the joint histogram equalization. It is verified by experiments that the root mean square of the absolute trajectory error of the classic ORB-SLAM2 system is reduced by 31.66% on average, and the standard deviation is reduced by 41.03% on average.

[0026] The above description of the embodiments is only used to help understand the method of the present application and its core idea. It should be noted that for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0027] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of fusing joint histogram equalization with ORB-SLAM2 map construction, characterized in that, The method comprises the following steps: S10, image acquisition and fusion combined histogram equalization enhancement preprocessing; S20, ORB feature extraction and matching; S30, tracking and positioning; S40, mapping and optimization.

2. The method of claim 1, wherein the fusion combined with joint histogram equalization ORB-SLAM2 map construction method is characterized by, The S10 specifically comprises the following steps: S11, sensor acquires original image data, acquires the size and color channel information of the image; S12, performs fusion combined histogram equalization enhancement processing on the original image.

3. The method of claim 2, wherein the fusion of joint histogram equalization ORB-SLAM2 map construction method is characterized by, The S12 specifically comprises the following steps: S121, if it is a color image, it is converted from the BGR color space to the YUV color space, and the luminance component and the chrominance component are separated; if it is a grayscale image, the original image is directly taken as the luminance component; S122, a 3*3 neighborhood average luminance matrix is calculated for the luminance component, and a two-dimensional joint histogram of the original luminance-average luminance is constructed; S123, the cumulative distribution function of the joint histogram is calculated, the cumulative distribution function is mapped to the 0-255 gray scale range through linear stretching, and an enhanced luminance component is generated; S124, the enhanced luminance component and the original chrominance component are recombined, converted back to the BGR color space, and an enhanced image is obtained.

4. The method of claim 3, wherein the fusion combined with joint histogram equalization ORB-SLAM2 map construction method is characterized by, The S2 performs ORB feature extraction and matching on the image obtained after the S10 enhancement, and specifically comprises the following steps: S21, the output enhanced image is extracted for ORB features, the ORB features are constructed based on FAST corner detection and BRIEF descriptor, and include key point position, direction and descriptor; S22, a bag-of-words model and a brute-force matching combination are used to realize feature matching of map points of the current frame and reference key frames, and a matching pair set is generated.

5. The method of claim 4, wherein the fusion combined with joint histogram equalization ORB-SLAM2 map construction method is characterized by, The S30 is based on the ORB feature points extracted and matched in the S20, and specifically comprises the following steps: S31, based on the matching pairs in the S22, the initial pose of the current frame is estimated through a PNP algorithm; S32, the pose is optimized by adopting bundle adjustment to minimize the re-projection error, and an accurate camera real-time pose is obtained; S33, if tracking fails, the failure includes that the number of matching pairs is lower than a threshold, a repositioning mechanism is triggered, historical key frames are searched based on a bag-of-words model, the pose is re-matched and optimized.

6. The method of claim 5, wherein the fusion combined with joint histogram equalization ORB-SLAM2 map construction method is characterized by, The S40 performs mapping and optimization according to the pose and map points calculated in the S30, and specifically comprises the following steps: S41, for a newly inserted key frame, three-dimensional map points are generated through triangulation, redundant map points are removed, and local bundle adjustment is performed to optimize the local map and the key frame pose; S42, loop detection, based on a bag-of-words model, a loop candidate frame is detected, after the loop is confirmed through geometric verification, pose graph optimization and global bundle adjustment are performed, the cumulative drift is eliminated, and the global consistency of the map is ensured.

Images