ORB feature point extraction method introduced with image enhancement and adaptive threshold under weak illumination condition

Abstract

The invention discloses an ORB feature point extraction method under a weak illumination condition by introducing image enhancement and a self-adaptive threshold value, which comprises the following steps: converting an input image from an RGB color space to an HSV color space, separating color information from brightness information, only processing a brightness component V in an enhancement process, keeping a hue component H and a saturation component S unchanged, and extracting the color information from the color space; processing the brightness component V by adopting an MSR algorithm based on a Retinex theory; further processing the brightness component V after MSR processing by using self-adaptive histogram equalization with limited contrast, and limiting the contrast enhancement degree of a local area while maintaining the overall brightness distribution of the image; fusing the unprocessed hue component H, the saturation component S and the processed brightness component V, and converting back to the RGB color space; meanwhile, a self-adaptive threshold fast corner extraction strategy is provided, the threshold can be dynamically adjusted according to the gray value of the image when illumination changes, and therefore the accuracy and robustness of feature point extraction are improved.

Description

Technical Field

[0001] The invention relates to an ORB feature point extraction method under weak illumination conditions by introducing image enhancement and adaptive threshold, and belongs to the field of visual SLAM positioning. Background Art

[0002] Simultaneous localization and mapping (SLAM) is a key technology in the field of robotics, which aims to enable robots to autonomously build maps in unknown environments and locate their own positions in real time. With the rapid development of computer vision and sensor technology, visual SLAM has gradually become an important branch of SLAM technology due to its advantages such as low cost and rich information. Visual SLAM uses visual sensors (such as cameras) to obtain environmental images, perform pose estimation and map construction, and provides strong support for the autonomous navigation and intelligent interaction of robots. However, when visual SLAM technology is in low light conditions, its performance often faces severe challenges, which has become a hot and difficult issue in current research. Due to insufficient light, key information such as texture, edge and color in the image becomes blurred or even lost, which has a great impact on key links such as feature extraction, matching, tracking and map construction in visual SLAM. Traditional visual SLAM algorithms usually rely on obvious feature points in the image for positioning and mapping, but under low light conditions, these feature points are often difficult to detect accurately, resulting in a significant decrease in the stability and accuracy of the algorithm. In addition, low light may also cause an increase in image noise, further interfering with the normal operation of the algorithm, causing deviations or distortions in the results of positioning and mapping. Since low-light conditions pose a severe challenge to visual SLAM technology, it is extremely important to study visual SLAM technology under this condition. First of all, from the perspective of practical applications, many scenarios may be carried out under low-light conditions, such as night patrols, underground space exploration, etc. In these scenarios, achieving autonomous navigation and map construction of robots is crucial to improving task execution efficiency and safety. Therefore, studying visual SLAM technology under low-light conditions will help expand the application scope of visual SLAM and improve its adaptability and reliability in practical scenarios. Summary of the invention

[0003] The present invention provides an ORB feature point extraction method under weak light conditions by introducing image enhancement and adaptive threshold, so as to realize ORB feature point extraction under weak light conditions by introducing image enhancement and adaptive threshold, and further realize visual SLAM optimization under insufficient light scenes according to the extracted ORB feature points.

[0004] The technical solution of the present invention is:

[0005] According to a first aspect of the present invention, there is provided an ORB feature point extraction method under weak light conditions by introducing image enhancement and adaptive threshold, comprising the following steps:

[0006] Step 1: Obtain the original low-light image;

[0007] Step 2: Convert the original low-light image from the red, green, and blue (RGB) color space to the hue, saturation, and brightness (HSV) space;

[0008] Step 3: Extract the brightness component V from the HSV image, and process the brightness component V using a multi-scale algorithm based on Retinex theory;

[0009] Step 4: Use a contrast-limited adaptive histogram equalization algorithm to optimize the brightness component V processed by the multi-scale algorithm based on Retinex theory;

[0010] Step 5: Convert the brightness component V optimized by the contrast-limited adaptive histogram equalization algorithm and the unprocessed H component and S component back to the RGB color space to obtain an optimized RGB image;

[0011] Step 6: For the optimized RGB image, ORB feature points are extracted according to the ORB feature extraction algorithm; wherein, the threshold used to extract FAST corner points in the ORB feature extraction algorithm adopts an adaptive threshold.

[0012] Furthermore, the determination of the adaptive threshold is specifically as follows: for each pixel point p in the optimized RGB image, the pixel points with the maximum and minimum grayscale values ​​on the circle with the pixel point p as the center point and the radius R are taken as the first pixel points, and the remaining pixel points are taken as the second pixel points; based on the grayscale value of the second pixel point, the local threshold of the pixel point p is obtained; for the optimized RGB image, a preset number of pixel points with the maximum grayscale value and the minimum grayscale value are found from the optimized RGB image, and the global threshold is obtained based on the preset number of pixel points with the maximum grayscale value and the minimum grayscale value; the local threshold and the global threshold are proportionally summed to obtain the adaptive threshold of the pixel point p.

[0013] Furthermore, the calculation formula of the local threshold is as follows:

[0014]

[0015] Where: T1 is the local threshold; α is the adaptive parameter; I(x i ) represents the second pixel x around pixel p i The gray value of represents the average grayscale value of all second pixels, and n is the number of pixels in a circle with a radius of R and a pixel p as the center point.

[0016] Furthermore, the expression of the global threshold is:

[0017]

[0018] Where: T2 is the global threshold; β is the adaptive parameter; I maxj ,I minj They respectively represent the grayscale values ​​of the j-th largest and j-th smallest pixel grayscale values ​​in the optimized RGB image, and J is a preset number.

[0019] Furthermore, the adaptive threshold is expressed as:

[0020] T sum =λ·T1+(1-λ)T2:

[0021] Among them, λ is an adaptive parameter, and 0<λ<1; T1 is the local threshold; T2 is the global threshold.

[0022] According to a second aspect of the present invention, there is provided an ORB feature point extraction device under weak light conditions introducing image enhancement and adaptive thresholding, comprising a module of any one of the above-mentioned methods for extracting ORB feature points under weak light conditions introducing image enhancement and adaptive thresholding.

[0023] The beneficial effects of the present invention are as follows: the present invention introduces an image enhancement module and an adaptive threshold part based on the ORB-SLAM3 framework, thereby improving the positioning accuracy and robustness of the weak light environment. Specifically, an image enhancement module is introduced in the improved framework to improve the quality of the input image under weak light conditions: first, the input image is converted from the RGB color space to the HSV color space to separate the color information from the brightness information. During the enhancement process, only the brightness component V is processed, while the hue component H and the saturation component S remain unchanged. For the brightness component V, the MSR algorithm based on the Retinex theory is used for processing. In order to further optimize the brightness component V, the present invention applies the CLAHE (contrast-limited adaptive histogram equalization) algorithm to process the brightness component V after MSR processing. CLAHE avoids noise amplification and detail loss caused by excessive enhancement by limiting the contrast enhancement degree of the local area while maintaining the overall brightness distribution of the image; finally, the unprocessed hue component H, the saturation component S and the processed brightness component V are fused and converted back to the RGB color space. At the same time, an adaptive threshold fast corner point extraction strategy is proposed, which can dynamically adjust the threshold according to the grayscale value of the image when the illumination changes, thereby improving the accuracy and robustness of feature point extraction, and using non-maximum suppression for further screening and feature point matching, thereby improving the positioning accuracy and robustness of traditional visual SLAM in weak light environments, and overcoming the interference of insufficient light environments on traditional SLAM. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a process framework diagram of the present invention;

[0025] Figure 2 It is a principle diagram of the multi-scale algorithm based on Retinex theory of the present invention;

[0026] Figure 3 is a schematic diagram of a circle drawn with pixel point p as the center point and radius R;

[0027] Figure 4 This is a comparison diagram of the effects of the present invention after enhancement processing in a weak light scene;

[0028] Figure 5 The figure is a comparison diagram of the estimated trajectory and the true trajectory of the MH04 sequence of the EUROC data set run by ORB-SLAM3 and the method of the present invention; wherein, Figure 5 (a) is a comparison chart of ORB-SLAM3; Figure 5 (b) is a comparison diagram of the method of the present invention;

[0029] Figure 6The figure is a comparison of the estimated trajectory and the true trajectory of the MH05 sequence of the EUROC data set run by ORB-SLAM3 and the method of the present invention; wherein, Figure 6 (a) is a comparison chart of ORB-SLAM3; Figure 6 (b) is a comparison diagram of the method of the present invention;

[0030] Figure 7 This is a comparison chart of the absolute trajectory errors of the MH04 sequence of the EUROC dataset run by the method of the present invention and ORB-SLAM3;

[0031] Figure 8 This is a comparison chart of the absolute trajectory errors of the MH05 sequence of the EUROC dataset run by the method of the present invention and ORB-SLAM3. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily without conflict.

[0033] Example 1: Figure 1-Figure 8 As shown, according to a first aspect of an embodiment of the present invention, a method for extracting ORB feature points under weak light conditions by introducing image enhancement and adaptive threshold is provided, comprising the following steps:

[0034] Step 1: Obtain the original low-light image;

[0035] Step 2: Convert the original low-light image from the red, green, and blue (RGB) color space to the hue, saturation, and brightness (HSV) space;

[0036] Step 3: Extract the brightness component V from the HSV image and process it using a multi-scale algorithm based on Retinex theory to improve the brightness and contrast of the image;

[0037] Step 4: Use the contrast-limited adaptive histogram equalization algorithm CLAHE to optimize the brightness component V processed by the multi-scale algorithm based on Retinex theory;

[0038] Step 5: Convert the brightness component V optimized by the contrast-limited adaptive histogram equalization algorithm and the unprocessed H component and S component back to the RGB color space to obtain an optimized RGB image;

[0039] Step 6: For the optimized RGB image, ORB feature points are extracted according to the ORB feature extraction algorithm; wherein, the threshold used to extract FAST corner points in the ORB feature extraction algorithm adopts an adaptive threshold.

[0040] Furthermore, visual SLAM under weak lighting conditions is realized based on the extracted ORB feature points, including:

[0041] Step 7: The ORB feature points extracted by the ORB feature point extraction method under weak light conditions with the introduction of image enhancement and adaptive threshold are further screened and feature point matching is performed using non-maximum suppression.

[0042] Step 8: Perform subsequent camera pose estimation and map construction based on the successfully matched feature points.

[0043] Furthermore, the multi-scale algorithm based on Retinex theory is specifically as follows: first extract the illumination component of the image retaining the brightness component V, then subtract the illumination component from the image retaining the brightness component V, and obtain the reflection component reflecting the intrinsic properties of the image. The image is processed using a multi-scale method, that is, the image is filtered using Gaussian at different scales, and then the filtering results at different scales are averaged and weighted to obtain the estimated illumination image, which can be expressed as the following formula:

[0044]

[0045] Where: I(x,y) represents the image with the brightness component V retained, R(x,y) is the reflection component, represents the convolution operation, F k (x, y) is the Gaussian surround function at the kth scale, W k The weight value of each scale is generally 1 / N; N is the number of scales, and the value of N in the present invention is 3, and: W1=W2=W3=1 / 3.

[0046] Furthermore, the contrast-limited adaptive histogram equalization algorithm, specifically, first divides the image into a number of non-overlapping small blocks, then calculates the histogram of each small block and performs equalization processing to enhance the contrast in the area. In order to avoid excessive amplification of noise and unnatural contrast effects, a contrast limiting parameter is introduced, that is, the peak in the histogram is cropped and the cropped pixel values ​​are redistributed to other parts of the histogram, so as to limit the contrast enhancement amplitude in each small block; finally, the pixel values ​​at the boundary are smoothed by using a bilinear interpolation method to obtain an image with global contrast enhancement and rich details.

[0047] Furthermore, the determination of the adaptive threshold is specifically as follows: for each pixel point p in the optimized RGB image, the pixel points with the maximum and minimum grayscale values ​​on the circle with the pixel point p as the center point and the radius R are taken as the first pixel points, and the remaining pixel points are taken as the second pixel points; based on the grayscale value of the second pixel point, the local threshold of the pixel point p is obtained; for the optimized RGB image, a preset number of pixel points with the maximum grayscale value and the minimum grayscale value are found from the optimized RGB image, and the global threshold is obtained based on the preset number of pixel points with the maximum grayscale value and the minimum grayscale value; the local threshold and the global threshold are proportionally summed to obtain the adaptive threshold of the pixel point p.

[0048] Furthermore, the calculation formula of the local threshold is as follows:

[0049]

[0050] Wherein: T1 is the local threshold; α is an adaptive parameter. For example, the adaptive parameter α can be 1, 2, or 3. In the specific simulation process, the present invention takes 1; I(x i ) represents the second pixel x around pixel p i The gray value of represents the average grayscale value of all second pixel points, n is the number of pixels in a circle with pixel point p as the center point and radius R; R is 3 pixels in the embodiment of the present invention, so the number of pixels on the circle with pixel point p as the center point and radius 3 is 16, the number of first pixels is 2, and the number of second pixels is 14.

[0051] Furthermore, the expression of the global threshold is:

[0052]

[0053] Wherein: T2 is the global threshold; β is an adaptive parameter. For example, the adaptive parameter β can be 1, 2, or 3. In the specific simulation process, the present invention takes 1;

[0054] I maxj ,I minj Respectively represent the grayscale values ​​of the jth largest and jth smallest pixel grayscale values ​​in the optimized RGB image, where J is a preset number and is set to 10 in the embodiment of the present invention;

[0055] Furthermore, the adaptive threshold is expressed as:

[0056] T sum =λ·T1+(1-λ)T2:

[0057] Wherein, λ is an adaptive parameter, and 0<λ<1, which can be determined according to specific lighting conditions, and is taken as 1 / 2 in the embodiment of the present invention.

[0058] Furthermore, the non-maximum suppression retains only the feature point with the largest response value in the local neighborhood of each feature point, and suppresses other feature points with smaller response values, and calculates the response size (score function) V for each detected feature point, where V is defined as the sum of the absolute deviations of the pixel point p and the 16 pixels around it; considers two adjacent feature points and compares their V values; points with lower response values ​​V will be deleted; only the feature point with the local maximum response value is retained as the final feature point, thereby reducing redundancy and improving the quality of feature points.

[0059] According to a second aspect of the present invention, there is provided an ORB feature point extraction device under weak light conditions by introducing image enhancement and adaptive threshold, comprising a module of the ORB feature point extraction method under weak light conditions by introducing image enhancement and adaptive threshold as described in any one of the above items; specifically comprising: a first module for executing step 1: acquiring an original weak light image; a second module for executing step 2: converting the original weak light image from a red, green, and blue RGB color space to a hue, saturation, and brightness HSV space; a third module for executing step 3: extracting a brightness component V from the HSV image, and performing a multi-scale algorithm based on the Retinex theory on the brightness component V Processing; the fourth module is used to execute step 4: using a contrast-limited adaptive histogram equalization algorithm to optimize the brightness component V processed by the multi-scale algorithm based on the Retinex theory; the fifth module is used to execute step 5: converting the brightness component V optimized by the contrast-limited adaptive histogram equalization algorithm and the unprocessed H component and S component back to the RGB color space to obtain an optimized RGB image; the sixth module is used to execute step 6: for the optimized RGB image, ORB feature point extraction is performed according to the ORB feature extraction algorithm; wherein, the threshold used for extracting FAST corner points in the ORB feature extraction algorithm adopts an adaptive threshold. The term "module" as used above can realize a combination of software and / or hardware for a predetermined function. Although the system described in the above embodiment is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived. For the parts not described in detail in each module, please refer to the relevant description of this embodiment.

[0060] According to a third aspect of an embodiment of the present invention, a processor is provided, wherein the processor is used to perform an operation, wherein the operation includes executing any one of the above-mentioned ORB feature point extraction methods under weak light conditions that introduce image enhancement and adaptive thresholding.

[0061] The following uses the public EUROC dataset to evaluate the solution of this embodiment. The EUROC dataset is an important robot vision dataset provided by the European Robotics Center, which is mainly used for the evaluation and research of inertial navigation and visual positioning algorithms. The EUROC dataset contains images, IMU data, camera calibration, ground truth data and other information collected using a micro aerial vehicle (MAV) in an indoor environment. The dataset provides a wealth of indoor scenes and motion patterns, and the EUROC dataset is used to illustrate the positioning of visual SLAM performed based on the ORB feature points extracted by the present invention.

[0062] The experiments were all run on a computer with a CPU model of R7-7735H, a memory capacity of 16GB, a GPU model of RTX4060, and a system environment of Ubuntu18.04.

[0063] Figure 4 The upper part shows a randomly selected low-light scene image from the EUROC dataset. The effect after processing by steps 1 to 5 of the present invention is as follows: Figure 4 As shown in the lower part, through Figure 4 It can be clearly seen from the lower part that after being processed by the present invention, the overall brightness and local contrast of the image are effectively improved, which helps to restore the details of the image under weak lighting conditions.

[0064] Table 1 Experiments on 4 sequences of the EUROC data set yielded the root mean square error RMSE, median Median, standard deviation SD and their improvement percentages corresponding to the ATE index of the method of the present invention and ORB-SLAM3 (where V1_01_easy_ is a normal lighting environment, and the other three sequences are weak lighting environments). Compared with ORB-SLAM3, the present invention achieves a significant improvement in weak lighting sequences. For example, there are a large number of scenes with insufficient lighting in the MH_04_difficult and MH_05_difficult sequences, and the improvement rates of the root mean square error RMSE are 47.45% and 44.08%, respectively. At the same time, the performance of the method of the present invention under normal lighting conditions is not significantly improved compared to ORB-SLAM3; however, the method of the present invention can effectively improve the performance of ORB-SLAM3 in weak lighting scenes.

[0065] Table 1. Absolute trajectory error (ATE) vs. unit (m)

[0066]

[0067] Figure 5 , Figure 6The figures are respectively the pose estimation trajectory comparison diagrams of the MH04 and MH05 sequences of the EUROC data set of the method of the present invention and the ORB-SLAM3 algorithm, where the dotted lines (data-MH-04, data-MH-05) are the real trajectories and the solid lines are the trajectory estimated by the algorithm; Figure 5 (a) Figure 6 (a) Comparison of MH04 and MH05 sequences running on ORB-SLAM3 with the real trajectory. Figure 5 (b) Figure 6 (b) Comparison of the MH04 and MH05 sequences run by the method of the present invention with the real trajectory. Figure 5 , Figure 6 It can be seen that when the camera enters a darker scene, the effect of ORB-SLAM3 is not good, because insufficient light leads to failure to extract sufficient and evenly distributed feature points, resulting in fewer key frames in this area and a large error in the local magnification position. The running trajectory of the method of the present invention is closer to the real trajectory and can maintain a small error even in scenes with insufficient light.

[0068] Figure 7 , Figure 8 The absolute trajectory error comparison diagrams of the MH04 and MH05 sequences of the EUROC dataset run by the method of the present invention and ORB-SLAM3 are shown in Figure 1 (in the figure, benfaming.txt represents the present invention, and ORBSLAM3.txt represents ORB-SLAM3). Figure 7 , Figure 8 The absolute trajectory errors of ORB-SLAM3 and the method of the present invention at a specific moment are more intuitively shown. Figure 7 and Figure 8 It can be seen that the absolute trajectory error of the method of the present invention is smaller than that of ORB-SLAM3 in low-light scenes, and the method of the present invention improves the accuracy and robustness of visual SLAM in low-light environments.

[0069] By applying the above technical solution, it can be known that the present invention separates color information from brightness information by converting the input image from RGB color space to HSV color space. In the enhancement process, only the brightness component V is processed, while the hue component H and the saturation component S remain unchanged. The brightness component V is processed by the MSR algorithm based on the Retinex theory; then the brightness component is further enhanced, and the brightness component V after MSR processing is processed by using contrast-limited adaptive histogram equalization, while maintaining the overall brightness distribution of the image, limiting the contrast enhancement degree of the local area; the unprocessed hue component H, saturation component S and the processed brightness component V are fused and converted back to the RGB color space. At the same time, an adaptive threshold fast corner point extraction strategy is proposed, which can dynamically adjust the threshold according to the grayscale value of the image when the illumination changes, thereby improving the accuracy and robustness of feature point extraction. The feature points extracted by the adaptive threshold are further screened using non-maximum suppression, and finally, the Euroc data set is used to conclude that the present invention can maintain good positioning accuracy and robustness in scenes with insufficient illumination.

[0070] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for extracting ORB feature points under weak light conditions by introducing image enhancement and adaptive threshold, characterized in that: The following steps are involved: Step 1: Obtain the original low-light image; Step 2: Convert the original low-light image from the red, green, and blue (RGB) color space to the hue, saturation, and brightness (HSV) space; Step 3: Extract the brightness component V from the HSV image, and process the brightness component V using a multi-scale algorithm based on Retinex theory; Step 4: Use a contrast-limited adaptive histogram equalization algorithm to optimize the brightness component V processed by the multi-scale algorithm based on Retinex theory; Step 5: Convert the brightness component V optimized by the contrast-limited adaptive histogram equalization algorithm and the unprocessed H component and S component back to the RGB color space to obtain an optimized RGB image; Step 6: For the optimized RGB image, ORB feature points are extracted according to the ORB feature extraction algorithm; wherein, the threshold used to extract FAST corner points in the ORB feature extraction algorithm adopts an adaptive threshold.

2. The ORB feature point extraction method under weak light conditions by introducing image enhancement and adaptive threshold according to claim 1, characterized in that: The determination of the adaptive threshold is specifically as follows: For each pixel point p in the optimized RGB image, the pixel points with the maximum and minimum grayscale values ​​on the circle with the pixel point p as the center point and the radius R are taken as the first pixel points, and the remaining pixel points are taken as the second pixel points; according to the grayscale value of the second pixel point, the local threshold of the pixel point p is obtained; For the optimized RGB image, a preset number of pixels with the largest grayscale value and a preset number of pixels with the smallest grayscale value are found from the optimized RGB image, and a global threshold is obtained according to the preset number of pixels with the largest grayscale value and a preset number of pixels with the smallest grayscale value; The adaptive threshold of pixel p is obtained by proportionally summing the local threshold and the global threshold.

3. The ORB feature point extraction method under weak light conditions with image enhancement and adaptive thresholding according to claim 2 is characterized in that: The calculation formula of the local threshold is as follows: Where: T1 is the local threshold; α is the adaptive parameter; I(x i ) represents the second pixel x around pixel p i The gray value of represents the average grayscale value of all second pixels, and n is the number of pixels in a circle with a radius of R and a pixel p as the center point.

4. The ORB feature point extraction method under weak light conditions with image enhancement and adaptive thresholding according to claim 2, characterized in that: The expression of the global threshold is: Where: T2 is the global threshold; β is the adaptive parameter; I maxj ,I minj They respectively represent the grayscale values ​​of the j-th largest and j-th smallest pixel grayscale values ​​in the optimized RGB image, and J is a preset number.

5. The ORB feature point extraction method under weak light conditions with image enhancement and adaptive thresholding according to claim 2, characterized in that: The adaptive threshold is expressed as: T sum =λ·T1+(1-λ)T2: Among them, λ is an adaptive parameter, and 0<λ<1; T1 is the local threshold; T2 is the global threshold.

6. An ORB feature point extraction device under weak light conditions with image enhancement and adaptive threshold, characterized in that: A module comprising the method for extracting ORB feature points under weak light conditions by introducing image enhancement and adaptive threshold as described in any one of claims 1 to 5.

Images