A visual SLAM positioning method with adaptability to low-light underground mine environment and related device

Abstract

The application belongs to the field of mine intelligent robots, and discloses a visual SLAM positioning method with adaptability to low-light environments in mines and a related device, the method comprising: collecting images in mines; detecting the brightness of the collected images to obtain the brightness of the images; judging whether the brightness of the images meets the standard; performing image enhancement processing on the images that do not meet the brightness standard to make the brightness of the images meet the standard; performing visual initialization on the images that meet the brightness standard; and using the initialized images to perform subsequent positioning processes to realize positioning. The application solves the poor positioning effect caused by low light and uneven light in the application of visual SLAM in underground robots, can automatically identify images that do not meet the light conditions and perform enhancement processing, improves the matching accuracy of the positioning algorithm, and ultimately improves the positioning accuracy of the underground robot.

Description

Technical Field

[0001] This invention relates to the field of intelligent robots for underground mining, and specifically to a visual SLAM localization method and related apparatus that are adaptable to low-light environments in underground mines. Background Technology

[0002] The application of intelligent robot technology in mining engineering mainly focuses on excavation, surface transportation, drilling, and explosives deployment operations. It can effectively improve productivity and safety, and through long-term application, reduce production costs. Especially in the dark, dusty, rugged terrain and large areas of soil cover of underground coal mines, deploying automated and intelligent robots can effectively handle important tasks such as underground work site inspection, exploration of unknown environments, and rescue operations in dangerous areas.

[0003] In complex and unpredictable environments such as coal mine tunnels and goafs, existing robot localization methods are limited: GPS signals are unavailable; harsh environments cause mainstream localization technologies such as RFID and ZigBee to fail or perform poorly underground; and in mine disaster sites, even prior landmarks and maps lose their reliability. Therefore, based on these problems, there is an urgent need for robots to utilize Simultaneous Localization and Mapping (SLAM) technology, which relies on multiple sensors and surrounding environmental information for self-localization, to solve the problem of digital map construction and localization for robots in network-free conditions in coal mines. Furthermore, in engineering environments like underground mines where insufficient lighting and unsuitable viewing angles are common, images often exhibit various degradation phenomena such as low contrast, color distortion, and noise. These degradation phenomena significantly affect the subsequent image analysis results, especially the high-precision environmental information required by visual SLAM systems. Therefore, it is crucial to integrate deep learning methods into visual SLAM systems to improve the quality of input images and ensure system robustness.

[0004] ORB-SLAM3 belongs to the Visual Simultaneous Localization and Mapping (SLAM) technology. Its system allows the camera and IMU to form a visual / inertial joint system, which greatly improves the localization accuracy of the algorithm. The introduction of a multi-map system ensures that the system can operate in weak texture environments.

[0005] EnlightenGAN is a type of Generative Adversarial Network (GAN) that uses an attention-guided U-net as the generator and a dual discriminator to guide global and local information. It achieves illumination enhancement and noise suppression, enhancing real-world low-light images in a completely unsupervised manner. While keeping the model lightweight, it avoids dependence on paired datasets.

[0006] Therefore, based on previous research, how to effectively apply the visual SLAM system that integrates GANs to the localization task of intelligent robots in mines has become a major problem that urgently needs to be solved in this field. Summary of the Invention

[0007] To address the existing technical problems, this invention proposes a visual SLAM localization method and related device that is adaptable to low-light environments in mines. This invention enables accurate localization of intelligent robots in low-light environments in mines.

[0008] The technical solution adopted in this invention is as follows:

[0009] A visual SLAM localization method adapted to low-light environments in underground mines includes the following process:

[0010] Acquire images from underground mines;

[0011] The brightness of the acquired image is detected to obtain the image brightness.

[0012] Determine whether the brightness of the image meets the standard;

[0013] Image enhancement processing is performed on images that do not meet the brightness standards to bring the image brightness up to the standard.

[0014] Perform visual initialization on images that meet the brightness requirements;

[0015] The image initialized visually is used for subsequent localization processes to achieve localization.

[0016] Preferably, the brightness of the acquired image is detected based on both the overall global and cropped local data to obtain the image brightness.

[0017] Preferably, when performing image enhancement processing on images that do not meet the brightness standards, the images that do not meet the brightness standards are enhanced by using the EnlightenGAN network with the self-attention module SAGAN.

[0018] Preferably, the construction process of the EnlightenGAN network with the introduction of the self-attention module SAGAN includes: in the generator network of EnlightenGAN, the attention module extracted directly from the original image is removed on the basis of the U-Net network, and SAGAN is introduced after the fourth convolutional block of the generator network of EnlightenGAN to obtain the EnlightenGAN network with the introduction of the self-attention module SAGAN.

[0019] Preferably, the process of visual initialization for an image with sufficient brightness includes, in sequence, pure visual initialization and IMU initialization.

[0020] Preferably, the IMU initialization uses IMU initialization based on analytical solutions.

[0021] Preferably, the subsequent positioning process includes: performing VIBA optimization, and using the VIBA optimization results and the ORB-SLAM3 algorithm process to obtain the positioning result.

[0022] The present invention also provides a visual SLAM positioning system with adaptability to low-light environments in mines, comprising:

[0023] Image acquisition module: used to acquire images from underground mines;

[0024] Brightness detection module: Used to detect the brightness of the acquired image and obtain the brightness of the image;

[0025] Judgment module: used to determine whether the brightness of the image meets the standard;

[0026] Image enhancement processing module: Used to enhance images that do not meet the brightness standards, so that the image brightness meets the standards;

[0027] Initialization module: Used for visual initialization of images that meet brightness standards;

[0028] Localization module: Used to perform subsequent localization processes using images initialized visually, thereby achieving localization.

[0029] The present invention also provides an electronic device, comprising:

[0030] One or more processors;

[0031] A storage device on which one or more programs are stored;

[0032] When the one or more programs are executed by the one or more processors, the one or more processors implement the visual SLAM localization method with adaptability to low-light environments in mines as described above.

[0033] The present invention also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the visual SLAM localization method with adaptability to low-light environments in mines as described above.

[0034] The present invention has the following beneficial effects:

[0035] In the visual SLAM localization method adapted to low-light environments in mines, the brightness of the acquired image is detected to determine whether the brightness meets the standard; for images with insufficient brightness, image enhancement processing is performed. Therefore, this invention can improve the quality of the input image, thereby increasing the number of feature points extracted, improving the feature matching quality, and ultimately improving the system localization accuracy in underground environments.

[0036] Furthermore, considering the low-light and unevenly lit images input to the visual SLAM system from underground mines, the EnlightenGAN-based image augmentation network can effectively detect the presence of dark areas and takes into account the dependencies between different regions of the image, achieving better enhancement results than common image augmentation networks. From the perspective of the high initialization failure rate and frequent system resets of visual SLAM systems underground, the system initialization strategy is more adapted to the mining environment, improving the response speed of the visual SLAM system. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the visual SLAM localization method with adaptability to low-light environments in mines provided in this embodiment of the invention.

[0039] Figure 2 This is a schematic diagram of the brightness detection module provided in an embodiment of the present invention;

[0040] Figure 3 This is the basic visual SLAM process provided in the embodiments of the present invention.

[0041] Figure 4 This is a structural block diagram of the visual SLAM positioning system with adaptability to low-light environments in mines, as described in this invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0043] This invention provides a visual SLAM localization method adapted to low-light environments in underground mines, comprising the following steps:

[0044] Step 1: Construct a non-matched image dataset containing images of both normal and low-light conditions. Images of common underground mining scenes, including roadways, chambers, locomotives, and working faces, are collected through field investigations, and the dataset size is increased through image augmentation.

[0045] Step 2: Detect image brightness based on global and cropped local values. Since the device used is a stereo camera, the input to the ORB-SLAM3 system is black and white frames extracted from black and white video. The brightness is evaluated using the gray-scale variance method without reference. In order to accurately detect dark lighting and uneven lighting, in addition to global detection, cropping detection is performed on images with moderate detection effect evaluation.

[0046] Step 3: Construct the EnlightenGAN network with the self-attention module SAGAN. In the generator network of EnlightenGAN, the attention module extracted directly from the original image is removed on the basis of the U-Net network. Instead, SAGAN is introduced after the fourth convolutional block to improve the dependency between different regions.

[0047] Step 4: Pure visual initialization. After inputting the first normal lighting image or enhanced image, the system changes from NO_IMAGES_YET (no image input) to NOT_INITIALIZED to enter initialization. This invention prioritizes pure visual initialization and directly enters the positioning process after visual initialization is completed.

[0048] Step 5: IMU initialization. The new IMU initialization strategy is to perform pure visual pose optimization (step 4) before the IMU initialization conditions are met, and the visual / IMU joint optimization (VIBA) start time limit is removed to avoid repeated system restarts that may affect the positioning effect.

[0049] Step 6: Subsequent localization process. After the IMU initialization conditions are met, VIBA is performed to complete the entire visual / inertial initialization and optimization, and then the ORB-SLAM3 algorithm process continues.

[0050] Example

[0051] This embodiment aims to provide a visual SLAM localization method and related apparatus adapted to low-light environments in underground mines. The invention creates a dataset of 3560 images based on real-world images collected from actual underground coal mines; designs an image brightness detection module; introduces the self-attention algorithm SAGAN into the EnlightenGAN image enhancement network to enhance the dependencies between different regions of the image; and redesigns the SLAM system IMU initialization process to improve the practicality of the localization method in actual underground environments. It can solve the problem of poor localization performance caused by low light and uneven lighting in the application of visual SLAM in underground robots, thereby improving the localization accuracy of underground robots.

[0052] See Figures 1 to 3 This embodiment of the visual SLAM localization method with adaptability to low-light environments in mines includes the following steps:

[0053] Step 1: Construct a non-matching image dataset containing images with normal lighting and images with dark lighting:

[0054] In step 1 of this embodiment, in order to meet the data requirements of the non-matching training dataset in the training GAN network and improve the image quality of the dark area of ​​the input mine image in the SLAM system, it is necessary to take images under three lighting conditions: normal lighting, uneven lighting, and dark lighting. Uneven lighting includes large areas of shadow occlusion, backlit objects, and the road ahead is normally lit but the positioning system carrier itself is under dark lighting.

[0055] The image dataset in this embodiment was captured using a Huawei P40 Pro in professional mode, with a resolution of 2340x1080 pixels, manual focus, and ISO 800. Scenes include, but are not limited to, tunnels, chambers, and underground locomotives. To closely resemble the actual operation of future unmanned transport vehicles and robots, the sampling and shooting were conducted near the center of the tunnel and the shooting device was kept 1m to 1.5m off the ground to ensure the usability of the dataset.

[0056] After capturing the original dataset, to optimize and expand it, overexposed and blurry images were first manually removed. The images were then cropped to 600×400 pixels for GAN network training. The final training set included images from various scenes as much as possible, and these images were flipped by 90° or 180°, mirrored, and randomly enlarged, resulting in 4670 images.

[0057] Step 2: Detect image brightness based on overall global and cropped local data.

[0058] To save computational resources and prevent the enhanced image from actually worsening the effect, this invention employs a global and cropped approach to image brightness detection. In this embodiment, the visual sensor used is the OAK-D-PRO binocular camera, detecting only the left-eye image. Since the input to the SLAM system is a black-and-white frame extracted from a black-and-white video, brightness is evaluated using a no-reference gray-scale variance method. Evar represents the image gray-scale variance, u represents the average gray-scale value of the image pixels, MN represents the image size, F(i,j) represents the pixel coordinates, and (i,j) represents the corresponding gray-scale value.

[0059]

[0060]

[0061] In order to accurately detect low light and uneven lighting, in addition to global detection, images with medium detection performance are cropped for detection. If the cropped image slices have low evaluation, the entire image is enhanced.

[0062] Step 3: Construct the EnlightenGAN network with the self-attention module SAGAN:

[0063] When the image brightness detection in the brightness detection module fails to meet the standard, the image is sent to the EnlightenGAN network, which incorporates the self-attention module SAGAN, for image enhancement processing.

[0064] The self-attention module, based on SAGAN, introduces the self-attention mechanism into GAN image generation to model long-distance relationships between pixels. This approach captures global dependencies while maintaining low computational cost, making it commonly used in image generation tasks. The feature map x input to the self-attention module is C×N, where N = width*height, representing the total number of pixels. The input feature map is divided into three parts: the first part W... f Given c^×c, we get f(x) = W f x, i.e., c^×c×c×n=c^×n, the second part W g Given c^×c, we get g(x) = W g x, i.e., c^×c×c×n=c^×n, the third part h(x) is c×c, so h(x)=W h x, that is, c × c × c × n = c × n. β j,i This indicates the degree of attention the module pays to the i-th position when composing the j-th region:

[0065]

[0066] This is used to perform an inner product with the original feature map input to the self-attention module, integrating the global and local spatial information of the image, resulting in:

[0067]

[0068] For the EnlightenGAN network that incorporates a self-attention module, L in the network generator... Global G In the middle, D Ra (x r ,x f D represents the probability that the real image is more realistic than the generated image. Ra (x f ,x r ) represents the probability that the generated image is more realistic than the real image, using least squares as the activation function. Local G In the example, adversarial loss D(x) is calculated using 5 randomly pruned patches. r D(x) represents the probability that the discriminator judges the real image as real. f The value represents the probability that the discriminator determines the generated image to be real. L SFP It is a self-feature preservation loss, through Φ ij () Determine the VGG feature distance between the input dark image and its enhanced output image, I L The input is a low-light image, W i,j and H i,j The size of the extracted feature map:

[0069]

[0070]

[0071]

[0072] In the network discriminator, a dual-scale discriminator is used to guide global and local information. In addition to using the real image and the generated image as the adversarial loss of the whole image, the two images are also randomly cropped to improve the network's image enhancement ability under uneven lighting conditions.

[0073] Step 4, Pure Visual Initialization:

[0074] After the ORB-SLAM3 system completes the transitions from "SYSTEM_NOT_READY" (indicating that the system has not yet completed all the preparations before operation) to "NO_IMAGES_YET" (indicating that the system has no image input) to "NOT_INITIALIZED" (indicating that the system has no image input), it inputs the first frame image. After creating the Frame object, extracting ORB features, and performing stereo feature matching, it officially enters the stereo initialization stage.

[0075] In the initialization function, the ORB-SLAM3 system checks if the number of feature points in the current frame exceeds a set threshold (set to 500). If the condition is met, it continues to attempt initialization. This involves a series of steps: setting the pose of the current frame to the origin, converting the current frame into a keyframe and inserting it into the map, adding successfully matched feature points from the current frame (based on a depth calculated using stereo cameras being greater than 0) to the map and associating them with the keyframe, and setting the previous frame as the current frame.

[0076] The system then estimates the pose based on visual-only data and performs optimizations based on visual-only data.

[0077] Step 5: IMU initialization. This embodiment uses IMU initialization based on analytical solutions.

[0078] In the original ORB-SLAM3 system, VIBA requires at least 15 seconds to complete initialization. This is prone to initialization failure for downhole robots with weak scene textures and slow movement, leading to repeated restarts. This invention redesigns the system initialization strategy. Before the IMU initialization conditions are met, the system performs pure visual pose optimization first, and the VIBA optimization start time limit is removed, avoiding repeated system restarts that could affect the localization effect.

[0079] The IMU initialization based on analytical solutions obtains the accelerometer bias, gravity direction, and scale factor using a maximum likelihood framework. First, the gyroscope bias is estimated based on gyroscope measurements and the orientation of keyframes at times k and k+1. Then, the optimization problem is solved iteratively using g2o.

[0080]

[0081] Obtain the gyroscope bias b g ,in Represents the gyroscope residual. This represents the covariance matrix.

[0082] Then, the accelerometer deviation b is found from the measured acceleration value and relative attitude. a The gravity direction g and the reconstruction scale factor s are considered, and three consecutive keyframes are taken into account to reduce complexity.

[0083] Finally, given that the magnitude of the gravity vector G is 9.81, the constrained optimization problem is solved using matrix calculations and the Lagrange multiplier method.

[0084]

[0085] subject to |g|=G

[0086] The accelerometer deviation b is obtained by solving for the roots of a polynomial. a The direction of gravity g and the scale factor s, where s represents the direction of gravity g and the scale factor s. Accelerometer residual.

[0087] Step 6, Subsequent positioning process:

[0088] After the first few steps, the SLAM system has achieved image brightness detection, enhanced images with uneven lighting / dark lighting, and a new IMU initialization strategy, enabling it to continue to complete the localization task of the downhole robot in the subsequent localization process.

[0089] In one embodiment, the image data is collected by a visual-inertial system consisting of a camera and an IMU on the micro-aircraft, and is categorized into three types—easy, medium, and difficult—based on the micro-aircraft's speed and ambient lighting conditions. The data is evaluated using the root mean square error (RMSE) in the absolute trajectory error (ATE), which represents the global consistency of the trajectory. RMSE reflects the system's accuracy and robustness better than the mean and median, where Y... i f(x) is the true value. i ) represents the algorithm test value, and F(x) represents f(x) i The average value, where n is the number of observations:

[0090]

[0091] Compared with the original ORB-SLAM3 system, Table 1 is obtained:

[0092] Table 1

[0093]

[0094] As shown in Table 1, when the method of the present invention is run on subsets 03, 04, and 05 with low light conditions, the RMSE decreases significantly, with an average decrease of 12.17%, which fully verifies that the method of the present invention can improve the system accuracy in low light conditions.

[0095] This invention also provides a system based on the visual SLAM localization method with adaptability to low-light environments in mines as described above, such as... Figure 4 As shown, the system includes:

[0096] Image acquisition module: used to acquire images from underground mines;

[0097] Brightness detection module: Used to detect the brightness of the acquired image and obtain the brightness of the image;

[0098] Judgment module: used to determine whether the brightness of the image meets the standard;

[0099] Image enhancement processing module: Used to enhance images that do not meet the brightness standards, so that the image brightness meets the standards;

[0100] Initialization module: Used for visual initialization of images that meet brightness standards;

[0101] Positioning module: Used to perform subsequent positioning processes using the initialized image to achieve positioning.

[0102] The embodiments of the present invention also provide corresponding electronic devices and computer-readable storage media for implementing the solutions provided in the embodiments of the present invention.

[0103] The device includes a memory and a processor. The memory stores instructions or code, and the processor executes the instructions or code to enable the device to perform the visual SLAM localization method with low-light environment adaptability in mines as described in any embodiment of this application.

[0104] The storage medium stores a computer program, which, when executed by a processor, implements the visual SLAM localization method with low-light environment adaptability in mines as described in any embodiment of this application.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual SLAM positioning method with adaptability to low-light underground mine environment, characterized in that, The process includes the following: Acquire images from underground mines; The brightness of the acquired image is detected to obtain the image brightness. Determine whether the brightness of the image meets the standard; Image enhancement processing is performed on images that do not meet the brightness standards to bring the image brightness up to the standard. When performing image enhancement processing on images that do not meet the brightness standards, the images that do not meet the brightness standards are enhanced by using the EnlightenGAN network with the self-attention module SAGAN. The construction process of the EnlightenGAN network with the introduction of the self-attention module SAGAN includes: in the generator network of EnlightenGAN, the attention module extracted directly from the original image is removed from the U-Net network, and SAGAN is introduced after the fourth convolutional block of the generator network of EnlightenGAN to obtain the EnlightenGAN network with the introduction of the self-attention module SAGAN. Visual initialization is performed using an image that meets the brightness standard; The image initialized visually is used for subsequent localization processes to achieve localization. 2.The visual SLAM positioning method with low-light environment adaptability in a mine according to claim 1, wherein, The brightness of the acquired image is obtained by detecting the brightness of the overall global and cropped local values.

3. The visual SLAM localization method with adaptability to low-light environments in mines according to claim 1, characterized in that, The process of visual initialization for an image with sufficient brightness includes: pure visual initialization and IMU initialization.

4. A visual SLAM localization method with adaptability to low-light environments in mines according to claim 3, characterized in that, The IMU initialization uses IMU initialization based on the analytical solution.

5. A visual SLAM localization method with adaptability to low-light environments in mines according to claim 3, characterized in that, The subsequent positioning process includes: performing VIBA optimization, and using the VIBA optimization results and the ORB-SLAM3 algorithm process to obtain the positioning result.

6. A visual SLAM positioning system adapted to low-light environments in underground mines, characterized in that, For implementing the visual SLAM localization method with adaptability to low-light environments in mines as described in any one of claims 1-5, the system comprises: Image acquisition module: used to acquire images from underground mines; Brightness detection module: Used to detect the brightness of the acquired image and obtain the brightness of the image; Judgment module: used to determine whether the brightness of the image meets the standard; Image enhancement processing module: Used to enhance images that do not meet the brightness standards, so that the image brightness meets the standards; Initialization module: Used for visual initialization of images that meet brightness standards; Localization module: Used to perform subsequent localization processes using images initialized visually, thereby achieving localization.

7. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the visual SLAM localization method with adaptability to low-light environments in mines as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the visual SLAM localization method with adaptability to low-light environments in mines as described in any one of claims 1 to 5.

Images