Coal mine subsurface environment sensing method and apparatus, and storage medium
By adaptively fusing visual and lidar environmental images, and combining the characteristics of cameras and radar for calibration and distortion compensation, the problem of insufficient information fusion in underground coal mine environmental perception is solved, the accuracy of environmental perception is improved, and various underground applications are supported.
Patent Information
- Application Number
- PCT/CN2023/116524
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2023-09-01
- Publication Date
- 2024-11-21
AI Technical Summary
Existing methods for fusing RGB images with LiDAR data suffer from insufficient information fusion in underground coal mine environmental perception, leading to inadequate accuracy.
By using adaptive fusion technology, visual environmental images and lidar environmental images are fused with different weights according to the environmental type of the area to be sensed in the underground coal mine. The characteristics of the camera and radar are combined for calibration and distortion compensation to obtain high-precision three-dimensional environmental information.
It improves the accuracy of underground environmental perception in coal mines, enabling accurate identification of environmental features and objects in different scenarios, and supports applications such as coal mine road environmental perception, visual measurement systems, unmanned mining truck navigation, and mine search and rescue robots.
Smart Images

Figure CN2023116524_21112024_PF_FP_ABST
Abstract
Description
A method, device and storage medium for sensing underground coal mine environment
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on the Chinese patent application with application number 202310552707.2 and application date of May 16, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field
[0003] The present disclosure relates to the technical field of intelligent coal operation, and in particular to a method, device and storage medium for sensing the underground environment of a coal mine. Background Art
[0004] The constantly changing scene, structure, lighting, and signals in coal mine operating environments make coal mining operations difficult and prone to accidents. With the rapid development of computer, communications, and microelectronics technologies, multi-sensor data fusion technology has gained widespread attention and application. Its application in coal mining operations and environmental perception has significantly reduced the difficulty of underground coal mine operations.
[0005] Among them, the fusion technology based on RGB images and lidar data has attracted much attention due to its practicality and high performance in depth perception. The current work mainly explores two different fusion methods: lidar and monocular image fusion, and lidar and stereo image fusion.
[0006] However, the depth estimation of images currently performed through the above-mentioned fusion method is usually based on pixel-based regression, which has inherent unreliability and ambiguity, and the information fusion is insufficient, and the advantages of sensor characteristics are not fully utilized, resulting in insufficient accuracy of the coal mine underground environment perception results.
[0007] Summary of the Invention
[0008] The present disclosure aims to solve one of the technical problems in the related art at least to a certain extent.
[0009] To this end, the present disclosure proposes a method, device and storage medium for coal mine underground environment perception, so as to fully integrate RGB images with lidar data and improve the accuracy of coal mine underground environment perception results.
[0010] To achieve the above objectives, an embodiment of the present disclosure provides a method, including:
[0011] Obtain visual environment images and lidar environment images of the area to be sensed in the coal mine;
[0012] Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image;
[0013] The fused image is input into task programs in different scenes to output environmental perception results of the area to be perceived in the corresponding scenes.
[0014] In some possible implementations, the fusing of the visual environment image and the lidar environment image based on the environment type of the area to be sensed includes:
[0015] When the environment type of the area to be sensed is a target environment, when fusing the visual environment image and the lidar environment image, the lidar environment image is used as a reference image and the visual environment image is used as a supplementary image;
[0016] When the visual environment image and the lidar environment image are fused, the pixel value weight of the reference image is greater than the pixel value weight of the supplementary image.
[0017] In some possible implementations, the fusing of the visual environment image and the lidar environment image based on the environment type of the area to be sensed includes:
[0018] When the environment type of the area to be perceived is a non-target environment, when the visual environment image and the lidar environment image are fused, when the imaging distance of the area to be perceived is a first-level distance, the visual environment image is used as a reference image and the lidar environment image is used as a supplementary image; when the imaging distance of the area to be perceived is a second-level distance, the lidar environment image is used as a reference image; the first-level distance and the second-level distance are divided according to distance.
[0019] In some possible implementations, fusing the visual environment image and the lidar environment image includes:
[0020] When acquiring the visual environment image, obtaining camera world three-dimensional coordinate information of the visual environment image based on the relationship between the spatial coordinate system and the world coordinate system during camera calibration;
[0021] The visual environment image and the lidar environment image are fused based on the world three-dimensional information of the lidar environment image obtained by the lidar, and the camera world three-dimensional coordinate information is used as a reference.
[0022] In some possible implementations, fusing the visual environment image and the lidar environment image includes:
[0023] First environmental feature information is extracted from the three-dimensional environmental space information corresponding to the visual environment image, and second environmental feature information is extracted from the three-dimensional environmental space information corresponding to the lidar environment image, and the obtained first environmental feature information and second environmental feature information are fused.
[0024] In some possible implementations, inputting the fused image into task programs in different scenarios to output environmental perception results of the area to be perceived in the corresponding scenarios includes:
[0025] In the case where the scene is a traffic environment scene, identifying moving objects and performing pedestrian recognition through target detection;
[0026] When the scene is a denoising scene, performing denoising and defogging through image enhancement;
[0027] In the case where the scene is a track line scene, edge lines are identified through field segmentation;
[0028] When the scene is a ground environment scene in an underground coal mine, the ground environment is extracted through semantic segmentation.
[0029] In some possible implementations, the method further includes:
[0030] The laser radar is subjected to distortion compensation based on the angular velocity and linear velocity of the laser radar to obtain the laser radar environment image.
[0031] To achieve the above objectives, a second embodiment of the present disclosure provides a coal mine underground environment sensing device, comprising:
[0032] An image acquisition module is used to acquire visual environment images and lidar environment images of the area to be sensed in the coal mine;
[0033] An image fusion module, configured to fuse the visual environment image and the lidar environment image based on the environment type of the area to be sensed to obtain a fused image;
[0034] The environmental perception module is used to input the fused image into task programs under different scenes to output environmental perception results of the area to be perceived in the corresponding scene.
[0035] In some possible implementations, the image acquisition module is used to: when the environment type of the area to be perceived is the target environment, when the visual environment image and the lidar environment image are fused, use the lidar environment image as the baseline image and the visual environment image as the supplementary image; when the visual environment image and the lidar environment image are fused, the pixel value weight of the baseline image is greater than the pixel value weight of the supplementary image.
[0036] In some possible implementations, the image fusion module is used to: when the environment type of the area to be perceived is a non-target environment, when the visual environment image and the lidar environment image are fused, when the imaging distance of the area to be perceived is a first-level distance, use the visual environment image as a reference image and the lidar environment image as a supplementary image; when the imaging distance of the area to be perceived is a second-level distance, use the lidar environment image as a reference image; the first-level distance and the second-level distance are divided according to distance.
[0037] In some possible implementations, the image fusion module is used to: when acquiring the visual environment image, obtain the camera world three-dimensional coordinate information of the visual environment image based on the relationship between the spatial coordinate system and the world coordinate system during the camera calibration process; and fuse the visual environment image and the lidar environment image based on the world three-dimensional information of the lidar environment image obtained by the lidar, and use the camera world three-dimensional coordinate information as a reference.
[0038] In some possible implementations, the image fusion module is used to extract first environmental feature information from the three-dimensional environmental space information corresponding to the visual environment image and second environmental feature information from the three-dimensional environmental space information corresponding to the lidar environment image, and fuse the obtained first environmental feature information and second environmental feature information.
[0039] In some possible implementations, the environmental perception module is used to: when the scene is a traffic environment scene, identify moving objects and perform pedestrian recognition through target detection; when the scene is a denoising scene, perform denoising and defogging through image enhancement; when the scene is a track line scene, identify edge lines through field segmentation; when the scene is a ground environment scene in a coal mine, extract the ground environment through semantic segmentation.
[0040] To achieve the above objectives, the third embodiment of the present disclosure provides a coal mine underground environment perception device, including a memory, a transceiver, and a processor:
[0041] A memory for storing a computer program; a transceiver for transmitting and receiving data under the control of the processor; and a processor for reading the computer program in the memory and performing the following operations:
[0042] Obtain visual environment images and lidar environment images of the area to be sensed in the coal mine;
[0043] Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image;
[0044] The fused image is input into task programs in different scenes to output environmental perception results of the area to be perceived in the corresponding scenes.
[0045] In order to achieve the above-mentioned objectives, the fourth aspect embodiment of the present disclosure proposes a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to execute a coal mine underground environment perception method proposed in the first aspect embodiment of the present disclosure.
[0046] In order to achieve the above-mentioned purpose, the fifth embodiment of the present disclosure proposes a computer program product. When the instruction processor in the computer program product is executed, the coal mine underground environment perception method proposed in the first embodiment of the present disclosure is executed.
[0047] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: obtaining a visual environment image and a LiDAR environment image of the area to be sensed in a coal mine; fusing the visual environment image and the LiDAR environment image based on the environment type of the area to be sensed; and inputting the fused image into a multi-tasking program containing different scenarios to output the environmental perception results of the area to be sensed in the corresponding scenario. Through adaptive fusion, the visual environment image and the LiDAR environment image are fully integrated, accurately reflecting the information of the area to be sensed, and improving the accuracy of the environmental perception results in coal mines.
[0048] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The above and / or additional aspects and advantages of the present disclosure will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0050] FIG1 is a schematic diagram of a process of a method for sensing an underground coal mine environment according to an embodiment of the present disclosure;
[0051] FIG2 is a flow chart of another method for sensing an underground coal mine environment according to an embodiment of the present disclosure;
[0052] FIG3 is a flow chart of another method for sensing an underground coal mine environment according to an embodiment of the present disclosure;
[0053] FIG4 is a schematic structural diagram of a coal mine underground environment sensing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0054] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0055] A method for sensing an underground coal mine environment according to an embodiment of the present disclosure will be described below with reference to the accompanying drawings.
[0056] FIG1 is a flow chart of a method for sensing an underground coal mine environment according to an embodiment of the present disclosure.
[0057] Cameras offer advantages in environmental perception, such as rich semantics and detailed information. However, they can provide inaccurate position estimation, are susceptible to interference from close proximity (such as dusty and foggy environments), and are susceptible to interference from the environment (e.g., high dust and fog). Radar offers advantages in environmental perception, such as accurate position estimation, long range detection, and resistance to environmental interference, but suffers from insufficient detail and inaccurate classification at long distances. Therefore, building on the respective strengths of cameras and radar, environmental perception is performed by fusing RGB images with LiDAR data to obtain high-precision three-dimensional environmental information. However, current fusion technologies do not fully integrate information, resulting in insufficient accuracy in environmental perception results in coal mines.
[0058] To address this issue, an embodiment of the present disclosure provides a method for sensing an underground coal mine environment to improve the accuracy of the results of sensing an underground coal mine environment. As shown in FIG1 , the method includes steps 101 to 103 .
[0059] Step 101: Acquire a visual environment image and a lidar environment image of the area to be sensed in a coal mine.
[0060] The area to be sensed is the area in the coal mine where environmental perception is required.
[0061] In some embodiments, an RGB image of the area to be sensed is collected by a camera as a visual environment image, and a lidar environment image of the area to be sensed is collected by a lidar.
[0062] Step 102: Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image.
[0063] Based on the different environmental types of the area to be perceived, the visual environment image and the lidar environment image are fused accordingly to obtain a fused image. The visual environment image and the lidar environment image are fully fused through adaptive fusion to accurately reflect the information of the area to be perceived.
[0064] Step 103: Input the fused image into the task program under different scenes to output the environmental perception result of the area to be perceived in the corresponding scene.
[0065] Taking the fused image as input, a multi-task program corresponding to multiple scenes is used to extract information of the area to be perceived, so as to obtain the environmental perception results of the area to be perceived in different scenes.
[0066] In some embodiments, the environmental perception result may be a recognition result of environmental features.
[0067] In the disclosed embodiments, a visual image of the environment and a LiDAR image of the area to be sensed are acquired underground in a coal mine. These images are then fused based on the environment type of the area to be sensed. The fused images are then fed into a multi-tasking program that includes different scenarios to output environmental perception results for the area to be sensed in the corresponding scenario. This adaptive fusion method fully integrates the visual image and LiDAR image, accurately reflecting information about the area to be sensed and improving the accuracy of environmental perception results in underground coal mines.
[0068] In order to clearly illustrate the previous embodiment, the embodiment of the present disclosure provides another method for sensing the underground environment of a coal mine. FIG2 is a flow chart of another method for sensing the underground environment of a coal mine provided by the embodiment of the present disclosure.
[0069] As shown in FIG. 2 , the coal mine underground environment perception method may include the following steps 201 to 203 .
[0070] Step 201: Acquire a visual environment image and a lidar environment image of the area to be sensed in a coal mine.
[0071] In some embodiments, the laser radar is subjected to distortion compensation based on the angular velocity and linear velocity of the laser radar to obtain a laser radar environment image.
[0072] Due to the inherent characteristics of cameras and radars, there will be problems with camera and radar calibration, synchronization errors, and looseness during movement. At the same time, in actual use, camera images will be missing, and radar will lose point cloud data and frames. In harsh environments, the camera will have blurred images and the radar will have condensation.
[0073] To address the above issues, the camera is calibrated for intrinsic and extrinsic parameters, and offset compensation is performed.
[0074] It should be noted that camera intrinsic calibration refers to the conversion from camera to image coordinate system; camera extrinsic calibration refers to the conversion from 3D space coordinates to camera coordinate system.
[0075] Radar attitude adjustment models and compensates for the radar's position information and the relative position relationship between the previous and next frames.
[0076] As for the synchronization error, there is a time difference in the radar scanning a circle, so the camera and lidar are synchronized.
[0077] It should be noted that hardware synchronization errors can be resolved by synchronizing the timestamps of different sensors. Distortion compensation is achieved by calculating the radar's own angular velocity and linear velocity to restore the true laser state.
[0078] In some embodiments, the GPS timestamp synchronization method involves first determining whether the sensor hardware supports this method. If so, the sensor data packets will have a global timestamp. These timestamps are based on GPS, thus using the same clock rather than the clock of each sensor. However, different sensors have different data rates. For example, the frequency of a lidar is 10Hz, while that of a camera is 25 / 30Hz. This means that data from different sensors may still experience delays. In this case, the nearest frame can be found by finding adjacent timestamps.
[0079] In some embodiments, a hard synchronization method uses the LiDAR as the trigger source for other sensors. Only when the LiDAR rotates to a certain angle does the camera at that angle trigger. This can significantly reduce timing errors. This time synchronization solution can be implemented in hardware, mitigating errors caused by timestamp lookups. This significantly reduces synchronization errors and improves data alignment.
[0080] Distortion compensation ensures that the point cloud at any moment and the camera at the corresponding position at that moment can be triggered simultaneously, so that the timestamps of the visual environment image and the lidar environment image are synchronized.
[0081] For motion compensation, when the object moves, the environment will change, and the lidar is also scanning supplementary information synchronously.
[0082] In some embodiments, a fault prompt is given when the radar loses part or all of the point cloud and / or condensation forms on the radar.
[0083] Step 202: Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image.
[0084] It should be noted that when obtaining the fused image, the front fusion method, the back fusion method, or the combination of the front fusion and the back fusion method can be adaptively selected according to the environment type of the area to be perceived to fuse the visual environment image and the lidar environment image.
[0085] Among them, the process of front fusion is: when collecting the visual environment image, based on the relationship between the spatial coordinate system and the world coordinate system in the camera calibration process, the camera world three-dimensional coordinate information of the visual environment image is obtained, and the world three-dimensional information of the lidar environment image obtained by the lidar is used as a reference to fuse the visual environment image and the lidar environment image.
[0086] Among them, the post-fusion process is: extracting the first environmental feature information from the three-dimensional environmental space information corresponding to the visual environment image, extracting the second environmental feature information from the three-dimensional environmental space information corresponding to the lidar environment image, and fusing the obtained first environmental feature information and second environmental feature information.
[0087] In some embodiments, when the environment type of the area to be sensed is the target environment, when the visual environment image and the lidar environment image are fused, the lidar environment image is used as the reference image and the visual environment image is used as the supplementary image.
[0088] It should be noted that the target environment is a foggy environment, a dusty environment or a dark environment, which is determined by an environment recognition model, and the environment recognition model is obtained by learning image features of different environment types.
[0089] When fusing the visual environment image and the lidar environment image, the pixel value weight of the reference image is greater than the pixel value weight of the supplementary image.
[0090] Furthermore, an environment recognition model is added to the camera, which automatically recognizes and determines whether the area to be sensed is the target environment by learning the image features of different environment categories.
[0091] The pixel value weight of the reference image is greater than the pixel value weight of the supplementary image. If the sum of the pixel value weight of the reference image and the pixel value weight of the supplementary image is 1, the pixel value weight of the reference image is greater than 0.5.
[0092] In some embodiments, when the environment type of the area to be perceived is a non-target environment, when the visual environment image and the lidar environment image are fused, when the imaging distance of the area to be perceived is a first-level distance, the visual environment image is used as the reference image and the lidar environment image is used as the supplementary image; when the imaging distance of the area to be perceived is a second-level distance, the lidar environment image is used as the reference image and the visual environment image is used as the supplementary image; the first-level distance and the second-level distance are divided according to the distance.
[0093] It's important to note that the distinction between primary and secondary ranges is determined by hardware specifications. Generally, cameras in coal mines are only used to perceive environmental details at close range, typically within 10 meters. Radar and lidar, for example, use millimeter-wave radar, typically 4-12mm, and have an effective operating range much longer. Lidar uses lasers with wavelengths typically between 900-1500nm, enabling more detailed and precise vision.
[0094] Based on the above hardware parameters, in some embodiments, an imaging distance less than 10 meters is a first-level distance, and otherwise it is a second-level distance.
[0095] It can be understood that no matter whether the fusion of the visual environment image and the lidar environment image is performed through the front fusion method or the back fusion method, the benchmark can be determined based on the distance.
[0096] By fusing images under different environmental types, using an image that can better reflect the precise information of the area to be perceived as a reference image, and then using supplementary images as an aid to improve the reference image, it is possible to obtain accurate fused images of the area to be perceived under different environmental types, obtain denser and more accurate disparity maps, improve the accuracy of environmental perception, and provide an accurate depth estimation for the entire environmental perception method. This can provide reliable image information and technical support for applications such as coal mine road environmental perception, visual measurement systems, unmanned mine car navigation systems, and mine search and rescue robots.
[0097] Step 203: Input the fused image into the task program in different scenes to output the environmental perception result of the area to be perceived in the corresponding scene.
[0098] For step 203, reference may be made to the relevant description of the corresponding steps in the aforementioned embodiment, which will not be described in detail in the present embodiment.
[0099] An embodiment of the present disclosure provides another method for sensing an underground coal mine environment. FIG3 is a flow chart of another method for sensing an underground coal mine environment provided by an embodiment of the present disclosure.
[0100] As shown in FIG. 3 , the method may include the following steps 301 to 304 .
[0101] Step 301: Acquire a visual environment image and a lidar environment image of the area to be sensed in a coal mine.
[0102] Step 302: Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image.
[0103] For step 301 and step 302, reference can be made to the relevant description of the corresponding steps in the aforementioned embodiment, which will not be repeated in the embodiment of the present disclosure.
[0104] Step 303: Input the fused image into the task program in different scenes to output the environmental perception result of the area to be perceived in the corresponding scene.
[0105] It should be noted that multi-tasking programs for different scenarios can be executed in parallel or serially, depending on actual computing power and the optimal path. That is, in some embodiments, the fused image can be input into different scenarios for parallel functional implementation; in other embodiments, the corresponding scenario can be identified first, the corresponding program selected, and then the fused image can be input into the corresponding scenario program to obtain the output result.
[0106] In some embodiments, when the scene is a traffic environment scene, moving objects are identified and pedestrians are recognized through target detection.
[0107] Traffic environment scenes refer to scenes that include traffic environments. Traffic environments include many moving targets, such as moving people, vehicles, and objects, and are constantly changing. Target detection can track the movement trajectory of objects. Therefore, for traffic environments with many moving targets, target detection is used to identify moving objects and pedestrians.
[0108] In some embodiments, when the scene is an obstacle scene, obstacles are identified through object detection.
[0109] Obstacle scenes refer to scenes that contain static obstacles, such as railings and rock piles. Target detection can locate different objects and identify the obstacles that need to be detected in the obstacle scene.
[0110] In some embodiments, when the scene is a denoising scene, denoising and defogging are performed through image enhancement.
[0111] A denoising scene refers to a scene containing an environment type that requires denoising, such as a foggy environment, a dusty environment, or a dark environment, that is, a target environment.
[0112] In some embodiments, when the scene is a track line scene, edge lines are identified through field segmentation.
[0113] Track line scenarios refer to scenarios that include track lines. Underground coal mines use underground rail transport vehicles to transport coal, explosives, and other materials. Some road surfaces may contain track lines.
[0114] In some embodiments, when the scene is a ground environment scene in a coal mine, the ground environment is extracted through semantic segmentation.
[0115] The ground environment scene refers to the scene that only exists on the underground ground and does not contain other obstacles, moving objects, etc.
[0116] Step 304 : Apply the environmental perception results of the area to be sensed in different scenarios to different devices to realize the control and application of the equipment in the coal mine.
[0117] Since different devices use environmental perception results for different purposes, the weight of the environmental perception results in each scenario is first determined based on the device to which the environmental perception results are to be applied.
[0118] In some embodiments, the environmental perception results need to be applied to a constantly moving device, such as a mobile robot, and the weights of the environmental perception results are arranged as follows: ground environment scene > track line scene > obstacle scene > traffic environment scene > denoising scene.
[0119] In some embodiments, as a device for fixed-point recognition, the weight arrangement of its environmental perception results is: denoising scene > ground environment scene > traffic environment scene > track line scene > obstacle scene.
[0120] After obtaining the weights of environmental perception results in different scenarios, decisions are made by controlling downhole equipment.
[0121] In some embodiments, for a mobile robot, after obtaining the environmental perception results, the environmental perception results are transmitted to its own processor. The processor sends commands to the controller, mainly the motor controller, according to the expert system or its own preset processing commands, and realizes steering, stopping, acceleration, emergency stop and other operations by controlling the corresponding speed.
[0122] In some embodiments, for a fixed-point environment, after obtaining the environmental perception results, the environmental perception results are uploaded to the local server and the cloud server through cloud-edge-end collaborative technology for corresponding data storage and processing.
[0123] However, when pre-set illegal and dangerous situations occur, such as falling rocks, pedestrians mistakenly entering a danger zone, or not wearing a helmet, the cloud-edge-end collaborative processing center will control the corresponding equipment according to the level of danger.
[0124] For example, if someone is not wearing a helmet, the processing results can be transmitted to the mobile edge controller for processing, and the mobile edge controller can control the alarm and other equipment in a coordinated manner; for falling rocks, the alarm can be recorded by the local or cloud server and sent to the central control center for manual collaborative processing, including on-site closure.
[0125] An embodiment of the present disclosure provides an underground coal mine environment sensing device. FIG4 is a schematic structural diagram of an underground coal mine environment sensing device provided by an embodiment of the present disclosure.
[0126] As shown in FIG4 , the coal mine underground environment perception device includes: an image acquisition module 401 , an image fusion module 402 and an environment perception module 403 .
[0127] The image acquisition module 401 is used to acquire the visual environment image and the laser radar environment image of the area to be sensed in the coal mine.
[0128] The image fusion module 402 is used to fuse the visual environment image and the lidar environment image based on the environment type of the area to be sensed to obtain a fused image.
[0129] The environment perception module 403 is used to input the fused image into the task program under different scenes to output the environment perception result of the area to be perceived in the corresponding scene.
[0130] In some embodiments, when the environment type of the area to be perceived is the target environment, the image fusion module 402 uses the lidar environment image as the baseline image and the visual environment image as the supplementary image when fusing the visual environment image and the lidar environment image; when fusing the visual environment image and the lidar environment image, the pixel value weight of the baseline image is greater than the pixel value weight of the supplementary image.
[0131] In some embodiments, the image fusion module 402, when the environment type of the area to be perceived is a non-target environment, when fusing the visual environment image and the lidar environment image, when the imaging distance of the area to be perceived is a first-level distance, the visual environment image is used as the reference image and the lidar environment image is used as the supplementary image; when the imaging distance of the area to be perceived is a second-level distance, the lidar environment image is used as the reference image; the first-level distance and the second-level distance are divided according to the distance.
[0132] In some embodiments, the image fusion module 402 further includes:
[0133] When collecting visual environment images, the camera world three-dimensional coordinate information of the visual environment image is obtained based on the relationship between the spatial coordinate system and the world coordinate system during the camera calibration process;
[0134] Based on the world 3D information of the lidar environment image obtained by the lidar, the visual environment image and the lidar environment image are fused with the camera's world 3D coordinate information as a reference.
[0135] In some embodiments, the image fusion module 402 further includes:
[0136] The first environmental feature information is extracted from the three-dimensional environmental space information corresponding to the visual environment image, and the second environmental feature information is extracted from the three-dimensional environmental space information corresponding to the lidar environment image, and the obtained first environmental feature information and second environmental feature information are fused.
[0137] In some embodiments, the environment perception module 403 further includes:
[0138] In the case of a traffic environment, target detection is used to identify moving objects and pedestrians;
[0139] When the scene is a denoising scene, denoising and defogging are performed through image enhancement;
[0140] In the case of a track line scene, edge lines are identified through field segmentation;
[0141] When the scene is a ground environment scene in a coal mine, the ground environment is extracted through semantic segmentation.
[0142] It should be noted that the above explanation of the embodiment of the coal mine underground environment perception method is also applicable to the coal mine underground environment perception device of this embodiment, and will not be repeated here.
[0143] In order to implement the above embodiment, the present disclosure further proposes another coal mine underground environment perception device, including: a processor, and a memory for storing instructions executable by the processor.
[0144] The processor is configured to execute the instructions to implement a method for sensing an underground coal mine environment:
[0145] Obtain visual environment images and lidar environment images of the area to be sensed in the coal mine;
[0146] Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image;
[0147] The fused image is input into the task program in different scenes to output the environmental perception results of the area to be perceived in the corresponding scene.
[0148] To implement the above embodiments, the present disclosure further provides a non-transitory computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a method for sensing an underground coal mine environment. The method includes:
[0149] Obtain visual environment images and lidar environment images of the area to be sensed in the coal mine;
[0150] Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image;
[0151] The fused image is input into the task program in different scenes to output the environmental perception results of the area to be perceived in the corresponding scene.
[0152] To implement the above embodiments, the present disclosure further provides a computer program product. When an instruction processor in the computer program product executes, a method for sensing an underground coal mine environment is performed. The method includes:
[0153] Obtain visual environment images and lidar environment images of the area to be sensed in the coal mine;
[0154] Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image;
[0155] The fused image is input into the task program in different scenes to output the environmental perception results of the area to be perceived in the corresponding scene.
[0156] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and features of different embodiments or examples, unless they are mutually inconsistent.
[0157] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.
[0158] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.
[0159] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.
[0160] It should be understood that various parts of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0161] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.
[0162] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. A person of ordinary skill in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.
Claims
1. A method for sensing an underground coal mine environment, comprising: Obtain visual environment images and lidar environment images of the area to be sensed in the coal mine; Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image; The fused image is input into task programs in different scenes to output the environmental perception results of the area to be sensed in the corresponding scenes.
2. The method according to claim 1, wherein: The fusing of the visual environment image and the laser radar environment image based on the environment type of the area to be sensed includes: In a case where the environment type of the area to be sensed is a target environment, when the visual environment image and the lidar environment image are fused, the lidar environment image is used as a reference image and the visual environment image is used as a supplementary image; When the visual environment image and the lidar environment image are fused, the pixel value weight of the reference image is greater than the pixel value weight of the supplementary image.
3. The method according to claim 2, wherein: The fusing of the visual environment image and the laser radar environment image based on the environment type of the area to be sensed includes: In the case where the environment type of the area to be perceived is a non-target environment, when the visual environment image and the lidar environment image are fused, when the imaging distance of the area to be perceived is a primary distance, the visual environment image is used as a reference image and the lidar environment image is used as a supplementary image; when the imaging distance of the area to be perceived is a secondary distance, the lidar environment image is used as a reference image; the primary distance and the secondary distance are divided according to distance.
4. The method according to claim 1, wherein: The fusing the visual environment image and the laser radar environment image comprises: When acquiring the visual environment image, obtaining the camera world three-dimensional coordinate information of the visual environment image based on the relationship between the spatial coordinate system and the world coordinate system in a camera calibration process; The world three-dimensional information of the laser radar environment image obtained by the laser radar is used, and the visual environment image and the laser radar environment image are fused with reference to the camera world three-dimensional coordinate information.
5. The method according to claim 1, wherein: The fusing the visual environment image and the laser radar environment image comprises: The first environment feature information is extracted from the three-dimensional environment space information corresponding to the visual environment image, and the second environment feature information is extracted from the three-dimensional environment space information corresponding to the lidar environment image, and the first environment feature information and the second environment feature information obtained are fused.
6. The method according to claim 1, wherein: The step of inputting the fused image into task programs in different scenes to output the environmental perception results of the area to be sensed in the corresponding scenes includes: In the case where the scene is a traffic environment scene, identifying moving objects and performing pedestrian recognition through target detection; When the scene is a denoising scene, denoising and defogging are performed by image enhancement; In the case where the scene is a track line scene, identifying edge lines through field segmentation; When the scene is a ground environment scene in an underground coal mine, the ground environment is extracted through semantic segmentation.
7. The method according to claim 1, further comprising: The laser radar is subjected to distortion compensation based on the angular velocity and linear velocity of the laser radar to obtain the laser radar environment image.
8. A coal mine underground environment sensing device, comprising a memory, a transceiver, and a processor: A memory for storing a computer program; a transceiver for transmitting and receiving data under the control of the processor; and a processor for reading the computer program in the memory and executing: Obtain visual environment images and lidar environment images of the area to be sensed in the coal mine; Based on the environment type of the area to be sensed, the visual environment image and the lidar environment image are fused to obtain a fused image; The fused image is input into task programs in different scenes to output the environmental perception results of the area to be sensed in the corresponding scenes.
9. A coal mine underground environment sensing device, comprising: An image acquisition module is used to acquire visual environment images and laser radar environment images of the area to be sensed in the coal mine; An image fusion module, used for fusing the visual environment image and the laser radar environment image based on the environment type of the area to be sensed to obtain a fused image; The environment perception module is used to input the fused image into task programs in different scenes to output the environment perception results of the area to be perceived in the corresponding scene.
10. The device according to claim 9, wherein: The image acquisition module is used for: In a case where the environment type of the area to be sensed is a target environment, when the visual environment image and the lidar environment image are fused, the lidar environment image is used as a reference image and the visual environment image is used as a supplementary image; When the visual environment image and the lidar environment image are fused, the pixel value weight of the reference image is greater than the pixel value weight of the supplementary image.
11. The device according to claim 10, wherein: The image fusion module is used for: In the case where the environment type of the area to be perceived is a non-target environment, when the visual environment image and the lidar environment image are fused, when the imaging distance of the area to be perceived is a primary distance, the visual environment image is used as a reference image and the lidar environment image is used as a supplementary image; when the imaging distance of the area to be perceived is a secondary distance, the lidar environment image is used as a reference image; the primary distance and the secondary distance are divided according to distance.
12. The device according to claim 9, wherein: The image fusion module is used for: When acquiring the visual environment image, obtaining the camera world three-dimensional coordinate information of the visual environment image based on the relationship between the spatial coordinate system and the world coordinate system in a camera calibration process; The three-dimensional information of the world of the laser radar environment image obtained by the laser radar is converted into the three-dimensional information of the world of the camera. The visual environment image and the lidar environment image are fused with reference to the dimensional coordinate information.
13. The device according to claim 9, wherein: The image fusion module is used for: The first environment feature information is extracted from the three-dimensional environment space information corresponding to the visual environment image, and the second environment feature information is extracted from the three-dimensional environment space information corresponding to the lidar environment image, and the first environment feature information and the second environment feature information obtained are fused.
14. The device according to claim 9, wherein: The environment perception module is used for: In the case where the scene is a traffic environment scene, identifying moving objects and performing pedestrian recognition through target detection; When the scene is a denoising scene, denoising and defogging are performed by image enhancement; In the case where the scene is a track line scene, identifying edge lines through field segmentation; When the scene is a ground environment scene in an underground coal mine, the ground environment is extracted through semantic segmentation.
15. A non-transitory computer-readable storage medium, wherein: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 7.
16. A computer program product, wherein: When a processor executes the instructions in the computer program product, the method according to any one of claims 1 to 7 is performed.
Citation Information
Patent Citations
Environment sensing system for automatic driving mining vehicle and mining vehicle
CN113002396A
Coal mine underground inspection operation robot
CN113858221A
Coal mine underground environment sensing method and device and storage medium
CN116453031A
Detection method, system, and device based on fusion of image and point cloud information, and storage medium
WO2022156175A1
Cited By
Image enhancement method and system in complex coal mine environment
CN120495604A
Autonomous operation method and system of underground coal mine roadway repairing robot
CN120592639A
Fusion sensing system and method for monitoring welding process of polar region low-temperature and low-pressure simulation cabin
CN121239811A
On-line optimization method for perception parameters of body-equipped robot oriented to cross-scene self-adaption
CN121613757A
Cross-scenario adaptive online optimization method for embodied robot perception parameters
CN121613757B
