A coal mine underground environment sensing method and device and a storage medium

Abstract

This invention proposes a method, device, and storage medium for underground environmental perception in coal mines, relating to the field of intelligent coal mining operations. The method includes: acquiring visual and lidar environmental images of the area to be perceived underground; fusing the visual and lidar environmental images based on the environment type of the area; and inputting the fused image into a multi-task program containing different scenarios to output the environmental perception results of the area under the corresponding scenarios. By using an adaptive fusion method, the visual and lidar environmental images are fully integrated, accurately reflecting the information of the area to be perceived and improving the accuracy of the underground environmental perception results in coal mines.

Description

Technical Field

[0001] This invention relates to the field of intelligent coal mining operations, and in particular to a method, device, and storage medium for sensing the underground environment in coal mines. Background Technology

[0002] The working environment in coal mines is constantly changing in terms of scenery, structure, lighting, and signals, making coal mining operations difficult and prone to accidents. With the rapid development of computer technology, communication technology, and microelectronics technology, multi-sensor data fusion technology has received widespread attention and application. Applying it to the field of coal mining operations for environmental sensing has greatly reduced the difficulty of underground coal mining operations.

[0003] Among them, the fusion technology based on RGB image and LiDAR data has attracted much attention due to its practicality and high performance in depth perception. Current work mainly explores two different fusion methods: LiDAR and monocular image fusion, and LiDAR and stereo image fusion.

[0004] However, current methods for depth estimation of images using the aforementioned fusion methods are typically based on pixel regression. This regression is inherently unreliable and ambiguous, and the information fusion is insufficient, failing to fully utilize the advantages of sensor characteristics, resulting in insufficient accuracy of the environmental perception results in coal mines. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] Therefore, the first objective of this invention is to propose a method for underground environmental perception in coal mines, so as to fully integrate RGB images with lidar data and improve the accuracy of underground environmental perception results in coal mines.

[0007] The second objective of this invention is to provide an underground environmental sensing device for coal mines.

[0008] The third objective of this invention is to propose another underground environmental sensing device for coal mines.

[0009] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0010] The fifth objective of this invention is to provide a computer program product.

[0011] To achieve the above objectives, one embodiment of the present invention provides a method comprising:

[0012] Acquire visual and lidar environmental images of the area to be sensed in an underground coal mine;

[0013] Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image;

[0014] The fused image is input into a task program for different scenarios to output the environmental perception results of the area to be perceived in the corresponding scenario.

[0015] In some possible implementations, the fusion of the visual environment image and the lidar environment image based on the environment type of the region to be perceived includes:

[0016] When the environment type of the area to be perceived is the target environment, when fusing the visual environment image and the lidar environment image, the lidar environment image is used as the reference image and the visual environment image is used as the supplementary image.

[0017] When fusing the visual environment image and the lidar environment image, the pixel value weight of the reference image is greater than the pixel value weight of the supplementary image.

[0018] In some possible implementations, the fusion of the visual environment image and the lidar environment image based on the environment type of the region to be perceived includes:

[0019] When the environment type of the area to be sensed is a non-target environment, when fusing the visual environment image and the lidar environment image, if the imaging distance of the area to be sensed is a primary distance, the visual environment image is used as the reference image and the lidar environment image is used as the supplementary image; if the imaging distance of the area to be sensed is a secondary distance, the lidar environment image is used as the reference image; the primary distance and the secondary distance are defined according to distance.

[0020] In some possible implementations, the fusion of the visual environment image and the lidar environment image includes:

[0021] When acquiring the visual environment image, the camera world three-dimensional coordinate information of the visual environment image is obtained based on the relationship between the spatial coordinate system and the world coordinate system during the camera calibration process.

[0022] Based on the world 3D information of the LiDAR environment image obtained by the LiDAR, and with the camera world 3D coordinate information as a reference, the visual environment image and the LiDAR environment image are fused.

[0023] In some possible implementations, the fusion of the visual environment image and the lidar environment image includes:

[0024] First environmental feature information is extracted from the three-dimensional environmental spatial information corresponding to the visual environment image, and second environmental feature information is extracted from the three-dimensional environmental spatial information corresponding to the lidar environment image. The obtained first environmental feature information and second environmental feature information are then fused.

[0025] In some possible implementations, the step of inputting the fused image into a task program under different scenes to output the environmental perception result of the region to be perceived in the corresponding scene includes:

[0026] In the case of a traffic environment, moving objects are identified and pedestrians are recognized through target detection.

[0027] In the case of a denoising scene, denoising and dehazing are performed through image enhancement;

[0028] In the case of a track line scenario, edge lines are identified through solid segmentation.

[0029] In the case of a ground environment scene in a coal mine, the ground environment is extracted through semantic segmentation.

[0030] In some possible implementations, the method further includes:

[0031] Distortion compensation is performed on the lidar based on its angular velocity and linear velocity to obtain the lidar environmental image.

[0032] To achieve the above objectives, a second aspect of the present invention provides a coal mine underground environmental sensing device, comprising:

[0033] The image acquisition module is used to acquire visual environment images and lidar environment images of the area to be sensed in the underground coal mine.

[0034] The image fusion module is used to fuse the visual environment image and the lidar environment image based on the environment type of the area to be perceived, to obtain a fused image;

[0035] The environment perception module is used to input the fused image into a task program under different scenarios, so as to output the environment perception result of the area to be perceived under the corresponding scenario.

[0036] To achieve the above objectives, a third aspect of the present invention provides a coal mine underground environmental sensing device, including a memory, a transceiver, and a processor:

[0037] A memory for storing computer programs; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer programs from the memory and performing the following operations:

[0038] Acquire visual and lidar environmental images of the area to be sensed in an underground coal mine;

[0039] Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image;

[0040] The fused image is input into a task program for different scenarios to output the environmental perception results of the area to be perceived in the corresponding scenario.

[0041] To achieve the above objectives, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a coal mine underground environment sensing method proposed in the first aspect of the present invention.

[0042] To achieve the above objectives, a fifth aspect of the present invention provides a computer program product that, when executed by an instruction processor, performs a method for sensing the underground environment in a coal mine, as proposed in a first aspect of the present invention.

[0043] The technical solution provided by the embodiments of the present invention can include the following beneficial effects: acquiring visual environment images and lidar environment images of the area to be sensed in an underground coal mine; fusing the visual environment images and lidar environment images based on the environment type of the area to be sensed; inputting the fused image into a multi-task program containing different scenes to output the environmental perception results of the area to be sensed in the corresponding scene. By using adaptive fusion, the visual environment images and lidar environment images are fully integrated, accurately reflecting the information of the area to be sensed and improving the accuracy of the environmental perception results in underground coal mines.

[0044] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0046] Figure 1 This is a flowchart illustrating a method for sensing the underground environment in a coal mine, as provided in an embodiment of the present invention.

[0047] Figure 2 This is a flowchart illustrating another method for sensing the underground environment in a coal mine, as provided in an embodiment of the present invention.

[0048] Figure 3 This is a flowchart illustrating another method for sensing the underground environment in a coal mine, as provided in an embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram of the structure of an underground environmental sensing device for coal mines provided in an embodiment of the present invention. Detailed Implementation

[0050] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0051] A method for sensing the underground environment in a coal mine according to an embodiment of the present invention is described below with reference to the accompanying drawings.

[0052] Figure 1 This is a flowchart illustrating a method for sensing the underground environment in a coal mine, as provided in an embodiment of the present invention.

[0053] Cameras offer advantages in environmental perception, such as rich semantics and abundant detail. However, they suffer from inaccurate location estimation within the environment, are susceptible to environmental interference (e.g., dusty or foggy environments) at close range, and are less affected by environmental interference. Radar, on the other hand, boasts advantages in environmental perception, such as accurate location estimation, long detection range, and resistance to environmental interference. However, it suffers from insufficient detail and inaccurate classification at distant locations. Therefore, leveraging the respective strengths of cameras and radar, an environmental perception fusion technique combining RGB images and LiDAR data is employed to obtain high-precision 3D environmental information. However, current fusion techniques suffer from insufficient information integration, leading to inadequate accuracy in underground coal mine environmental perception results.

[0054] To address this problem, embodiments of the present invention provide a method for sensing the underground environment in coal mines, thereby improving the accuracy of the sensing results. Figure 1 As shown, the method includes the following steps:

[0055] Step 101: Obtain visual environment images and lidar environment images of the area to be sensed in the underground coal mine.

[0056] The area to be sensed is the area in the underground coal mine where environmental sensing is required.

[0057] Optionally, an RGB image of the area to be perceived can be acquired by a camera as a visual environment image, and a lidar environment image of the area to be perceived can be acquired by a lidar.

[0058] Step 102: Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image.

[0059] Based on different environmental types of the area to be perceived, the visual environment image and the LiDAR environment image are fused accordingly to obtain a fused image. The visual environment image and the LiDAR environment image are fully fused through adaptive fusion to accurately reflect the information of the area to be perceived.

[0060] Step 103: Input the fused image into the task program under different scenarios to output the environmental perception results of the area to be perceived in the corresponding scenario.

[0061] Using the fused image as input, a multi-task program corresponding to multiple scenes is used to extract information of the area to be perceived in order to obtain the environmental perception results of the area to be perceived under different scenes.

[0062] Optionally, the environmental perception result can be the result of environmental feature identification.

[0063] In this embodiment, visual environment images and lidar environment images of the area to be sensed in the underground coal mine are acquired; the visual environment images and lidar environment images are fused based on the environment type of the area to be sensed; the fused image is input into a multi-task program containing different scenes to output the environmental perception results of the area to be sensed in the corresponding scene. By using adaptive fusion, the visual environment images and lidar environment images are fully integrated, accurately reflecting the information of the area to be sensed and improving the accuracy of the environmental perception results in the underground coal mine.

[0064] To clearly illustrate the previous embodiment, this embodiment provides another method for sensing the underground environment in coal mines. Figure 2 This is a flowchart illustrating another method for sensing the underground environment in coal mines provided in an embodiment of the present invention.

[0065] like Figure 2 As shown, the method may include the following steps:

[0066] Step 201: Obtain visual environment images and lidar environment images of the area to be sensed in the underground coal mine.

[0067] Optionally, distortion compensation is performed on the lidar based on its angular velocity and linear velocity to obtain a lidar environmental image.

[0068] Due to the inherent characteristics of cameras and radars, there will be issues with camera and radar calibration, synchronization errors, and loosening during movement. In actual use, camera images may be missing, and radar may lose point cloud data and drop frames. In harsh environments, camera images may be blurry, and radar may experience condensation.

[0069] To address the aforementioned issues, the camera's intrinsic and extrinsic parameters were calibrated, and offset compensation was performed.

[0070] It should be noted that camera intrinsic calibration refers to the transformation from camera coordinate system to image coordinate system; camera extrinsic calibration refers to the transformation from 3D spatial coordinate system to camera coordinate system.

[0071] Radar attitude adjustment involves modeling and compensating for the radar's position information and the relative positional relationship between consecutive frames.

[0072] To address synchronization errors, which occur due to time differences in a single radar scan, synchronization between the camera and the lidar is implemented.

[0073] It should be noted that, regarding the synchronization error in the hardware, resolving the synchronization error requires synchronizing the timestamps of different sensors. By calculating the radar's own angular velocity and linear velocity, the true situation of the laser is reconstructed, and distortion compensation is achieved.

[0074] Optionally, the GPS timestamp time synchronization method is as follows: First, determine whether the sensing hardware supports this method. If it does, the data packets provided by the sensor will have a global timestamp. These timestamps are based on GPS, thus using the same clock instead of the individual clocks of each sensor. However, different sensors have different data frequencies. For example, the frequency of a LiDAR is 10Hz, and the frequency of a camera is 25 / 30Hz, so there is still a delay between the data from different sensors. In this case, the nearest frame can be found by finding adjacent timestamps.

[0075] Alternatively, a hard synchronization method can be used: The LiDAR (Light Detection and Ranging) acts as the trigger for other sensors. The camera at a specific angle is only triggered when the LiDAR rotates to that angle, significantly reducing time lag issues. This time synchronization scheme can be implemented in hardware, mitigating errors caused by timestamp lookup. It can greatly reduce synchronization errors and improve data alignment.

[0076] Distortion compensation ensures that the point cloud at any given moment and the camera at the corresponding location at that moment can be triggered simultaneously, thus synchronizing the timestamps of the visual environment image and the LiDAR environment image.

[0077] For motion compensation, when an object moves, the environment changes, and the lidar scans and supplements the information synchronously.

[0078] Optionally, a fault warning may be issued when the radar loses part or all of the point cloud and / or when condensation forms on the radar.

[0079] Step 202: Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image.

[0080] It should be noted that when acquiring the fused image, the method of front fusion, back fusion, or a combination of front fusion and back fusion can be adaptively selected according to the environment type of the area to be perceived to fuse the visual environment image and the LiDAR environment image.

[0081] The pre-fusion process is as follows: when acquiring visual environment images, the camera world 3D coordinate information of the visual environment image is obtained based on the relationship between the spatial coordinate system and the world coordinate system during the camera calibration process. The world 3D information of the LiDAR environment image is obtained based on the LiDAR, and the camera world 3D coordinate information is used as a reference to fuse the visual environment image and the LiDAR environment image.

[0082] The post-fusion process involves extracting first environmental feature information from the three-dimensional environmental spatial information corresponding to the visual environment image and extracting second environmental feature information from the three-dimensional environmental spatial information corresponding to the lidar environment image, and then fusing the obtained first and second environmental feature information.

[0083] Optionally, when the environment type of the area to be perceived is the target environment, when fusing the visual environment image and the lidar environment image, the lidar environment image is used as the reference image and the visual environment image is used as the supplementary image.

[0084] It should be noted that the target environment is a foggy environment, a dusty environment, or a low-light environment, which is determined by an environment recognition model. The environment recognition model is obtained by learning the image features of different environment types.

[0085] When fusing visual environment images and LiDAR environment images, the pixel value weights of the reference image are greater than those of the supplementary image.

[0086] Furthermore, an environment recognition model is added to the camera. By learning the image features of different environment categories, the model can automatically recognize and judge the environment in the photos to determine whether the area to be perceived is the target environment.

[0087] If the pixel value weight of the reference image is greater than the pixel value weight of the supplementary image, and the sum of the pixel value weights of the reference image and the supplementary image is 1, then the pixel value weight of the reference image is greater than 0.5.

[0088] Optionally, when the environment type of the area to be sensed is a non-target environment, when fusing the visual environment image and the lidar environment image, if the imaging distance of the area to be sensed is a first-level distance, the visual environment image is used as the reference image and the lidar environment image is used as the supplementary image; if the imaging distance of the area to be sensed is a second-level distance, the lidar environment image is used as the reference image and the visual environment image is used as the supplementary image; the first-level distance and the second-level distance are divided according to distance.

[0089] It should be noted that the distinction between Level 1 and Level 2 range is determined by hardware parameters. Generally, cameras in coal mines are only used for close-range environmental detail perception, typically within 10 meters; while radar and lidar, such as Radar, use millimeter waves, usually 4-12mm, with a much longer effective working distance. LiDAR uses laser wavelengths typically between 900-1500nm, allowing for even finer and more precise vision.

[0090] Based on the above hardware parameters, as one possible implementation, an imaging distance of less than 10 meters is considered a first-level distance, otherwise it is considered a second-level distance.

[0091] It is understandable that regardless of whether the fusion of visual environment images and LiDAR environment images is performed using a pre-fusion or post-fusion method, the reference can be determined based on distance.

[0092] By fusing images under different environmental types, using an image that better reflects the precise information of the area to be perceived as the reference image, and then using supplementary images as an aid to improve the reference image, an accurate fused image of the area to be perceived can be obtained under different environmental types. This results in a denser and more accurate disparity map, improving the accuracy of environmental perception and providing an accurate depth estimate for the entire environmental perception method. In turn, this can provide reliable image information and technical support for applications such as coal mine road environmental perception, visual measurement systems, unmanned mining truck navigation systems, and mine search and rescue robots.

[0093] Step 203: Input the fused image into the task program under different scenarios to output the environmental perception results of the area to be perceived in the corresponding scenario.

[0094] Step 203 can be found in the relevant descriptions of the corresponding steps in the foregoing embodiments, and will not be repeated in this embodiment.

[0095] This embodiment provides another method for underground environmental sensing in coal mines. Figure 3 This is a flowchart illustrating another method for sensing the underground environment in coal mines provided in an embodiment of the present invention.

[0096] like Figure 3 As shown, the method may include the following steps:

[0097] Step 301: Obtain visual environment images and lidar environment images of the area to be sensed in the underground coal mine.

[0098] Step 302: Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image.

[0099] Steps 301 and 302 can be found in the relevant descriptions of the corresponding steps in the foregoing embodiments, and will not be repeated in this embodiment.

[0100] Step 303: Input the fused image into the task program under different scenarios to output the environmental perception result of the area to be perceived in the corresponding scenario.

[0101] It should be noted that multi-tasking programs in different scenarios can be implemented in parallel or sequentially, depending on the actual computing power and the optimal path selection. That is, as one possible implementation, the fused image can be input into different scenarios for parallel function implementation; as another possible implementation, the corresponding scenario can be identified first, the corresponding program can be selected, and then the fused image can be input into the program of the corresponding scenario to obtain the output result.

[0102] As one possible implementation, in traffic environments, moving objects can be identified and pedestrians can be recognized through object detection.

[0103] Traffic environment scenarios refer to scenarios that include traffic environments. Traffic environments include many moving targets, such as moving people, vehicles, and objects, and are constantly changing. Object detection can track the movement trajectory of objects. Therefore, for traffic environments with many moving targets, object detection is used to identify moving objects and pedestrians.

[0104] As one possible implementation, in the case of an obstacle-prone scene, obstacles can be identified through object detection.

[0105] An obstacle scene refers to a scene that includes static obstacles, such as railings or piles of rocks. Object detection can locate different objects and identify the obstacles that need to be detected in the obstacle scene.

[0106] As one possible approach, in the case of a denoising scenario, denoising and dehazing can be achieved through image enhancement.

[0107] Noise reduction scenarios refer to scenarios that include environmental types that require noise reduction, such as foggy environments, dusty environments, or low-light environments, which are also known as target environments.

[0108] As one possible implementation, in the case of a track line scene, edge lines can be identified through ground segmentation.

[0109] The track line scenario refers to a scenario that includes track lines. In coal mines, underground rail transport vehicles are used to transport coal, explosives, etc., and some road surfaces may contain track lines.

[0110] As one possible approach, in the case of a ground environment scene in a coal mine, the ground environment can be extracted through semantic segmentation.

[0111] Ground environment scene refers to a scene that does not contain other obstacles, moving objects, etc., and only exists on the underground ground.

[0112] Step 304: Apply the environmental perception results of the area to be perceived under different scenarios to different devices to realize the control and application of underground equipment in coal mines.

[0113] Since different devices use environmental perception results for different purposes, the weight of the environmental perception results in each scenario is first determined based on the device to which the environmental perception results will be applied.

[0114] As one possible approach, the environmental perception results need to be applied to continuously moving devices, such as mobile robots. The weights of the environmental perception results are ranked as follows: ground environment scene > track line scene > obstacle scene > traffic environment scene > noise reduction scene.

[0115] As another possible implementation, for devices used for fixed-point identification, the weights of their environmental perception results are ranked as follows: denoised scene > ground environment scene > traffic environment scene > track line scene > obstacle scene.

[0116] After obtaining the weights of environmental perception results under different scenarios, decisions are made by controlling downhole equipment.

[0117] As one possible implementation, for mobile robots, after obtaining the environmental perception results, the environmental perception results are transmitted to their own processor. The processor sends the commands to the controller, mainly the motor controller, according to the expert system or its own preset processing commands. By controlling the corresponding speed, it can realize operations such as turning, stopping, accelerating, and emergency stopping.

[0118] As another possible implementation, for fixed-location environments, after obtaining the environmental perception results, the environmental perception results are uploaded to the local terminal server and the cloud server through cloud-edge-device collaboration technology for corresponding data storage and processing.

[0119] However, when pre-set violations or dangerous situations occur, such as falling rocks, pedestrians accidentally entering dangerous areas, or not wearing safety helmets, the cloud-edge-device collaborative processing center will control the corresponding equipment according to the level of danger.

[0120] For example, if someone is not wearing a safety helmet, the result can be transmitted to a mobile edge controller for processing, and the mobile edge controller can then control devices such as alarms in conjunction with the action. For incidents such as falling rocks, alarms can be recorded by a local or cloud server and sent to the central control center for manual intervention, including site closure.

[0121] This embodiment provides an underground environmental sensing device for coal mines. Figure 4 This is a schematic diagram of the structure of an underground environmental sensing device for coal mines provided in an embodiment of the present invention.

[0122] like Figure 4 As shown, the underground environmental sensing device in a coal mine includes: an image acquisition module 401, an image fusion module 402, and an environmental sensing module 403.

[0123] The image acquisition module 401 is used to acquire visual environment images and lidar environment images of the area to be sensed in the underground coal mine.

[0124] The image fusion module 402 is used to fuse visual environment images and lidar environment images based on the environment type of the area to be perceived, so as to obtain a fused image.

[0125] The environment perception module 403 is used to input the fused image into the task program under different scenes, so as to output the environment perception result of the area to be perceived under the corresponding scene.

[0126] As one possible implementation, when the environment type of the area to be perceived is the target environment, the image fusion module 402 uses the LiDAR environment image as the reference image and the visual environment image as the supplementary image when fusing the visual environment image and the LiDAR environment image; when fusing the visual environment image and the LiDAR environment image, the pixel value weight of the reference image is greater than the pixel value weight of the supplementary image.

[0127] As one possible implementation, the image fusion module 402, when the environment type of the area to be perceived is a non-target environment, when fusing the visual environment image and the lidar environment image, uses the visual environment image as the reference image and the lidar environment image as the supplementary image when the imaging distance of the area to be perceived is a first-level distance; when the imaging distance of the area to be perceived is a second-level distance, the lidar environment image is used as the reference image; the first-level distance and the second-level distance are divided according to distance.

[0128] As one possible implementation, the image fusion module 402 also includes:

[0129] When acquiring visual environment images, the camera's three-dimensional world coordinate information of the visual environment images is obtained based on the relationship between the spatial coordinate system and the world coordinate system during the camera calibration process.

[0130] Based on the world 3D information of the LiDAR environment image obtained by the LiDAR, and with the camera world 3D coordinate information as a reference, the visual environment image and the LiDAR environment image are fused.

[0131] As one possible implementation, the image fusion module 402 also includes:

[0132] First environmental feature information is extracted from the three-dimensional environmental spatial information corresponding to the visual environment image, and second environmental feature information is extracted from the three-dimensional environmental spatial information corresponding to the lidar environment image. The obtained first environmental feature information and second environmental feature information are then fused.

[0133] As one possible implementation, the environment perception module 403 also includes:

[0134] In traffic environments, moving objects are identified and pedestrians are recognized through target detection.

[0135] In the case of a denoising scene, denoising and dehazing are performed through image enhancement.

[0136] In the case of a track line scene, edge lines are identified through solid segmentation.

[0137] In scenarios where the scene is an underground coal mine environment, the ground environment is extracted through semantic segmentation.

[0138] It should be noted that the foregoing explanation of the embodiment of the underground environmental sensing method in coal mines also applies to the underground environmental sensing device in this embodiment, and will not be repeated here.

[0139] To implement the above embodiments, the present invention also proposes another underground environmental sensing device for coal mines, comprising: a processor, and a memory for storing executable instructions of the processor.

[0140] The processor is configured to execute the instructions to implement a method for environmental perception in underground coal mines.

[0141] Acquire visual and lidar environmental images of the area to be sensed in an underground coal mine;

[0142] Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image;

[0143] The fused images are input into task programs in different scenarios to output the environmental perception results of the area to be perceived in the corresponding scenario.

[0144] To implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to execute a method for sensing the underground environment of a coal mine, the method comprising:

[0145] Acquire visual and lidar environmental images of the area to be sensed in an underground coal mine;

[0146] Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image;

[0147] The fused images are input into task programs in different scenarios to output the environmental perception results of the area to be perceived in the corresponding scenario.

[0148] To implement the above embodiments, the present invention also proposes a computer program product, which, when executed by an instruction processor, performs a method for sensing the underground environment of a coal mine, the method comprising:

[0149] Acquire visual and lidar environmental images of the area to be sensed in an underground coal mine;

[0150] Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image;

[0151] The fused images are input into task programs in different scenarios to output the environmental perception results of the area to be perceived in the corresponding scenario.

[0152] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0153] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0154] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0156] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0157] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0158] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for sensing the underground environment in coal mines, characterized in that, Includes the following steps: Acquire visual and lidar environmental images of the area to be sensed in an underground coal mine; Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image; The fused image is input into a task program in different scenarios to output the environmental perception result of the area to be perceived in the corresponding scenario. The fusion of the visual environment image and the lidar environment image based on the environment type of the region to be perceived includes: When the environment type of the area to be perceived is the target environment, when fusing the visual environment image and the lidar environment image, the lidar environment image is used as the reference image and the visual environment image is used as the supplementary image. When fusing the visual environment image and the lidar environment image, the pixel value weight of the reference image is greater than the pixel value weight of the supplementary image; as well as When the environment type of the area to be sensed is a non-target environment, when fusing the visual environment image and the lidar environment image, if the imaging distance of the area to be sensed is a primary distance, the visual environment image is used as the reference image and the lidar environment image is used as the supplementary image; if the imaging distance of the area to be sensed is a secondary distance, the lidar environment image is used as the reference image; the primary distance and the secondary distance are defined according to distance.

2. The method according to claim 1, characterized in that, The fusion of the visual environment image and the lidar environment image includes: When acquiring the visual environment image, the camera world three-dimensional coordinate information of the visual environment image is obtained based on the relationship between the spatial coordinate system and the world coordinate system during the camera calibration process. Based on the world 3D information of the LiDAR environment image obtained by the LiDAR, and with the camera world 3D coordinate information as a reference, the visual environment image and the LiDAR environment image are fused.

3. The method according to claim 1, characterized in that, The fusion of the visual environment image and the lidar environment image includes: First environmental feature information is extracted from the three-dimensional environmental spatial information corresponding to the visual environment image, and second environmental feature information is extracted from the three-dimensional environmental spatial information corresponding to the lidar environment image. The obtained first environmental feature information and second environmental feature information are then fused.

4. The method according to claim 1, characterized in that, The step of inputting the fused image into a task program for different scenarios to output the environmental perception result of the region to be perceived in the corresponding scenario includes: In the case of a traffic environment, moving objects are identified and pedestrians are recognized through target detection. In the case of a denoising scene, denoising and dehazing are performed through image enhancement; In the case of a track line scenario, edge lines are identified through solid segmentation. In the case of a ground environment scene in a coal mine, the ground environment is extracted through semantic segmentation.

5. The method according to claim 1, characterized in that, The method further includes: Distortion compensation is performed on the lidar based on its angular velocity and linear velocity to obtain the lidar environmental image.

6. A coal mine underground environmental sensing device, characterized in that, The device is used to implement the coal mine underground environment sensing method as described in claim 1. The device includes a memory, a transceiver, and a processor. A memory for storing computer programs; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer programs from the memory and performing the following operations: Acquire visual and lidar environmental images of the area to be sensed in an underground coal mine; Based on the environment type of the area to be perceived, the visual environment image and the lidar environment image are fused to obtain a fused image; The fused image is input into a task program for different scenarios to output the environmental perception results of the area to be perceived in the corresponding scenario.

7. A coal mine underground environmental sensing device, characterized in that, The device is used to implement the coal mine underground environment sensing method as described in claim 1, and the device includes: The image acquisition module is used to acquire visual environment images and lidar environment images of the area to be sensed in the underground coal mine. The image fusion module is used to fuse the visual environment image and the lidar environment image based on the environment type of the area to be perceived, to obtain a fused image; The environment perception module is used to input the fused image into a task program under different scenarios, so as to output the environment perception result of the area to be perceived under the corresponding scenario.

8. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 5.

Images