Coal mine tunneling visual auxiliary tunneling operation method and equipment based on multi-sensor fusion

Abstract

The invention provides a coal mine tunneling visual auxiliary tunneling operation method and equipment based on multi-sensor fusion, and the method comprises the steps: collecting roadway dust concentration data, roadway section image data and distance data between a cutting mechanism and a roadway wall in a working process of the cutting mechanism of a tunneling machine; performing data preprocessing on the roadway section image data; calculating pose information data of a cutting mechanism relative to the roadway section based on the preprocessed roadway section image data; performing data fusion on the distance data and the pose information data, and adjusting a data fusion result based on the roadway dust concentration data; and a data fusion result is displayed on a display mechanism in a driving cab of the heading machine in real time. According to the invention, the visual interface can be used for assisting the heading machine to complete cutting operation, and the forming effect of the cut section and the heading efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of mining control technology, and more specifically, to a method, apparatus, equipment, and storage medium for visually assisted mining operations based on multi-sensor fusion. Background Technology

[0002] Currently, coal mine roadways in my country mainly rely on tunneling machines / roadheader-anchor machines for excavation. During actual excavation, a large amount of dust and smoke is generated, affecting workers' health and production efficiency. To avoid this problem, ventilation fans are typically used at the working face to introduce fresh air into ventilation ducts to reduce dust concentration. Simultaneously, high-pressure spray systems are built into the cutting devices to cool the cutting teeth, eliminate sparks, and degrade the dust generated during cutting. Furthermore, specialized dry / wet dust collection equipment is used to remove dust from the roadways.

[0003] However, existing dust control methods are insufficient to completely and effectively solve the dust problem. Furthermore, due to the development of intelligent coal mine construction, tunneling machines are becoming heavier and more integrated, with most equipped with temporary support devices, which obstruct the view of the cutting head from the machine body. In actual tunneling operations, remote-controlled cutting is rarely used; operation still relies primarily on the tunneling machine operator. The operator cannot clearly observe the tunneling machine's progress or the cutting and shaping of the face roadway cross-section, requiring manual (co-operator) guidance or reliance on their own experience. This easily leads to under-excavation or over-excavation of the roadway. Additionally, the increased number of personnel operating the tunneling machine increases the incidence of production accidents. Summary of the Invention

[0004] In order to solve at least one of the above-mentioned technical problems in the prior art, the present invention provides a method, device, equipment and storage medium for visually assisted tunneling operations in coal mines based on multi-sensor fusion, which aims to assist the tunneling machine in completing the cutting operation through a visual interface, thereby improving the cutting section forming effect and tunneling efficiency.

[0005] A visualization-assisted tunneling operation method based on multi-sensor fusion in coal mine tunneling includes: During the operation of the tunneling machine's cutting mechanism, data on roadway dust concentration, roadway cross-sectional image data, and distance data between the cutting mechanism and the roadway wall are collected, and the roadway cross-sectional image data is preprocessed. The pose information of the cutting mechanism relative to the roadway cross-section is calculated based on the preprocessed roadway cross-section image data. Distance data and pose information data are fused together, and the data fusion result is adjusted based on the dust concentration data in the tunnel. The data fusion results are displayed in real time on the display mechanism inside the tunneling machine's cab.

[0006] During the operation of the tunneling machine's cutting mechanism, data on roadway dust concentration, roadway cross-sectional images, and the distance between the cutting mechanism and the roadway wall are collected, including: A dust concentration sensor is installed on the tunneling machine to collect dust concentration data in the tunnel. An explosion-proof ultra-wide-angle infrared camera is installed at a designated position on the tunneling machine near the cutting mechanism to collect cross-sectional image data of the tunnel. Ultrasonic sensors are evenly arranged at preset intervals around the cutting arm of the tunneling machine to collect distance data between the cutting mechanism and the tunnel wall.

[0007] The preprocessing of the tunnel cross-section image data includes: An adaptive histogram equalization method is used to enhance the image contrast of the tunnel cross-section image data; A nonlocal means denoising algorithm is used to denoise the tunnel cross-section image data; Based on Retinex theory, color constancy processing is performed on tunnel cross-section image data; among which... The formula for nonlocal means denoising is expressed as: in, p For the target pixel, I(p) The denoised pixel value represents the pixel. p The new intensity value after nonlocal mean denoising; represents the (x,y) coordinates in the tunnel cross-section image; q Represents the pixels within the search window; w(p,q) The weight coefficient represents the pixel value. p and q The similarity weight between them takes a value in the range of [0,1]. C(p) This is the normalization coefficient.

[0008] The calculation of the pose information of the cutting mechanism relative to the roadway cross-section based on the preprocessed roadway cross-section image data includes: The ORB feature detection algorithm is used to extract the tunnel cross-section feature points from the tunnel cross-section image data; The RANSAC algorithm is used to match the features of the tunnel cross-section features and eliminate false matching points. The PnP algorithm is used to solve for the feature points of the roadway cross section after removing mismatched points, and the pose information data is obtained.

[0009] Specifically, the PnP algorithm is used to solve for the feature points of the roadway cross-section after removing mismatched points, thereby obtaining the pose information data, expressed by the formula: in, Pi The world coordinates of the pre-set reflective markers on the tunnel cross-section; u i The pixel coordinates of the marker points detected in the image captured by the infrared camera; K The intrinsic parameters of the infrared camera obtained through calibration; R,t This refers to the pose information data of the tunneling machine cutter head relative to the roadway cross-section that needs to be solved.

[0010] This includes fusing distance data and pose information data, including: A Kalman filter model is established to fuse distance and pose information data; among which, The state equation of the Kalman filter model is expressed as: in, x k A represents the state vector at time k; A is the state transition matrix, indicating how the system state transitions from one time step to the next; B is the control input matrix, which converts external control inputs into state changes. u k−1 This is the control input vector at time k-1; w k−1 The process noise at time k-1 follows a Gaussian distribution with zero mean and covariance Q; The observation equation of the Kalman filter model is expressed as follows: in, z k Let H be the observation vector at time k; H is the measurement matrix used to map the state vector to the observation space. v k Let be the observation noise at time k, which follows a Gaussian distribution with zero mean and covariance R.

[0011] Among them, the data fusion results adjusted based on tunnel dust concentration data include: When the dust concentration data in the roadway is less than the first dust concentration threshold, the weight of the pose information data is adjusted to the first pose information data weight threshold, and the weight of the distance data is adjusted to the first distance data weight threshold. When the dust concentration data in the roadway is greater than or equal to the first dust concentration threshold and less than the second first dust concentration threshold, the weight of the pose information data is adjusted to the second pose information data weight threshold, and the weight of the distance data is adjusted to the second distance data weight threshold. When the dust concentration data in the roadway is greater than or equal to the second dust concentration threshold, the weight of the pose information data is adjusted to the third pose information data weight threshold, and the weight of the distance data is adjusted to the third distance data weight threshold. Among them, the first pose information data weight threshold > the second pose information data weight threshold > the third pose information data weight threshold, the first distance data weight threshold < the second distance data weight threshold < the third distance data weight threshold, and the first pose information data weight threshold + the first distance data weight threshold = the second pose information data weight threshold + the second distance data weight threshold = the third pose information data weight threshold + the third distance data weight threshold = 1.

[0012] This invention discloses a visualization-assisted tunneling operation device for coal mine tunneling based on multi-sensor fusion, comprising: The data acquisition module is used to collect roadway dust concentration data, roadway cross-sectional image data, and distance data between the cutting mechanism and the roadway wall during the operation of the tunneling machine's cutting mechanism, and to perform data preprocessing on the roadway cross-sectional image data. The calculation module is used to calculate the pose information data of the cutting mechanism relative to the roadway cross-section based on the preprocessed roadway cross-section image data. The fusion module is used to fuse distance data and pose information data, and adjust the data fusion result based on the roadway dust concentration data; The display module is used to display the data fusion results in real time on the display mechanism inside the tunneling machine's cab.

[0013] The present invention discloses a computer device, including an input / output unit, a memory, and a processor. The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps in the method described in the foregoing embodiments.

[0014] The present invention discloses a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the method described in the foregoing embodiments.

[0015] Unlike existing technologies, this invention provides a visualization-assisted tunneling operation method for coal mine tunneling based on multi-sensor fusion. During the operation of the tunneling machine's cutting mechanism, it collects roadway dust concentration data, roadway cross-sectional image data, and distance data between the cutting mechanism and the roadway wall. The roadway cross-sectional image data undergoes preprocessing. Based on the preprocessed roadway cross-sectional image data, the pose information of the cutting mechanism relative to the roadway cross-section is calculated. The distance data and pose information are fused, and the fusion result is adjusted based on the roadway dust concentration data. The fusion result is displayed in real-time on a display mechanism within the tunneling machine's cab. This invention enables the tunneling machine to complete cutting operations through a visual interface, improving the cutting section forming effect and tunneling efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a visualization-assisted tunneling operation method for coal mine tunneling based on multi-sensor fusion, provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of a coal mine tunneling visualization-assisted tunneling operation device based on multi-sensor fusion provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of a non-transitory computer-readable storage medium storing computer instructions provided by the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please refer to Figure 1 This invention discloses a visualization-assisted tunneling operation method for coal mine tunneling based on multi-sensor fusion, comprising: S110: During the operation of the tunneling machine's cutting mechanism, collect data on roadway dust concentration, roadway cross-sectional image data, and distance data between the cutting mechanism and the roadway wall, and perform data preprocessing on the roadway cross-sectional image data.

[0022] In this embodiment of the invention, sensors are deployed at designated locations on the tunnel boring machine (TBM) to collect operating data. Specifically, a dust concentration sensor is installed on the TBM to sense and collect real-time dust concentration data in the tunnel environment at the TBM's working location; multiple explosion-proof ultra-wide-angle infrared cameras are installed on the TBM near the cutting head to collect cross-sectional image data of the tunnel at the TBM's excavation location; and multiple ultrasonic sensors are installed at preset intervals around the cutting arm of the TBM to collect distance data between the cutting mechanism and the tunnel wall.

[0023] After data acquisition, the process also includes data preprocessing of the acquired tunnel cross-section image data, including: An adaptive histogram equalization method is employed to enhance the image contrast of tunnel cross-section image data. The core of the adaptive histogram equalization method is to redistribute pixel brightness values ​​to balance the histogram brightness distribution. This method can enhance global contrast by expanding the commonly used brightness range.

[0024] A nonlocal means denoising algorithm is used to denoise the tunnel cross-section image data. The nonlocal means denoising algorithm finds similar regions in the tunnel cross-section image on a patch-by-pattern basis, and then averages the found similar regions, which can effectively remove Gaussian noise in the image.

[0025] The formula for nonlocal means denoising is expressed as: in, p For the target pixel, I(p) The denoised pixel value represents the pixel. p The new intensity value after nonlocal mean denoising; represents the (x,y) coordinates in the tunnel cross-section image; q Represents the pixels within the search window; w(p,q) The weight coefficient represents the pixel value. p and q The similarity weight between them takes a value in the range of [0,1]. C(p) This is the normalization coefficient.

[0026] Finally, based on Retinex theory, color constancy processing was performed on the tunnel cross-section image data. Retinex theory decomposes the image into three components: off-field components, reflection edges, and noise, and extracts the reflection edge structure using Gaussian functions and Laplacian operators. During the calculation process, the multi-scale Retinex algorithm (MSR) and its improved version MSRCR-PT preprocessing method are used to improve image contrast and mitigate the effects of noise.

[0027] S120: Calculate the pose information of the cutting mechanism relative to the roadway cross-section based on the preprocessed roadway cross-section image data.

[0028] After preprocessing the tunnel cross-section image data, the pose information of the cutting mechanism relative to the tunnel cross-section is calculated based on the preprocessed tunnel cross-section image data.

[0029] Specifically, in the step of calculating the pose information of the cutting mechanism relative to the roadway cross-section based on the roadway cross-section image data, the ORB feature detection algorithm is first used to extract the roadway cross-section feature points in the roadway cross-section image data.

[0030] ORB stands for Oriented Brief, and it uses the FAST (features from accelerated seqmenttest) algorithm to detect feature points. ORB consists of two parts: feature point extraction and feature point description. Feature extraction is based on the grayscale values ​​of the image around a feature point. It detects the pixel values ​​around a candidate feature point. If a candidate point has a sufficiently large number of pixels in its neighborhood with a significantly different grayscale value, then the candidate point is considered a feature point. Feature point description is an improvement on the BRIEF feature description algorithm.

[0031] After extracting tunnel cross-section feature points from the tunnel cross-section image data using the ORB feature detection algorithm, the RANSAC algorithm is used to perform feature matching on these feature points and eliminate mismatched points. RANSAC, short for Random Sample Consensus Algorithm, is an efficient method for estimating mathematical model parameters from a sample set containing outliers. The core algorithm involves randomly sampling a subset of samples to construct an initial model, calculating the remainder error to filter for a consensus set, and selecting the largest consensus set through multiple samplings to determine the final model. Its optimization strategy can involve improving the sampling method or expanding the interior point set for parameter reestimation. This invention uses this method to identify mismatched points among the extracted tunnel cross-section feature points.

[0032] The PnP algorithm is used to solve for the feature points of the tunnel cross-section after removing mismatched points, thus obtaining pose information data. PnP is a method for solving the correspondence between 3D and 2D points.

[0033] The PnP algorithm formula is expressed as follows: in, P i The world coordinates of the pre-set reflective markers on the tunnel cross-section; u i The pixel coordinates of the marker points detected in the image captured by the infrared camera; KThe intrinsic parameters of the infrared camera obtained through calibration; R,t This refers to the pose information data of the tunneling machine cutter head relative to the roadway cross-section that needs to be solved.

[0034] S130: The distance data and pose information data are fused together, and the data fusion result is adjusted based on the dust concentration data in the roadway.

[0035] In an embodiment of the present invention, the data fusion of distance data and pose information data is achieved by establishing a Kalman filter model.

[0036] The state equation of the Kalman filter model is expressed as: in, x k A represents the state vector at time k; A is the state transition matrix, indicating how the system state transitions from one time step to the next; B is the control input matrix, which converts external control inputs into state changes. u k−1 This is the control input vector at time k-1; w k−1 The process noise at time k-1 follows a Gaussian distribution with zero mean and covariance Q; The observation equation of the Kalman filter model is expressed as follows: in, z k Let H be the observation vector at time k; H is the measurement matrix used to map the state vector to the observation space. v k Let be the observation noise at time k, which follows a Gaussian distribution with zero mean and covariance R.

[0037] After fusing distance and pose information data using a Kalman filter model, the data fusion results are further adjusted based on the roadway dust concentration data.

[0038] Specifically, when the dust concentration data in the tunnel is less than the first dust concentration threshold, the weight of the pose information data is adjusted to the first pose information data weight threshold, and the weight of the distance data is adjusted to the first distance data weight threshold. When the dust concentration data in the roadway is greater than or equal to the first dust concentration threshold and less than the second first dust concentration threshold, the weight of the pose information data is adjusted to the second pose information data weight threshold, and the weight of the distance data is adjusted to the second distance data weight threshold. When the dust concentration data in the roadway is greater than or equal to the second dust concentration threshold, the weight of the pose information data is adjusted to the third pose information data weight threshold, and the weight of the distance data is adjusted to the third distance data weight threshold. Among them, the first pose information data weight threshold > the second pose information data weight threshold > the third pose information data weight threshold, the first distance data weight threshold < the second distance data weight threshold < the third distance data weight threshold, and the first pose information data weight threshold + the first distance data weight threshold = the second pose information data weight threshold + the second distance data weight threshold = the third pose information data weight threshold + the third distance data weight threshold = 1.

[0039] In this embodiment, the first dust concentration threshold, the second dust concentration threshold, the first pose information data weight threshold, the second pose information data weight threshold, the third pose information data weight threshold, the first distance data weight threshold, the second distance data weight threshold, and the third distance data weight threshold are all derived from empirical data. Specifically, the first dust concentration threshold is set to 100 mg / m³, the second dust concentration threshold is set to 150 mg / m³, the first pose information data weight threshold is set to 0.7, the second pose information data weight threshold is set to 0.4, and the third pose information data weight threshold is set to 0.3; correspondingly, the first distance data weight threshold is set to 0.3, the second distance data weight threshold is set to 0.6, and the third distance data weight threshold is set to 0.7.

[0040] The data fusion weight adjustment strategy is as follows: when the dust concentration is <100mg / m³, the weight of visual data is adjusted to 0.7 and the weight of ultrasonic data is adjusted to 0.3; when the dust concentration is ≥100mg / m³, the weight of visual data is adjusted to 0.4 and the weight of ultrasonic data is adjusted to 0.6; when the dust concentration is >150mg / m³, the weight of visual data is adjusted to 0.3 and the weight of ultrasonic data is adjusted to 0.7.

[0041] S140: The data fusion results are displayed in real time on the display mechanism in the tunneling machine's cab.

[0042] After the above steps are completed, the following information will be displayed on the display mechanism in the tunneling machine's cab: ① Real-time infrared images of the tunnel cross-section; ② Distance between the cutting head and the cutting head (display accuracy 0.01m); ③ Deviation analysis between tunnel outline and design cross-section; ④ Warning signs for over-excavation / under-excavation areas (red / yellow color markings); Based on the displayed information, the tunneling machine operator can clearly observe the distance between the cutting head and the face and both sides, thus enabling them to carry out tunneling operations. It should be noted that the display module on the operator's console also needs dust protection to ensure that it does not affect the operator's observation.

[0043] Based on the method of this invention, under normal operating conditions, the real-time data collected by the sensor can be directly displayed on the display mechanism. When the dust concentration sensor detects that the dust concentration in the roadway is ≥100 mg / m³, the roadway cross-section image data is preprocessed using the method of this invention, and then fused with the distance data between the cutting mechanism and the roadway wall. The fusion result is adjusted using the roadway dust concentration data, and finally, the fusion result is displayed on the display mechanism.

[0044] like Figure 2 As shown, this invention proposes a visualization-assisted tunneling operation device for coal mine tunneling based on multi-sensor fusion, comprising: The data acquisition module 210 is used to collect roadway dust concentration data, roadway cross-sectional image data, and distance data between the cutting mechanism and the roadway wall during the operation of the tunneling machine cutting mechanism, and to perform data preprocessing on the roadway cross-sectional image data. Calculation module 220 is used to calculate the pose information data of the cutting mechanism relative to the roadway cross-section based on the preprocessed roadway cross-section image data; The fusion module 230 is used to fuse distance data and pose information data, and adjust the data fusion result based on the roadway dust concentration data; Display module 240 is used to display the data fusion results in real time on the display mechanism in the tunneling machine cab.

[0045] To implement the embodiments, the present invention also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method described above.

[0046] like Figure 3 As shown, the non-transitory computer-readable storage medium includes a memory 810 for instructions and an interface 830, the instructions of which can be executed by a processor 820 to complete the method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0047] To implement the embodiments, the present invention also proposes a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the embodiments of the present invention.

[0048] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0049] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0050] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0051] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0052] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the described embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0053] Those skilled in the art will understand that all or part of the steps of the method described in the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0054] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0055] The storage medium mentioned may be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the embodiments within the scope of the present invention.

Claims

1. A visualization-assisted tunneling operation method for coal mine tunneling based on multi-sensor fusion, characterized in that, include: During the operation of the tunneling machine's cutting mechanism, data on roadway dust concentration, roadway cross-sectional image data, and distance data between the cutting mechanism and the roadway wall are collected, and the roadway cross-sectional image data is preprocessed. The pose information of the cutting mechanism relative to the roadway cross-section is calculated based on the preprocessed roadway cross-section image data. The distance data and the pose information data are fused together, and the fusion result is adjusted based on the roadway dust concentration data. The data fusion results are displayed in real time on the display mechanism inside the tunneling machine's cab.

2. The method for visually assisted tunneling operations in coal mines based on multi-sensor fusion according to claim 1, characterized in that, During the operation of the tunneling machine's cutting mechanism, data on roadway dust concentration, roadway cross-sectional images, and the distance between the cutting mechanism and the roadway wall are collected, including: A dust concentration sensor is installed on the tunneling machine to collect dust concentration data in the tunnel. An explosion-proof ultra-wide-angle infrared camera is installed at a designated position on the tunneling machine near the cutting mechanism to collect cross-sectional image data of the tunnel. Ultrasonic sensors are evenly arranged at preset intervals around the cutting arm of the tunneling machine to collect distance data between the cutting mechanism and the tunnel wall.

3. The method for visually assisted tunneling operations in coal mines based on multi-sensor fusion according to claim 2, characterized in that, Data preprocessing of the tunnel cross-section image data includes: An adaptive histogram equalization method is used to enhance the image contrast of the roadway cross-section image data; The nonlocal mean denoising algorithm is used to denoise the tunnel cross-section image data; Based on Retinex theory, color constancy processing is performed on the tunnel cross-section image data; wherein... The nonlocal mean denoising algorithm is expressed as follows: in, p For the target pixel, I(p) The denoised pixel value represents the pixel. p The new intensity value after nonlocal mean denoising represents the (x,y) coordinates in the tunnel cross-section image. q Represents the pixels within the search window; w(p,q) The weight coefficient represents the pixel value. p and q The similarity weight between them takes a value in the range of [0,1]. C(p) This is the normalization coefficient.

4. The method for visually assisted tunneling operations in coal mines based on multi-sensor fusion according to claim 1, characterized in that, Based on the preprocessed tunnel cross-section image data, the pose information data of the cutting mechanism relative to the tunnel cross-section is calculated, including: The ORB feature detection algorithm is used to extract the tunnel cross-section feature points from the tunnel cross-section image data; The RANSAC algorithm is used to perform feature matching on the feature points of the tunnel cross section, and mismatched points are eliminated. The pose information data is obtained by solving the feature points of the roadway cross section after removing mismatched points using the PnP algorithm.

5. The method for visually assisted tunneling operations in coal mines based on multi-sensor fusion according to claim 4, characterized in that, The pose information data is obtained by solving the feature points of the roadway cross-section after removing mismatched points using the PnP algorithm, and the formula is expressed as: in, P i Indicates the world coordinates of the preset reflective markers on the tunnel cross-section; u i The pixel coordinates of the marker points detected in the image captured by the infrared camera; K The intrinsic parameters of the infrared camera obtained through calibration; R,t This refers to the pose information data of the tunneling machine cutter head relative to the roadway cross-section that needs to be solved.

6. The method for visually assisted tunneling operations in coal mines based on multi-sensor fusion according to claim 1, characterized in that, The data fusion of the distance data and the pose information data includes: A Kalman filter model is established to fuse the distance data and the pose information data; wherein, The state equation of the Kalman filter model is expressed as follows: in, x k A represents the state vector at time k; A is the state transition matrix, indicating how the system state transitions from one time step to the next; B is the control input matrix, which converts external control inputs into state changes. u k−1 This is the control input vector at time k-1; w k−1 The process noise at time k-1 follows a Gaussian distribution with zero mean and covariance Q; The observation equation of the Kalman filter model is expressed as follows: in, z k Let H be the observation vector at time k; H is the measurement matrix used to map the state vector to the observation space. v k Let be the observation noise at time k, which follows a Gaussian distribution with zero mean and covariance R.

7. The method for visually assisted tunneling operations in coal mines based on multi-sensor fusion according to claim 6, characterized in that, The data fusion results are adjusted based on the aforementioned roadway dust concentration data, including: When the dust concentration data in the roadway is less than the first dust concentration threshold, the weight of the pose information data is adjusted to the first pose information data weight threshold, and the weight of the distance data is adjusted to the first distance data weight threshold. When the dust concentration data of the roadway is greater than or equal to the first dust concentration threshold and less than the second dust concentration threshold, the weight of the pose information data is adjusted to the second pose information data weight threshold, and the weight of the distance data is adjusted to the second distance data weight threshold. When the dust concentration data of the roadway is greater than or equal to the second dust concentration threshold, the weight of the pose information data is adjusted to the third pose information data weight threshold, and the weight of the distance data is adjusted to the third distance data weight threshold. Among them, the first pose information data weight threshold > the second pose information data weight threshold > the third pose information data weight threshold, the first distance data weight threshold < the second distance data weight threshold < the third distance data weight threshold, and the first pose information data weight threshold + the first distance data weight threshold = the second pose information data weight threshold + the second distance data weight threshold = the third pose information data weight threshold + the third distance data weight threshold = 1.

8. A visualization-assisted tunneling operation device for coal mine tunneling based on multi-sensor fusion, characterized in that, include: The data acquisition module is used to collect roadway dust concentration data, roadway cross-sectional image data, and distance data between the cutting mechanism and the roadway wall during the operation of the tunneling machine cutting mechanism, and to perform data preprocessing on the roadway cross-sectional image data. The calculation module is used to calculate the pose information data of the cutting mechanism relative to the roadway cross-section based on the preprocessed roadway cross-section image data; The fusion module is used to fuse the distance data and the pose information data, and adjust the data fusion result based on the roadway dust concentration data; The display module is used to display the data fusion results in real time on the display mechanism inside the tunneling machine's cab.

9. A computer device, characterized in that, The method includes an input / output unit, a memory, and a processor, wherein the memory stores computer-readable instructions that, when executed by the processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, they cause the one or more processors to perform the steps in the method as described in any one of claims 1 to 7.

Images