Coal mining equipment obstacle avoidance method and system based on machine vision

Abstract

The invention relates to the field of excavation equipment obstacle avoidance based on image analysis, in particular to a coal mine excavation equipment obstacle avoidance method and system based on machine vision, and the method comprises the steps: obtaining the frequency domain characteristic parameters of the current electromagnetic noise during the operation of underground electromechanical equipment, and synchronously collecting the current original image frame output by a visual sensor; after a current de-noised image frame is obtained after dynamic de-noising processing, an obstacle contour feature point set is extracted through multi-scale edge detection, and an obstacle position confidence map is obtained according to space-time consistency analysis; obtaining point cloud coordinates of an image data loss area caused by electromagnetic interference, and mapping the point cloud coordinates to an image coordinate system to obtain a three-dimensional geometric complemented image; and constructing a dynamic occupation grid map of the underground environment according to the three-dimensional geometric complementation image and the obstacle position confidence map, and planning an obstacle avoidance path of the mining equipment according to the dynamic occupation grid map. And the obstacle avoidance capability of the mining equipment is improved, and safe and efficient coal mining operation is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of obstacle avoidance for mining equipment in image analysis, specifically to a method and system for obstacle avoidance of coal mine mining equipment based on machine vision. Background Art

[0002] In coal mine mining operations, the underground environment is complex and changeable, with a large number of uncertain factors, such as irregularly distributed rock protrusions, obstacles formed by collapses, and unexpected situations that may occur during the mining process. To ensure the safe and efficient operation of mining equipment and avoid collisions between the equipment and obstacles, it is crucial to achieve a reliable obstacle avoidance function. By using a vision sensor to obtain real-time image information of the underground environment and analyzing and processing the image data to identify obstacles and plan an obstacle avoidance path, it has the advantages of intuitive and rich information acquisition.

[0003] However, in practical applications, the operation of underground electromechanical equipment will generate strong electromagnetic interference, which will seriously affect the working performance of the vision sensor, resulting in a large amount of noise in the collected image data and even the loss of image data. Specifically, electromagnetic interference will cause snowflakes, stripes and other noises in the image, reducing the image quality and affecting the accurate identification of obstacles; while the loss of image data will cause the lack of information in a local area, making it difficult to obtain the complete contour of the obstacle, and thus unable to accurately determine the position and shape of the obstacle.

[0004] Therefore, there is an urgent need for a method for obstacle avoidance of coal mine mining equipment based on machine vision to plan a safe and reliable obstacle avoidance path, significantly improve the obstacle avoidance ability of coal mine mining equipment in a complex underground environment, and ensure the safe and efficient progress of mining operations. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for obstacle avoidance of coal mine mining equipment based on machine vision to solve the problems of large image noise, data loss caused by underground electromagnetic interference, and the influence of dynamic interference noise on the accuracy of obstacle position judgment during the operation of underground coal mine mining equipment.

[0006] To achieve the above purpose, on the one hand, the present invention provides a method for obstacle avoidance of coal mine mining equipment based on machine vision, and the method includes: S1. Real-time obtain the electromagnetic interference intensity distribution data generated during the operation of underground electromechanical equipment, and synchronously collect the current original image frame output by the vision sensor; extract the frequency domain characteristic parameters of the current electromagnetic noise from the electromagnetic interference intensity distribution data through spectrum analysis.

[0007] S2. Dynamically denoise the current original image frame according to the coupling relationship between the frequency-domain characteristic parameters and the operating frequency band of the visual sensor to obtain the current denoised image frame. Extract the obstacle contour feature point set from the current denoised image frame through multi-scale edge detection, and obtain the obstacle position confidence map according to the spatio-temporal consistency analysis of the obstacle contour feature point set of the previous denoised image frame adjacent in time series.

[0008] S3. When it is detected that the pixel gray variance of consecutive preset rows in the current denoised image frame is lower than the preset noise threshold, it is determined as the image data loss area caused by electromagnetic interference, and the point cloud coordinates of the image data loss area are obtained; map the point cloud coordinates to the image coordinate system to obtain the three-dimensional geometric completion image.

[0009] S4. Construct a dynamic occupancy grid map of the underground environment according to the three-dimensional geometric completion image and the obstacle position confidence map, and plan the obstacle avoidance path of the mining equipment according to the dynamic occupancy grid map.

[0010] Further, the method for dynamically denoising the current original image frame according to the coupling relationship between the frequency-domain characteristic parameters and the operating frequency band of the visual sensor includes: The frequency-domain characteristic parameters include the main interference frequency, the harmonic energy ratio, and the noise bandwidth; calculate the absolute value of the frequency difference between the main interference frequency in the frequency-domain characteristic parameters and the pixel sampling frequency band in the operating frequency band of the visual sensor. If the absolute value of the frequency difference is less than or equal to the preset frequency difference threshold, it is determined as the frequency band coupling interference mode, and a zero-phase IIR notch filter with the main interference frequency as the center frequency and the bandwidth associated with the noise bandwidth in the frequency-domain characteristic parameters is constructed to filter the frequency-domain data of the current original image frame to obtain the current denoised image frame.

[0011] If the absolute value of the frequency difference is greater than the preset frequency difference threshold and the harmonic energy ratio in the frequency-domain characteristic parameters exceeds the preset energy ratio threshold, it is determined as the broadband noise interference mode. The current original image frame is decomposed into a time-domain sequence by row scanning signals and subjected to multi-layer wavelet packet decomposition. The sub-band range to be suppressed is determined according to the noise bandwidth. After screening out the sub-bands with excessive harmonic energy, the denoised row scanning signals are reconstructed and recombined into the current denoised image frame; perform gray level equalization processing on the current denoised image frame.

[0012] Further, the method for determining the frequency band coupling interference mode and constructing a zero-phase IIR notch filter with the main interference frequency as the center frequency and the bandwidth associated with the noise bandwidth in the frequency-domain characteristic parameters to filter the frequency-domain data of the current original image frame to obtain the current denoised image frame includes: Detect whether the main interference frequency causes a periodic shift in the pixel sampling phase due to the frequency pulling effect. If the shift amount of the pixel sampling phase with a detected periodic shift exceeds a preset phase shift threshold, generate a phase correction filter according to the phase shift period and shift direction of the main interference frequency. The phase correction filter is used to compensate for the misalignment between pixel rows.

[0013] Apply the phase correction filter and the zero-phase IIR notch filter in series to the frequency-domain data of the current original image frame. First, suppress the main interference frequency and its integer multiple harmonic components through the zero-phase IIR notch filter, and then perform phase compensation on the filtered frequency-domain data through the phase correction filter to restore the alignment of pixel rows and columns. Perform an inverse Fourier transform on the phase-compensated frequency-domain data to generate the current denoised image frame after phase correction.

[0014] Further, the method for detecting whether the main interference frequency causes a periodic shift in the pixel sampling phase due to the frequency pulling effect includes: Perform a two-dimensional Fourier transform on the current original image frame to extract the spatial spectrum, analyze the periodic peak components corresponding to the main interference frequency in the spatial spectrum, and calculate the spatial frequency and phase distribution of the peak components. Determine the period of the misalignment between pixel rows according to the spatial frequency, and judge the misalignment direction according to the positive and negative polarities of the phase distribution. If it is detected that the amplitude of the periodic peak component exceeds a preset amplitude threshold and the absolute value of the frequency difference between the spatial frequency and the main interference frequency is inversely proportional, it is determined that the periodic shift of the pixel sampling phase is caused by the frequency pulling effect.

[0015] Further, the method for obtaining the obstacle position confidence map according to the spatio-temporal consistency analysis of the set of obstacle contour feature points of the previous denoised image frame adjacent in time sequence includes: Match the set of obstacle contour feature points of the current denoised image frame with the set of feature points of the previous denoised image frame point by point, calculate the displacement vectors of the successfully matched feature point pairs, and assign initial confidence weights to the successfully matched feature points. Construct a motion vector field according to the direction and amplitude of the displacement vectors, and extract the direction and amplitude of the displacement vectors of the motion vector field through a density clustering algorithm as clustering features. Calculate the direction variance and amplitude standard deviation of the displacement vectors within each clustering cluster. If the direction variance is lower than a preset direction consistency threshold and the amplitude standard deviation is less than a preset amplitude stability threshold, it is determined that the clustering cluster is a set of static obstacle feature points; otherwise, it is determined that the clustering cluster is a set of dynamic interference noise feature points.

[0016] For the set of feature points of static obstacles, an adjustment coefficient is obtained by calculating the cosine similarity between the displacement of the feature points of the static obstacles and the displacement of the clustering center, and the initial confidence weight is corrected according to the adjustment coefficient; for the set of feature points of dynamic interference noise, a noise intensity factor is calculated according to its direction variance and amplitude standard deviation, and the initial confidence weight is corrected according to the noise intensity factor; the obstacle position confidence map is obtained after correcting the initial confidence weights of the set of feature points of static obstacles and the set of feature points of dynamic interference noise and performing temporal smoothing filtering.

[0017] Further, the method for determining that when the pixel gray variance of consecutive preset rows in the current denoised image frame is lower than the preset noise threshold, it is the image data loss area caused by electromagnetic interference includes: Calculate the pixel gray variance row by row for the current denoised image frame. If the pixel gray variances of consecutive preset rows are all lower than the preset noise threshold, it is determined as a candidate image data loss area; obtain the real-time electromagnetic interference intensity distribution data corresponding to the candidate image data loss area. If the electromagnetic interference intensity distribution data in this area exceeds the preset interference intensity threshold, it is determined as the image data loss area caused by electromagnetic interference; if the electromagnetic interference intensity does not exceed the threshold, verify whether there is persistent pixel gray variance abnormality in the candidate image data loss area through multiple frames of historical denoised images.

[0018] Further, the method for mapping the point cloud coordinates to the image coordinate system to obtain a three-dimensional geometric completion image includes: Calculate the rigid body transformation matrix according to the joint calibration parameters of the vision sensor and the lidar. The rigid body transformation matrix includes a rotation matrix and a translation vector; obtain the point cloud coordinates of the image data loss area through the lidar; project the point cloud coordinates to the image coordinate system through the rigid body transformation matrix to obtain a three-dimensional geometric completion point cloud projection corresponding to the image data loss area; perform surface reconstruction on the three-dimensional geometric completion point cloud projection to generate a triangular mesh model, and render the triangular mesh model into a three-dimensional geometric completion image with the same resolution and viewing angle as the current denoised image frame.

[0019] Further, the method for constructing a dynamic occupancy grid map of the underground environment according to the three-dimensional geometric completion image and the obstacle position confidence map includes: Perform rasterization segmentation on the three-dimensional geometric completion image to generate an initial grid map, and extract the geometric structure integrity parameter of each grid cell in the initial grid map; map the confidence value of the corresponding grid cell in the obstacle position confidence map to the same grid coordinate system, and calculate the initial occupancy probability of each grid cell according to the weighted fusion of the geometric structure integrity parameter and the confidence value.

[0020] Update the historical occupancy probability in time series according to the real-time coordinate information of the mining equipment, exponentially decay the historical occupancy probability through a dynamic decay factor, and fuse the initial occupancy probability to generate the updated occupancy probability; if the updated occupancy probability exceeds the preset static obstacle determination threshold, mark it as an impassable area; otherwise, mark it as a passable area; combine the impassable area and the passable area with the initial grid map to obtain a dynamic occupancy grid map.

[0021] Further, the method for extracting the geometric structure integrity parameter of each grid unit in the initial grid map includes: Perform statistical filtering on the point cloud data of the three-dimensional geometric completion image within the grid unit to obtain the effective point cloud quantity, and calculate the ratio of the effective point cloud quantity to the grid unit volume as the point cloud density; perform consistency analysis on the normal vector distribution of the point cloud within the grid unit, calculate the eigenvalue of the covariance matrix of the normal vector through principal component analysis, if the ratio of the maximum eigenvalue to the minimum eigenvalue of the covariance matrix is lower than the preset feature threshold, it is determined that the surface continuity is strong, otherwise it is determined as surface fracture; normalize the point cloud density to obtain the density integrity factor, and convert the strong surface continuity and surface fracture into surface continuity coefficients, and obtain the geometric structure integrity parameter through weighted calculation according to the density integrity factor and the surface continuity coefficient.

[0022] On the other hand, based on the same inventive concept, the present invention also provides an obstacle avoidance system for coal mine mining equipment based on machine vision, the system includes: a data analysis module, an obstacle position confidence map acquisition module, a three-dimensional geometric completion image acquisition module, and an obstacle avoidance planning management module, and the modules are sequentially communicatively connected; The data analysis module is used to obtain the electromagnetic interference intensity distribution data generated during the operation of the underground electromechanical equipment in real time, and synchronously collect the current original image frame output by the vision sensor; extract the frequency domain characteristic parameters of the current electromagnetic noise from the electromagnetic interference intensity distribution data through spectrum analysis; The obstacle position confidence map acquisition module is used to perform dynamic denoising processing on the current original image frame according to the coupling relationship between the frequency domain characteristic parameters and the working frequency band of the vision sensor to obtain the current denoised image frame, extract the obstacle contour feature point set from the current denoised image frame through multi-scale edge detection, and obtain the obstacle position confidence map according to the spatio-temporal consistency analysis of the obstacle contour feature point set of the previous denoised image frame adjacent in time series; The three-dimensional geometric completion image acquisition module is used to determine that it is an image data loss area caused by electromagnetic interference when the pixel gray variance of consecutive preset rows in the current denoised image frame is lower than the preset noise threshold, and obtain the point cloud coordinates of the image data loss area; map the point cloud coordinates to the image coordinate system to obtain a three-dimensional geometric completion image; The obstacle avoidance planning and management module is used to construct a dynamic occupancy grid map of the underground environment based on the three-dimensional geometric completion image and the obstacle position confidence map, and plan an obstacle avoidance path for the mining equipment according to the dynamic occupancy grid map.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By analyzing the coupling relationship between the frequency domain characteristic parameters of electromagnetic interference and the working frequency band of the vision sensor, methods such as zero-phase IIR notch filter or wavelet packet decomposition are specifically used to perform dynamic denoising processing on the image, effectively removing electromagnetic noise interference. At the same time, for the image data loss area caused by electromagnetic interference, the three-dimensional geometry is completed by obtaining the point cloud coordinates and mapping them to the image coordinate system, solving the problem of missing image information, significantly improving the image quality, and providing a reliable data basis for subsequent obstacle recognition and environmental map construction.

[0024] 2. The spatio-temporal consistency analysis is carried out by using the obstacle contour feature point sets of adjacent denoised image frames in time series. Through a series of operations such as point-by-point matching, constructing a motion vector field, density clustering, and confidence weight correction, the static obstacle feature point set and the dynamic interference noise feature point set are accurately distinguished, and an accurate obstacle position confidence map is obtained, effectively reducing the influence of dynamic interference noise on the judgment of obstacle positions and improving the accuracy of obstacle position determination.

[0025] 3. Combining the three-dimensional geometric completion image and the obstacle position confidence map, rasterizing and segmenting the three-dimensional geometric completion image, extracting the geometric structure integrity parameters of the grid cells, and weighted fusion with the confidence values to calculate the initial occupancy probability, and then obtaining the updated occupancy probability through time series update and dynamic decay factor, thereby constructing a reliable dynamic occupancy grid map, providing accurate environmental information for obstacle avoidance path planning, and ensuring the rationality and safety of obstacle avoidance path planning. Brief Description of the Drawings

[0026] Figure 1 It is a flowchart of the obstacle avoidance method for coal mining and excavation equipment based on machine vision in Embodiment 1 of the present invention.

[0027] Figure 2 It is a schematic diagram of the module composition of the obstacle avoidance system for coal mining and excavation equipment based on machine vision in Embodiment 2 of the present invention. Detailed Embodiments

[0028] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0029] Before giving examples, it is necessary to elaborate on the application scenario of the inventive concept. The present invention is an obstacle avoidance method and system for coal mining and excavation equipment based on machine vision, which is applied to coal mining and excavation equipment to achieve reliable obstacle detection and obstacle avoidance path planning in the strong electromagnetic interference environment underground. It effectively solves the problems of large image noise, data loss caused by underground electromagnetic interference, and the influence of dynamic interference noise on the accuracy of obstacle position judgment, and significantly improves the operation safety and operation efficiency of the excavation equipment.

[0030] Example 1: As Figure 1 shown, this embodiment provides an obstacle avoidance method for coal mining and excavation equipment based on machine vision, and the method includes: S1. Real-time obtain the electromagnetic interference intensity distribution data generated during the operation of underground electromechanical equipment, and synchronously collect the current original image frame output by the visual sensor; extract the frequency domain characteristic parameters of the current electromagnetic noise through spectrum analysis of the electromagnetic interference intensity distribution data; in the coal mine underground environment, the operation of electromechanical equipment will generate electromagnetic interference. The electromagnetic field intensity change generated by large electromechanical equipment such as shearers, roadheaders, and conveyors is captured through an electromagnetic field intensity sensor array installed around the excavation equipment. At the same time, the visual sensor (including high-definition cameras and industrial cameras installed at the front end of the excavation equipment) will synchronously collect the original image frame of the current underground environment. For example, when the shearer cuts the coal seam or the roadheader drills into the rock stratum, equipment such as motors and frequency converters will generate strong electromagnetic interference in the range of 75 - 120 Hz, and these interferences will directly affect the image quality. For the obtained electromagnetic interference intensity distribution data, the frequency domain characteristic parameters of the current electromagnetic noise are extracted through fast Fourier transform (FFT) spectrum analysis. It includes the main interference frequency (usually the drive frequency of the working motor and its harmonics), the harmonic energy ratio (reflecting the energy distribution of the interference signal), and the noise bandwidth (characterizing the spectral broadening degree of the interference signal). For example, when the shearer cuts hard coal seams, the motor load increases, and electromagnetic interference with a main frequency of 100 Hz, a harmonic energy ratio of up to 35%, and a noise bandwidth of about 20 Hz may be generated.

[0031] Performing dynamic denoising processing on the current original image frame according to the coupling relationship between the frequency domain characteristic parameters and the operating frequency band of the visual sensor to obtain the current denoised image frame, extracting the obstacle contour feature point set from the current denoised image frame through multi-scale edge detection, and obtaining the obstacle position confidence map according to the spatio-temporal consistency analysis of the obstacle contour feature point set of the previous denoised image frame adjacent in time series; in the underground coal mine environment, electromagnetic interference may cause obvious horizontal stripes, snowflake noise, or periodic ripples in the image. The multi-scale edge detection algorithm can be implemented by combining the Canny edge detection algorithm with the Gaussian pyramid. Applying the Canny edge detection on the Gaussian pyramid images at different scales to obtain edge information at different scales, and then fusing these edge information to extract the obstacle contour feature point set. These feature point sets describe the edge contours of potential obstacles such as rock outcrops, support equipment, and pipeline systems in the underground environment.

[0032] When it is detected that the pixel gray variance of consecutive preset rows in the current denoised image frame is lower than the preset noise threshold, it is determined as the image data loss area caused by electromagnetic interference, and the point cloud coordinates of the image data loss area are obtained; mapping the point cloud coordinates to the image coordinate system to obtain a three-dimensional geometric completion image; when it is detected that there are consecutive preset rows in the current denoised image frame whose pixel gray variance is lower than the preset noise threshold, these areas will be determined as the image data loss areas caused by electromagnetic interference. This situation is particularly common when high-power mining equipment starts or the load changes underground, such as when the shearer cuts through a fault or the roadheader encounters hard rock strata. Subsequently, the point cloud coordinates of these image data loss areas are obtained through auxiliary sensors (such as lidar), and these point cloud coordinates provide the three-dimensional spatial position information of the obstacles. The completed image makes up for the lost information in the original image and can obtain the complete underground environment information.

[0033] Constructing a dynamic occupancy grid map of the underground environment according to the three-dimensional geometric completion image and the obstacle position confidence map, and planning an obstacle avoidance path for the mining equipment according to the dynamic occupancy grid map. Using the A* algorithm or the Dijkstra algorithm to plan an obstacle avoidance path for the mining equipment on the dynamic occupancy grid map to ensure the safe and efficient progress of the mining operation.

[0034] The method of obtaining the current denoised image frame by performing dynamic denoising processing on the current original image frame according to the coupling relationship between the frequency domain characteristic parameters and the operating frequency band of the visual sensor includes: The frequency-domain characteristic parameters include the main interference frequency, the harmonic energy ratio, and the noise bandwidth; calculate the absolute value of the frequency difference between the main interference frequency in the frequency-domain characteristic parameters and the pixel sampling frequency band in the working frequency band of the vision sensor. If the absolute value of the frequency difference is less than or equal to the preset frequency difference threshold, it is determined as the frequency band coupling interference mode, and a zero-phase IIR notch filter with the main interference frequency as the center frequency and the bandwidth associated with the noise bandwidth in the frequency-domain characteristic parameters is constructed to filter the frequency-domain data of the current original image frame to obtain the current denoised image frame; the pixel sampling frequency of the vision sensor is usually in the range of 20 - 60 Hz, while the interference frequencies of underground motor equipment are mostly in the range of 50 - 120 Hz. When the absolute value of the frequency difference is less than or equal to the preset frequency difference threshold (usually set to 15 - 20 Hz), it is determined as the frequency band coupling interference mode. In the frequency band coupling interference mode, a zero-phase IIR notch filter with the main interference frequency as the center frequency and the bandwidth associated with the noise bandwidth will be constructed. For example, when the detected main interference frequency generated during the operation of the shearer is 98 Hz and the noise bandwidth is 15 Hz, a notch filter with a center frequency of 98 Hz and a bandwidth of 15 Hz will be constructed to filter the frequency-domain data of the current original image frame, thereby obtaining the current denoised image frame. The zero-phase design ensures that the filtering process does not introduce phase distortion and maintains the accuracy of the object edge positions in the image.

[0035] If the absolute value of the frequency difference is greater than the preset frequency difference threshold and the harmonic energy ratio in the frequency-domain characteristic parameters exceeds the preset energy ratio threshold, it is determined as the broadband noise interference mode. The current original image frame is decomposed into a time-domain sequence according to the line scan signal and undergoes multi-level wavelet packet decomposition. The sub-band range to be suppressed is determined according to the noise bandwidth. After screening out the sub-bands with excessive harmonic energy, the denoised line scan signal is reconstructed and recombined into the current denoised image frame; the current denoised image frame is subjected to gray-level equalization processing.

[0036] When the absolute value of the frequency difference is greater than the preset frequency difference threshold and the harmonic energy ratio exceeds the preset energy ratio threshold (usually set to 30%), it is determined as the broadband noise interference mode. This situation is more common when the load of the mining equipment suddenly changes or multiple devices are operating simultaneously. The current original image frame will be decomposed into a time-domain sequence according to the line scan signal and undergo multi-level wavelet packet decomposition. The Daubechies wavelet can be used as the wavelet basis function. The sub-band range to be suppressed is determined according to the noise bandwidth. After screening out the sub-bands with excessive harmonic energy, the denoised line scan signal is reconstructed and recombined into the current denoised image frame. For example, when broadband noise interference is detected, 3 - 5 levels of wavelet packet decomposition will be performed, and the energy of the corresponding sub-bands will be suppressed according to the interference bandwidth characteristics. The histogram equalization algorithm is used to perform gray-level equalization processing on the current denoised image frame to enhance the contrast of the image and make the obstacle contours clearer.

[0037] The method for determining that it is in the frequency band coupling interference mode and constructing a zero-phase IIR notch filter with the main interference frequency as the center frequency and the bandwidth associated with the noise bandwidth in the frequency domain characteristic parameters to filter the frequency domain data of the current original image frame to obtain the current denoised image frame includes: Detect whether the main interference frequency causes a periodic offset of the pixel sampling phase due to the frequency pulling effect. If the detected periodic offset of the pixel sampling phase offset exceeds the preset phase offset threshold, a phase correction filter is generated according to the phase offset period and direction of the main interference frequency. The phase correction filter is used to compensate for the misalignment between pixel rows.

[0038] Wherein, ϕ(z) is a phase compensation function designed according to the phase offset period and direction of the main interference frequency, and linear phase compensation or non-linear phase compensation methods can be adopted to align the pixel rows and columns after filtering.

[0039] The phase correction filter and the zero-phase IIR notch filter are serially applied to the frequency domain data of the current original image frame. First, the zero-phase IIR notch filter is used to suppress the main interference frequency and its integer multiple harmonic components, and then the phase correction filter is used to perform phase compensation on the filtered frequency domain data to restore the alignment of pixel rows and columns; the inverse Fourier transform is performed on the phase-compensated frequency domain data to generate the current denoised image frame after phase correction.

[0040] When high-power equipment in the coal mine works, unstable power grid fluctuations will cause slight fluctuations in the motor drive frequency, which in turn causes periodic offsets in the sampling phase of the vision sensor, manifested as periodic horizontal misalignments in the image. When the detected periodic offset of the pixel sampling phase offset exceeds the preset phase offset threshold (usually π / 8), a phase correction filter is generated according to the phase offset period and direction of the main interference frequency. The phase correction filter is used to compensate for the misalignment between pixel rows and avoid the distortion of the obstacle contour caused by row misalignment. For example, when the detected phase offset period is one cycle per 20 rows and the offset direction is right offset, the corresponding correction filter parameters are generated. Finally, the inverse Fourier transform is performed on the phase-compensated frequency domain data to generate the current denoised image frame after phase correction.

[0041] The method for detecting whether the main interference frequency causes a periodic offset of the pixel sampling phase due to the frequency pulling effect includes: Perform a two-dimensional Fourier transform on the current original image frame to extract the spatial spectrum, analyze the periodic peak components corresponding to the main interference frequency in the spatial spectrum, calculate the spatial frequency and phase distribution of the peak components; determine the period of pixel row misalignment according to the spatial frequency, and judge the misalignment direction according to the positive and negative polarities of the phase distribution; if it is detected that the amplitude of the periodic peak component exceeds a preset amplitude threshold and the absolute value of the frequency difference between the spatial frequency and the main interference frequency is inversely proportional, it is determined that the periodic offset of the pixel sampling phase is caused by the frequency pulling effect.

[0042] Use the two-dimensional fast Fourier transform (2D - FFT) algorithm to process the current original image frame and convert the image matrix into a frequency domain matrix. For example, when the main interference frequency is 100 Hz, the corresponding periodic peak components are searched for in the spatial spectrum. The period of pixel row misalignment can be determined according to the calculated spatial frequency, such as misalignment occurring every 15 rows or every 20 rows, and the misalignment direction can be judged according to the positive and negative polarities of the phase distribution, such as misalignment to the left or right. When it is detected that the amplitude of the periodic peak component exceeds a preset amplitude threshold, usually set to 3 - 5 times the average energy of the spatial spectrum, and the absolute value of the frequency difference between the spatial frequency and the main interference frequency is inversely proportional, it is determined that the periodic offset of the pixel sampling phase is caused by the frequency pulling effect. This situation is particularly obvious when the mining equipment starts or the load suddenly changes.

[0043] The method for obtaining the obstacle position confidence map through the spatio-temporal consistency analysis of the set of spatio-temporal feature points of the obstacle in the previous denoised image frame adjacent in time sequence includes: The obstacle contour feature point set of the current denoised image frame is matched point by point with the feature point set of the previous denoised image frame. The displacement vectors of the successfully matched feature point pairs are calculated, and initial confidence weights are assigned to the successfully matched feature points. A motion vector field is constructed based on the direction and amplitude of the displacement vectors. The displacement vectors of the motion vector field are used as clustering features by extracting the direction and amplitude of the displacement vectors through a density clustering algorithm. The direction variance and amplitude standard deviation of the displacement vectors within each clustering cluster are calculated. If the direction variance is lower than a preset direction consistency threshold and the amplitude standard deviation is less than a preset amplitude stability threshold, then the clustering cluster is determined to be a static obstacle feature point set; otherwise, it is determined to be a dynamic interference noise feature point set. An improved SIFT or ORB feature descriptor is used for matching, the displacement vectors of the successfully matched feature point pairs are calculated, and initial confidence weights are assigned to the successfully matched feature points. For example, in the case where the interval between two frames of images is 100 ms, the feature points of static obstacles such as support equipment and rock protrusions can be effectively tracked. A motion vector field is constructed based on the direction and amplitude of the displacement vectors, and then the displacement vectors of the motion vector field are used as clustering features by extracting the direction and amplitude of the displacement vectors through a density clustering algorithm (such as the DBSCAN algorithm). The direction variance and amplitude standard deviation of the displacement vectors within each clustering cluster are calculated. If the direction variance is lower than a preset direction consistency threshold (usually 0.1 - 0.2 radians) and the amplitude standard deviation is less than a preset amplitude stability threshold (usually 2 - 3 pixels), then the clustering cluster is determined to be a static obstacle feature point set; otherwise, it is determined to be a dynamic interference noise feature point set. This can effectively distinguish static obstacles (such as rock protrusions and support equipment in the roadway) and dynamic interference noises (such as underground dust, water mist, or temporarily moving objects) in the underground environment.

[0044] For the static obstacle feature point set, an adjustment coefficient is obtained by calculating the cosine similarity between the displacement of the static obstacle feature points and the displacement of the clustering center, and the initial confidence weight is corrected according to the adjustment coefficient. For the dynamic interference noise feature point set, a noise intensity factor is calculated based on its direction variance and amplitude standard deviation, and the initial confidence weight is corrected according to the noise intensity factor. The obstacle position confidence map is obtained after performing temporal smoothing filtering based on the corrected initial confidence weights of the static obstacle feature point set and the dynamic interference noise feature point set. Correcting the initial confidence weight according to the noise intensity factor usually reduces its weight value. The obstacle position confidence map is obtained by applying exponential weighted moving average filtering based on the corrected initial confidence weights of the static obstacle feature point set and the dynamic interference noise feature point set. This processing method can accurately determine the position of obstacles even when the underground dust concentration is relatively high or the water mist interference is relatively strong.

[0045] The method of determining the image data loss area caused by electromagnetic interference when the pixel gray variance of consecutive preset rows in the current denoised image frame is lower than the preset noise threshold includes: Calculate the pixel gray variance row by row for the current denoised image frame. If the pixel gray variances of consecutive preset rows are all lower than the preset noise threshold, it is determined as a candidate image data loss area; obtain the real-time electromagnetic interference intensity distribution data corresponding to the candidate image data loss area. If the electromagnetic interference intensity distribution data in this area exceeds the preset interference intensity threshold, it is determined as the image data loss area caused by electromagnetic interference; if the electromagnetic interference intensity does not exceed the threshold, verify whether there is persistent pixel gray variance abnormality in the candidate image data loss area through multiple frames of historical denoised images. Calculate the pixel gray variance row by row for the current denoised image frame. If the pixel gray variances of consecutive preset rows (usually 10 - 15% of the image height) are all lower than the preset noise threshold (usually set to 5 - 10 gray levels), it is determined as a candidate image data loss area. For example, in an image with a resolution of 1080p, if the pixel gray variances of consecutive 100 rows are all lower than 8 gray levels, these areas will be marked as candidate image data loss areas. Obtain the real-time electromagnetic interference intensity distribution data corresponding to the candidate image data loss area. If the electromagnetic interference intensity distribution data in this area exceeds the preset interference intensity threshold, usually set to 5 - 10 times the background electromagnetic field intensity, it is determined as the image data loss area caused by electromagnetic interference. If the electromagnetic interference intensity does not exceed the threshold, verify whether there is persistent pixel gray variance abnormality in the candidate image data loss area through multiple frames of historical denoised images. This verification can effectively distinguish the image data loss caused by electromagnetic interference from the low gray variance situation caused by the relatively uniform underground environment itself (such as a flat roadway wall).

[0046] The method of mapping the point cloud coordinates to the image coordinate system to obtain a three-dimensional geometric completion image includes: Calculate the rigid body transformation matrix according to the joint calibration parameters of the vision sensor and the lidar. The rigid body transformation matrix includes a rotation matrix and a translation vector; obtain the point cloud coordinates of the missing area of the image data through the lidar; project the point cloud coordinates into the image coordinate system through the rigid body transformation matrix to obtain the three-dimensional geometric completion point cloud projection corresponding to the missing area of the image data; perform surface reconstruction on the three-dimensional geometric completion point cloud projection to generate a triangular mesh model, and render the triangular mesh model into a three-dimensional geometric completion image consistent with the resolution and viewing angle of the current denoised image frame. Joint calibration is usually completed through a calibration board before the device is put into use, and the geometric relationship between the camera and the lidar will be calculated during the calibration process. The lidar can work stably in an electromagnetic interference environment and provide high-precision three-dimensional structure information. For example, when data loss is detected in the central area of the image, the point cloud data within the viewing angle range of the lidar will be extracted and coordinate transformation will be performed. For example, by using an improved Poisson surface reconstruction algorithm or a greedy triangulation algorithm, point cloud data with uneven distribution can be processed. Render the triangular mesh model into a three-dimensional geometric completion image consistent with the resolution and viewing angle of the current denoised image frame, so as to fill the missing area information in the original image. This completion method can ensure that complete underground environment information can still be obtained in the case of severe electromagnetic interference.

[0047] The method for constructing a dynamic occupancy grid map of the underground environment according to the three-dimensional geometric completion image and the obstacle position confidence map includes: Perform rasterization segmentation on the three-dimensional geometric completion image to generate an initial grid map, and extract the geometric structure integrity parameter of each grid cell in the initial grid map; map the confidence value of the corresponding grid cell in the obstacle position confidence map to the same grid coordinate system, and calculate the initial occupancy probability of each grid cell according to the weighted fusion of the geometric structure integrity parameter and the confidence value; for example, the grid cell size is usually set to 10-20 cm, and such a resolution can balance the calculation efficiency and obstacle avoidance accuracy. Extract the geometric structure integrity parameter of each grid cell in the initial grid map, and the geometric structure integrity parameter reflects the density and surface continuity of the point cloud distribution within the grid cell. For example, when the geometric structure integrity parameter of a certain grid cell is 0.8 and the corresponding confidence value is 0.75, an initial occupancy probability of 0.78 may be calculated.

[0048] Update the historical occupancy probability in time series according to the real-time coordinate information of the mining equipment, perform exponential decay on the historical occupancy probability through a dynamic decay factor, and fuse the initial occupancy probability to generate the updated occupancy probability; if the updated occupancy probability exceeds the preset static obstacle determination threshold, mark it as an impassable area; otherwise, mark it as a passable area; combine the impassable area and the passable area with the initial grid map to obtain a dynamic occupancy grid map. In the underground environment, as the mining operation progresses, the state of some areas may change, and the dynamic decay mechanism can ensure the timeliness of the map information. If the updated occupancy probability exceeds the preset static obstacle determination threshold (usually set to 0.6 - 0.7), mark it as an impassable area; otherwise, mark it as a passable area. Combine the impassable area and the passable area with the initial grid map to obtain a dynamic occupancy grid map, which serves as the basis for obstacle avoidance path planning.

[0049] The method for extracting the geometric structure integrity parameter of each grid cell in the initial grid map includes: Perform statistical filtering on the point cloud data of the three-dimensional geometric completion image within the grid cell to obtain the number of effective point clouds, and calculate the ratio of the number of effective point clouds to the volume of the grid cell as the point cloud density; perform consistency analysis on the normal vector distribution of the point cloud within the grid cell, calculate the eigenvalue of the covariance matrix of the normal vector through principal component analysis, if the ratio of the maximum eigenvalue to the minimum eigenvalue of the covariance matrix is lower than the preset feature threshold, it is determined that the surface continuity is strong, otherwise it is determined as surface fracture; normalize the point cloud density to obtain the density integrity factor, and convert the strong surface continuity and surface fracture into surface continuity coefficients, and obtain the geometric structure integrity parameter through weighted calculation based on the density integrity factor and the surface continuity coefficient. For example, in a grid cell of 10cm × 10cm × 10cm, if the number of effective point clouds is 120, the point cloud density is 120 / (10×10×10) = 0.12 points / cm³. Strong surface continuity indicates that the surface within the grid cell is relatively smooth and uniform; surface fracture indicates that there may be discontinuous structures such as edges and corners within the grid cell. Normalize the point cloud density to obtain the density integrity factor, and convert the strong surface continuity and surface fracture into surface continuity coefficients. Usually, the strong continuity is assigned a value of 0.9 - 1.0, and the surface fracture is assigned a value of 0.5 - 0.7.

[0050] Embodiment 2: Based on the same inventive concept, as Figure 2 shown, this embodiment also provides an obstacle avoidance system for coal mine mining equipment based on machine vision. The system includes: a data analysis module, an obstacle position confidence map acquisition module, a three-dimensional geometric completion image acquisition module, and an obstacle avoidance planning management module, and the modules are communicatively connected in sequence; The data analysis module is used to obtain the electromagnetic interference intensity distribution data generated during the operation of underground electromechanical equipment in real time, and synchronously collect the current original image frame output by the vision sensor; extract the frequency domain characteristic parameters of the current electromagnetic noise from the electromagnetic interference intensity distribution data through spectrum analysis; The obstacle position confidence map acquisition module is used to perform dynamic denoising processing on the current original image frame according to the coupling relationship between the frequency domain characteristic parameters and the working frequency band of the vision sensor to obtain the current denoised image frame, extract the obstacle contour feature point set from the current denoised image frame through multi-scale edge detection, and obtain the obstacle position confidence map according to the spatio-temporal consistency analysis of the obstacle contour feature point set of the previous denoised image frame adjacent in time series; The three-dimensional geometric completion image acquisition module is used to determine that the area is an image data loss area caused by electromagnetic interference when it is detected that the pixel gray variance of consecutive preset rows of pixels in the current denoised image frame is lower than the preset noise threshold, and obtain the point cloud coordinates of the image data loss area; map the point cloud coordinates to the image coordinate system to obtain a three-dimensional geometric completion image; The obstacle avoidance planning and management module is used to construct a dynamic occupancy grid map of the underground environment according to the three-dimensional geometric completion image and the obstacle position confidence map, and plan the obstacle avoidance path of the mining equipment according to the dynamic occupancy grid map.

[0051] It should be noted that regarding the system in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0052] Finally, it should be noted that although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A coal mining equipment obstacle avoidance method based on machine vision, characterized in that: The method comprises: Real-time acquisition of electromagnetic interference intensity distribution data generated by underground electromechanical equipment during operation, and synchronous acquisition of the current original image frame output by the visual sensor; extracting frequency domain characteristic parameters of the current electromagnetic noise through spectrum analysis of the electromagnetic interference intensity distribution data; According to the coupling relationship between the frequency domain feature parameters and the working frequency band of the visual sensor, the current original image frame is dynamically denoised to obtain a current denoised image frame, the obstacle contour feature point set is extracted from the current denoised image frame through multi-scale edge detection, and the obstacle position confidence map is obtained according to the spatiotemporal consistency analysis of the obstacle contour feature point set of the previous denoised image frame adjacent in time sequence; When it is detected that the pixel grayscale variance of a preset number of consecutive rows in the current denoised image frame is lower than a preset noise threshold, it is determined to be an image data loss area caused by electromagnetic interference, and the point cloud coordinates of the image data loss area are obtained; the point cloud coordinates are mapped to the image coordinate system to obtain a three-dimensional geometric completion image; A dynamic occupancy grid map of the underground environment is constructed according to the three-dimensional geometric completion image and the obstacle position confidence map, and an obstacle avoidance path of the mining equipment is planned according to the dynamic occupancy grid map.

2. The method for avoiding obstacles in coal mining equipment based on machine vision according to claim 1, characterized in that: The method of obtaining a current denoised image frame by dynamically denoising the current original image frame according to the coupling relationship between the frequency domain characteristic parameters and the working frequency band of the visual sensor comprises: The frequency domain characteristic parameters include the main interference frequency, the harmonic energy ratio, and the noise bandwidth; the absolute value of the frequency difference between the main interference frequency in the frequency domain characteristic parameters and the pixel sampling frequency band in the working frequency band of the visual sensor is calculated; if the absolute value of the frequency difference is less than or equal to the preset frequency difference threshold, it is determined to be a frequency band coupling interference mode, and a zero-phase IIR notch filter with the main interference frequency as the center frequency and a bandwidth associated with the noise bandwidth in the frequency domain characteristic parameters is constructed to filter the frequency domain data of the current original image frame to obtain the current denoised image frame; If the absolute value of the frequency difference is greater than the preset frequency difference threshold and the harmonic energy ratio in the frequency domain characteristic parameter exceeds the preset energy ratio threshold, it is determined to be a broadband noise interference mode, and the current original image frame is decomposed into a time domain sequence according to the line scanning signal and multi-layer wavelet packet decomposition is performed, and the sub-band range to be suppressed is determined according to the noise bandwidth, and the sub-band with excessive harmonic energy is screened out and the denoised line scanning signal is reconstructed and reorganized into the current denoised image frame; grayscale equalization processing is performed on the current denoised image frame.

3. The method for avoiding obstacles in coal mining equipment based on machine vision according to claim 2, characterized in that: The method of determining that the interference mode is frequency band coupling, constructing a zero-phase IIR notch filter with a main interference frequency as the center frequency and a bandwidth associated with the noise bandwidth in the frequency domain characteristic parameter to filter the frequency domain data of the current original image frame to obtain the current denoised image frame includes: Detecting whether the main interference frequency causes a periodic shift in the pixel sampling phase due to a frequency pulling effect, and if it is detected that the periodically shifted pixel sampling phase shift exceeds a preset phase shift threshold, generating a phase correction filter according to the phase shift period and shift direction of the main interference frequency, wherein the phase correction filter is used to compensate for the misalignment between pixel rows; The phase correction filter and the zero-phase IIR notch filter are connected in series and applied to the frequency domain data of the current original image frame. The main interference frequency and its integer multiple harmonic components are first suppressed by the zero-phase IIR notch filter, and then the filtered frequency domain data is phase compensated by the phase correction filter to restore the pixel row and column alignment; an inverse Fourier transform is performed on the phase-compensated frequency domain data to generate a current denoised image frame after phase correction.

4. The method for avoiding obstacles in coal mining equipment based on machine vision according to claim 3, characterized in that: The method for detecting whether the main interference frequency causes the periodic shift of the pixel sampling phase due to the frequency pulling effect includes: Perform a two-dimensional Fourier transform on the current original image frame to extract the spatial spectrum, analyze the periodic peak component corresponding to the main interference frequency in the spatial spectrum, and calculate the spatial frequency and phase distribution of the peak component; determine the period of pixel row misalignment based on the spatial frequency, and judge the misalignment direction based on the positive and negative polarity of the phase distribution; if it is detected that the amplitude of the periodic peak component exceeds the preset amplitude threshold and the spatial frequency is inversely proportional to the absolute value of the frequency difference between the main interference frequency, it is determined to be a periodic offset of the pixel sampling phase caused by the frequency pulling effect.

5. The method for avoiding obstacles in coal mining equipment based on machine vision according to claim 1, characterized in that: The method for obtaining the obstacle position confidence map based on the spatiotemporal consistency analysis of the obstacle contour feature point set of the previous denoised image frame adjacent in time sequence comprises: Match the obstacle contour feature point set of the current denoised image frame with the feature point set of the previous denoised image frame point by point, calculate the displacement vector of the successfully matched feature point pair and assign an initial confidence weight to the successfully matched feature point; construct a motion vector field according to the direction and amplitude of the displacement vector, and extract the direction and amplitude of the displacement vector of the motion vector field as clustering features through a density clustering algorithm; calculate the directional variance and amplitude standard deviation of the displacement vector in each cluster cluster, and if the directional variance is lower than a preset directional consistency threshold and the amplitude standard deviation is lower than a preset amplitude stability threshold, determine that the cluster cluster is a static obstacle feature point set; otherwise, determine that the cluster cluster is a dynamic interference noise feature point set; For the static obstacle feature point set, the adjustment coefficient is obtained by calculating the cosine similarity between the displacement of the static obstacle feature point and the displacement of the cluster center, and the initial confidence weight is corrected according to the adjustment coefficient; for the dynamic interference noise feature point set, the noise intensity factor is calculated according to its directional variance and amplitude standard deviation, and the initial confidence weight is corrected according to the noise intensity factor; according to the corrected initial confidence weights of the static obstacle feature point set and the dynamic interference noise feature point set and after time series smoothing filtering, the obstacle position confidence map is obtained.

6. The method for avoiding obstacles in coal mining equipment based on machine vision according to claim 1, characterized in that: The method of determining that the image data loss area is caused by electromagnetic interference when the pixel grayscale variance of a preset number of consecutive rows in the current denoised image frame is lower than a preset noise threshold comprises: The pixel grayscale variance of the current denoised image frame is calculated row by row. If the pixel grayscale variance of a preset number of consecutive rows is lower than the preset noise threshold, it is determined to be a candidate image data loss area. The real-time electromagnetic interference intensity distribution data corresponding to the candidate image data loss area is obtained. If the electromagnetic interference intensity distribution data of the area exceeds the preset interference intensity threshold, it is determined to be an image data loss area caused by electromagnetic interference. If the electromagnetic interference intensity does not exceed the threshold, it is verified through multiple frames of historical denoised images whether there is a continuous pixel grayscale variance abnormality in the candidate image data loss area.

7. The method for avoiding obstacles in coal mining equipment based on machine vision according to claim 1, characterized in that: The method of mapping the point cloud coordinates to the image coordinate system to obtain a three-dimensional geometric completion image comprises: A rigid body transformation matrix is ​​calculated according to the joint calibration parameters of the visual sensor and the lidar, wherein the rigid body transformation matrix includes a rotation matrix and a translation vector; point cloud coordinates of the image data missing area are obtained by the lidar; the point cloud coordinates are projected to the image coordinate system through the rigid body transformation matrix to obtain a three-dimensional geometric completion point cloud projection corresponding to the image data missing area; surface reconstruction is performed on the three-dimensional geometric completion point cloud projection to generate a triangular mesh model, and the triangular mesh model is rendered as a three-dimensional geometric completion image consistent with the current denoised image frame resolution and viewing angle.

8. The method for avoiding obstacles in coal mining equipment based on machine vision according to claim 1, characterized in that: The method for constructing a dynamic occupancy grid map of an underground environment according to the three-dimensional geometric completion image and the obstacle position confidence map comprises: The three-dimensional geometric completion image is rasterized and segmented to generate an initial grid map, and the geometric structure integrity parameters of each grid cell in the initial grid map are extracted; the confidence values ​​of the corresponding grid cells in the obstacle position confidence map are mapped to the same grid coordinate system, and the initial occupancy probability of each grid cell is calculated according to the weighted fusion of the geometric structure integrity parameters and the confidence values; The historical occupancy probability is updated in time series according to the real-time coordinate information of the mining equipment, the historical occupancy probability is exponentially decayed by the dynamic attenuation factor, and the initial occupancy probability is fused to generate the updated occupancy probability; if the updated occupancy probability exceeds the preset static obstacle judgment threshold, it is marked as an inaccessible area; otherwise it is marked as a passable area; the inaccessible area and the passable area are combined with the initial grid map to obtain a dynamic occupancy grid map.

9. The method for avoiding obstacles in coal mining equipment based on machine vision according to claim 8, characterized in that: The method for extracting the geometric structure integrity parameter of each grid cell in the initial grid map comprises: The point cloud data of the three-dimensional geometric completion image in the grid unit is statistically filtered to obtain the number of valid point clouds, and the ratio of the number of valid point clouds to the volume of the grid unit is calculated as the point cloud density; the normal vector distribution of the point cloud in the grid unit is analyzed for consistency, and the covariance matrix eigenvalue of the normal vector is calculated by principal component analysis. If the ratio of the maximum eigenvalue to the minimum eigenvalue of the covariance matrix eigenvalue is lower than a preset characteristic threshold, the surface continuity is determined to be strong, otherwise it is determined to be a surface break; the point cloud density is normalized to obtain a density integrity factor, and the strong surface continuity and the surface break are converted into a surface continuity coefficient. The geometric structure integrity parameter is obtained by weighted calculation based on the density integrity factor and the surface continuity coefficient.

10. The machine vision-based obstacle avoidance system for coal mining equipment is characterized by: The system includes: a data analysis module, an obstacle position confidence map acquisition module, a three-dimensional geometric completion image acquisition module, and an obstacle avoidance planning management module, and each module is sequentially connected in communication; The data analysis module is used to obtain the electromagnetic interference intensity distribution data generated by the operation of underground electromechanical equipment in real time, and synchronously collect the current original image frame output by the visual sensor; extract the frequency domain characteristic parameters of the current electromagnetic noise through spectrum analysis of the electromagnetic interference intensity distribution data; The obstacle position confidence map acquisition module is used to obtain a current denoised image frame after dynamically denoising the current original image frame according to the coupling relationship between the frequency domain feature parameters and the working frequency band of the visual sensor, extract the obstacle contour feature point set from the current denoised image frame through multi-scale edge detection, and obtain the obstacle position confidence map according to the spatiotemporal consistency analysis of the obstacle contour feature point set of the previous denoised image frame adjacent in time sequence; A three-dimensional geometric completion image acquisition module is used to determine that the image data loss area is caused by electromagnetic interference when the pixel grayscale variance of a preset number of consecutive rows in the current denoised image frame is lower than a preset noise threshold, and obtain the point cloud coordinates of the image data loss area; map the point cloud coordinates to the image coordinate system to obtain a three-dimensional geometric completion image; The obstacle avoidance planning management module is used to construct a dynamic occupancy grid map of the underground environment according to the three-dimensional geometric completion image and the obstacle position confidence map, and plan the obstacle avoidance path of the mining equipment according to the dynamic occupancy grid map.

Images