Robotic dredging method, system, and apparatus based on coal slime cross-sectional sensing knowledge

Abstract

The present application relates to the technical field of coal mine underground dredging, in particular to a robot dredging method, system and equipment based on coal slime section sensing knowledge, the method of the present application first obtains infrared images and point cloud data of different exposure of the coal mine sump, the different exposure infrared images are subjected to fusion processing, noise reduction and fog removal processing, and the obtained sump enhanced image accurately locates the two-dimensional boundary of the coal slime section from the texture details; at the same time, the point cloud data is preprocessed to obtain point cloud labels, and the spatial depth and morphological characteristics of the coal slime section are analyzed from the three-dimensional structure information; then, the section two-dimensional boundary mask is mapped to the sump label point cloud, and the obtained sump fusion point cloud is processed by a semantic classification model to construct a three-dimensional model of the coal slime section, which completely presents the spatial form of the coal slime section; finally, based on the depth, slope and area information of the three-dimensional model of the coal slime section, the dredging robot path is planned, and the robot is controlled to carry out dredging work, which can improve the dredging quality and efficiency.

Description

Technical Field

[0001] This invention relates to the field of coal mine underground dredging technology, specifically to a robotic dredging method, system, and equipment based on coal sludge cross-section perception and recognition. Background Technology

[0002] Dredging underground water tanks in coal mines is a crucial step in ensuring mine safety. Silt buildup in the water tanks reduces their effective volume, weakens the drainage system's resilience, and can lead to flooding accidents. Simultaneously, coal sludge deposits corrode the tank structure, clog pipes, increase equipment failure rates, and disrupt production continuity. Dredging restores the water tank's capacity, reduces the risk of mine flooding, and extends equipment lifespan. Currently, dredging robots are commonly used for water tank dredging. Compared to manual dredging, robots can adapt to the harsh underground environment of high humidity, high dust, and low oxygen, avoiding personnel safety risks. Furthermore, dredging robots can operate continuously, improving dredging efficiency.

[0003] In existing technologies, images of water tanks are often acquired to identify coal sludge cross-sections and plan dredging paths for dredging robots. However, due to poor lighting conditions in the water tanks, turbid water, high humidity, and the susceptibility to fog, the images of the water tanks are of poor quality, lack detail, and have blurred boundaries between the coal sludge and the water. This leads to redundancy or omissions in path planning, reducing the effectiveness of robot dredging. Summary of the Invention

[0004] The purpose of this invention is to provide a robotic dredging method, system, and equipment based on coal sludge cross-section perception and recognition.

[0005] The technical solution of this invention is as follows:

[0006] A robotic sludge removal method based on coal sludge cross-section sensing and recognition includes the following operations:

[0007] S1. Acquire infrared images and point cloud data of coal mine water tanks at different exposure levels; after fusion processing, obtain infrared fused images of the water tanks; after noise reduction and defogging processing, obtain enhanced images of the water tanks; identify the coal slime cross-section in the enhanced images of the water tanks and construct a two-dimensional boundary mask for the cross-section.

[0008] The point cloud data of the coal mine water sump is preprocessed to obtain the water sump preprocessed point cloud; based on the geometric features and reflection features of the point cloud, the point cloud label of each point cloud in the water sump preprocessed point cloud is obtained and marked in the water sump preprocessed point cloud to obtain the water sump labeled point cloud.

[0009] S2. Map the two-dimensional boundary mask of the cross section onto the water tank label point cloud to obtain the water tank fusion point cloud; process the water tank fusion point cloud with a semantic classification model to obtain semantic labels; based on the semantic labels, fit the coal slime cross section in the water tank fusion point cloud to construct a three-dimensional model of the coal slime cross section.

[0010] S3. Based on the cross-sectional depth and / or slope and / or area information of the coal slime cross-section 3D model, plan the movement path of the dredging robot; based on the movement path of the dredging robot, control the robot to perform dredging.

[0011] The fusion process in S1 is as follows: infrared images with different exposures are processed by a fully convolutional neural network model trained in multiple rounds until the illumination gradient of the current round output exposure fusion image is greater than the illumination gradient threshold. The current round output exposure fusion image is then used as the water tank infrared fusion image.

[0012] The training process for the fully convolutional neural network model is as follows: Infrared images at different exposure levels form an initial infrared exposure image set; each image in the initial infrared exposure image set undergoes feature extraction and weighted fusion processing to obtain a first exposure fusion image; the first exposure fusion image and the initial infrared exposure image set form a first detail enhancement image set; each image in the first detail enhancement image set undergoes feature extraction and weighted fusion processing to obtain a second exposure fusion image; the second exposure fusion image and the first detail enhancement image set form a second detail enhancement image set; each image in the second detail enhancement image set undergoes feature extraction and weighted fusion processing to obtain a third exposure fusion image; and so on, until the loss value of the current round's exposure fusion image and the previous round's exposure fusion image is less than a loss threshold, at which point the current round's exposure fusion image is used as the water tank infrared fusion image.

[0013] The operation of constructing the two-dimensional boundary mask of the cross section in S1 is as follows: the enhanced image of the water tank is subjected to grayscale threshold segmentation to obtain the coal slime region segmentation map; the coal slime region segmentation map is processed by edge detection to obtain the initial cross section boundary; the initial cross section boundary is processed by morphology to obtain the continuous cross section boundary; the continuous cross section boundary is processed by polygon approximation to obtain the optimized cross section boundary; the optimized cross section boundary is processed by mask generation to obtain the two-dimensional boundary mask of the coal slime cross section.

[0014] The geometric features of the point cloud in S1 include normal vectors, curvature, and neighborhood point density.

[0015] The operation to obtain the fused point cloud of the water tank in S2 is as follows: the pixel coordinates of the two-dimensional boundary mask of the cross section are transformed to the camera coordinate system to obtain the boundary points of the image coordinate system; the boundary points of the image coordinate system are projected to the world coordinate system to obtain the three-dimensional boundary points of the coal slime cross section; the three-dimensional boundary points of the coal slime cross section and the label point cloud of the water tank are spatially registered and the label is corrected to obtain the aligned point cloud of the coal slime cross section; the overlapping point clouds in the aligned point cloud of the coal slime cross section are fused to obtain the fused point cloud of the water tank.

[0016] In S2, the operation to obtain semantic labels is as follows: the spatial features, geometric features, reflection features, point cloud labels, and image labels of the water tank fused point cloud are embedded and then spliced ​​to obtain point cloud feature vectors; the point cloud feature vectors are processed by training a point cloud deep learning model to obtain the semantic labels of the point cloud.

[0017] A robotic sludge removal system for sensing and recognizing coal sludge cross-sections, used to implement the aforementioned robotic sludge removal method for sensing and recognizing coal sludge cross-sections, includes:

[0018] A module for generating 2D boundary and point cloud labels for cross-sections is used to acquire infrared images and point cloud data of coal mine water sumps at different exposure levels. The infrared images at different exposure levels are fused to obtain an infrared fused image of the water sump. The infrared fused image is then denoised and defogged to obtain an enhanced image of the water sump. The coal slime cross-section in the enhanced image of the water sump is identified, and a 2D boundary mask for the cross-section is constructed. The point cloud data of the coal mine water sump is preprocessed to obtain a preprocessed point cloud of the water sump. Based on the geometric and reflection features of the point cloud, the point cloud label of each point cloud in the preprocessed point cloud of the water sump is obtained and marked in the preprocessed point cloud of the water sump to obtain a labeled point cloud of the water sump.

[0019] The coal slime cross-section 3D model generation module is used to map the cross-section 2D boundary mask onto the water tank label point cloud to obtain the water tank fusion point cloud; the water tank fusion point cloud is processed by a semantic classification model to obtain semantic labels; based on the semantic labels, the coal slime cross-section in the water tank fusion point cloud is fitted to construct the coal slime cross-section 3D model.

[0020] The dredging robot movement path generation and control module is used to plan the dredging robot's movement path based on the cross-sectional depth and / or slope and / or area information of the three-dimensional model of the coal slime cross-section; and to control the robot to perform dredging based on the dredging robot's movement path.

[0021] A robotic sludge removal device for sensing and recognizing coal sludge cross-sections includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the aforementioned robotic sludge removal method for sensing and recognizing coal sludge cross-sections.

[0022] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned robotic sludge removal method for sensing and recognizing coal sludge cross-sections.

[0023] The beneficial effects of this invention are as follows:

[0024] This invention provides a robotic dredging method based on coal slime cross-section perception and recognition. First, infrared images and point cloud data of coal mine water tanks at different exposure levels are acquired. The infrared images at different exposure levels are then fused to enhance image details. Noise reduction and defogging processes are performed to eliminate noise and fog interference, resulting in an enhanced water tank image that accurately locates the two-dimensional boundary of the coal slime cross-section based on texture details. Simultaneously, point cloud data is preprocessed to obtain point cloud labels. Spatial depth and morphological features of the coal slime cross-section are analyzed based on three-dimensional structural information, providing multimodal data support of "high-fidelity two-dimensional boundary + accurate three-dimensional label" for subsequent coal slime cross-section recognition. Then, the two-dimensional boundary mask of the cross-section is mapped onto the water tank label points. The cloud data retains the precise outline of the two-dimensional boundary while integrating the spatial information of the three-dimensional point cloud, making the boundary between coal slime and non-coal slime areas clearer in three-dimensional space. The resulting fused point cloud of the water tank, after processing by a semantic classification model, further refines the classification of coal slime and water, reducing cross-class confusion. Based on this, a three-dimensional model of the coal slime cross-section is constructed, fully presenting the spatial morphology of the coal slime cross-section and providing precise spatial constraints for subsequent path planning of the dredging robot. Finally, based on the depth, slope, and area information of the three-dimensional model of the coal slime cross-section, the path of the dredging robot is planned, allowing the path to conform to the actual siltation situation, which is used to control the robot to carry out dredging work and improve the working quality and efficiency of the dredging robot. Detailed Implementation

[0025] This embodiment provides a robotic sludge removal method based on coal slime cross-section sensing and recognition, including the following operations:

[0026] S1. Acquire infrared images and point cloud data of coal mine water tanks at different exposure levels; after fusion processing, obtain infrared fused images of the water tanks; after noise reduction and defogging processing, obtain enhanced images of the water tanks; identify the coal slime cross-section in the enhanced images of the water tanks and construct a two-dimensional boundary mask for the cross-section.

[0027] The point cloud data of the coal mine water sump is preprocessed to obtain the water sump preprocessed point cloud; based on the geometric features and reflection features of the point cloud, the point cloud label of each point cloud in the water sump preprocessed point cloud is obtained and marked in the water sump preprocessed point cloud to obtain the water sump labeled point cloud.

[0028] S2. Map the two-dimensional boundary mask of the cross section onto the water tank label point cloud to obtain the water tank fusion point cloud; process the water tank fusion point cloud with a semantic classification model to obtain semantic labels; based on the semantic labels, fit the coal slime cross section in the water tank fusion point cloud to construct a three-dimensional model of the coal slime cross section.

[0029] S3. Based on the cross-sectional depth and / or slope and / or area information of the coal slime cross-section 3D model, plan the movement path of the dredging robot; based on the movement path of the dredging robot, control the robot to perform dredging.

[0030] The specific operating steps are detailed below.

[0031] S1. Acquire infrared images and point cloud data of the coal mine water sump at different exposure levels; after fusion processing, obtain the water sump infrared fused image; after noise reduction and defogging processing, obtain the water sump enhanced image; identify the coal slime cross-section in the water sump enhanced image and construct a two-dimensional boundary mask for the cross-section; after preprocessing the point cloud data of the coal mine water sump, obtain the water sump preprocessed point cloud; based on the geometric features and reflection features of the point cloud, obtain the point cloud label of each point cloud in the water sump preprocessed point cloud and mark it in the water sump preprocessed point cloud to obtain the water sump labeled point cloud.

[0032] Due to the poor lighting conditions and turbid water in underground coal mine water sump (hereinafter referred to as coal mine water sump), and the high humidity, water vapor condensation, and dust suspension in the mine easily form fog, images obtained by ordinary cameras are difficult to accurately identify the cross-sectional boundaries of coal slime. To solve this technical problem, this embodiment selects to use an infrared camera to obtain infrared images of the coal mine water sump, which can well show the difference in infrared reflection between water and coal slime. However, a single-exposure infrared image often cannot take into account both bright and dark areas. Therefore, this embodiment further obtains infrared images of the coal mine water sump at different exposure levels. Underexposed images can retain highlight details, while overexposed images can retain shadow details, thereby improving the detail richness of the coal mine water sump images.

[0033] In addition, to improve the accuracy of coal slurry cross-section identification in coal mine water sumps, point cloud data reflecting the depth information of coal mine water sumps is also acquired to improve the anti-interference ability and spatial accuracy of coal mine water sump data. Combined with infrared images of coal mine water sumps, the geometric features of coal mine water sump data are complemented, which is conducive to accurate analysis of water and coal slurry and accurate identification of coal slurry cross-sections.

[0034] Before acquiring infrared images and point cloud data separately, the camera acquiring infrared images and the lidar acquiring point cloud data are jointly calibrated to obtain camera intrinsic parameters, extrinsic parameters, and lidar intrinsic parameters. In the calibration method, a fixed reference object (such as a metal pipe or a marker point on the lidar wall) is selected from the coal mine water sump, and synchronized images and point clouds are acquired. Extrinsic parameters are calculated using the PNP algorithm or a calibration board (such as a checkerboard 3D calibration board) to ensure that rotation and translation errors are less than their respective set thresholds. Simultaneously, to ensure time synchronization, hardware triggering (such as a GPS synchronization pulse) ensures that the time difference between the acquisition of the dredging robot's image and point cloud is less than a time difference threshold (10ms), avoiding spatial misalignment caused by equipment movement.

[0035] Next, the infrared images and point cloud data of the coal mine water tank were processed to obtain the respective coal slime cross-section identification label results. The two-dimensional boundary of the coal slime cross-section was accurately located from the texture details, and the spatial depth and morphological features of the coal slime cross-section were analyzed from the three-dimensional structural information.

[0036] The processing details for the infrared images are as follows.

[0037] First, infrared images with different exposure levels are fused together to avoid overexposure of the water surface while preserving the texture details of the coal slime, effectively improving image clarity and detail recognition, thus obtaining a fused infrared image of the water tank.

[0038] The fusion process is as follows: infrared images with different exposures are processed by a fully convolutional neural network model trained in multiple rounds until the illumination gradient of the current round output fusion image is greater than the illumination gradient threshold, ensuring the uniformity of the fusion image. The current round output fusion image is then used as the water tank infrared fusion image.

[0039] The operation of training a fully convolutional neural network model is as follows: Infrared images with different exposure levels form an initial infrared exposure image set; in the initial infrared exposure image set, each image is subjected to feature extraction (feature extraction can be achieved through channel attention mechanism and spatial attention mechanism), and then weighted fusion processing is performed to obtain a first exposure fusion image; the first exposure fusion image and the initial infrared exposure image set form a first detail enhancement image set; in the first detail enhancement image set, each image is subjected to feature extraction, and then weighted fusion processing is performed to obtain a second exposure fusion image; the second exposure fusion image and the first detail enhancement image set form a second detail enhancement image set; in the second detail enhancement image set, each image is subjected to feature extraction, and then weighted fusion processing is performed to obtain a third exposure fusion image; and so on, until the loss value (memory loss) of the current round's exposure fusion image and the previous round's exposure fusion image is less than the loss threshold (memory loss threshold), ensuring that details are not lost during the fusion process, and the current round's exposure fusion image is used as the water tank infrared fusion image.

[0040] The memory loss of the two images mentioned above can be obtained based on mean squared error, and / or structural similarity index, and / or peak signal-to-noise ratio.

[0041] Finally, the infrared fused image is subjected to noise reduction and dehazing processing. While preserving edge details, noise is suppressed, and the edge contour details of the coal slime cross section are highlighted by the dehazing process, thereby improving the overall detail quality of the image and obtaining the enhanced image of the water tank.

[0042] The noise reduction process is as follows: The infrared fusion image of the water tank is divided into blocks of fixed size, such as 8×8 pixel blocks, which do not overlap. Overlapping blocks in the edge areas are retained to avoid information loss, resulting in multiple image blocks. These multiple image blocks are then processed by block matching. For each image block, blocks with similar grayscale distributions are searched within a preset search window. Similar blocks are stacked according to their spatial positions to form a three-dimensional data array. After reducing noise energy through discrete cosine transform and threshold filtering, weighted aggregation is performed. Overlapping denoised blocks are fused according to similarity weights to eliminate block effects and avoid blurring of structural information during the filtering process. A globally consistent denoising result is output, resulting in the denoised infrared image of the water tank.

[0043] The dehazing process is as follows: For each pixel in the infrared denoised image of the water tank, the minimum value within the neighborhood of the single-channel infrared band is taken, forming a dark channel image reflecting the fog concentration distribution; the pixel with the highest brightness in the dark channel image, representing the first percentage (0.1%) of the total brightness, is selected as the average pixel value of the original infrared image, used as the atmospheric light estimate to characterize the overall brightness of the fog in the environment; based on the atmospheric light value and the dark channel image, the initial transmittance of the fog is obtained; guided filtering is used to smooth and optimize the initial transmittance, eliminating block artifacts and preserving the transmittance distribution of image edge details, resulting in optimized transmittance; based on the optimized transmittance and atmospheric light value, the atmospheric scattering model is used to inversely calculate the infrared denoised image of the water tank, removing fog interference, restoring the original grayscale distribution and detail features of the image, resulting in the enhanced image of the water tank.

[0044] Because the water in a coal mine water tank is fluid, the acquired image may contain motion blur (such as water flow jitter), which manifests as edge blurring and detail blurring, affecting the distinction between water and coal slurry. Therefore, in order to further improve image quality, this embodiment also includes motion blur processing on the water tank enhanced image after acquiring the water tank enhanced image to obtain an optimized water tank enhanced image, which is used to perform the coal slurry cross-section recognition operation.

[0045] The motion blur processing operation is as follows: Based on the gradient mean and variance of the gradient map of the enhanced water tank image, it is determined whether there are blurred regions in the enhanced water tank image; if not, no processing is performed on the enhanced water tank image, and it is directly used to perform the coal slime cross-section recognition operation; if so, the strong edge regions of the enhanced water tank image are obtained (the strong edges of the coal slime cross-section and water flow boundary in the enhanced water tank image can be extracted by the Canny operator), and an initial blur kernel is constructed; based on the initial blur, blind deconvolution iteration processing is performed on the enhanced water tank image to obtain an optimized blur kernel and an intermediate clear image; the optimized blur kernel and the enhanced water tank image are processed by non-blind deconvolution to restore the details hidden by motion blur and improve the sharpness of details, thus obtaining the optimized enhanced water tank image.

[0046] Finally, coal slime cross-sections in the enhanced (or optimized) water tank images are identified, and a two-dimensional boundary mask for the cross-sections is constructed.

[0047] The specific steps for constructing a two-dimensional boundary mask for the cross-section are as follows: Based on the difference in grayscale values ​​between coal slime and water in the infrared image, the enhanced image of the water tank is segmented using grayscale thresholding. The coal slime region is marked as 1 and labeled "coal slime"; the water region is marked as 0 and labeled "water"; and other regions are marked as 2 and labeled "other," resulting in a coal slime region segmentation map. The coal slime region segmentation map is then processed by edge detection to obtain the initial contour line of the coal slime cross-section, resulting in the initial cross-section boundary. The initial cross-section boundary is then processed by morphological processing (which can be achieved by erosion and dilation of the boundary) to obtain a continuous cross-section boundary. The continuous cross-section boundary is then processed by polygon approximation to reduce redundant vertices and retain key inflection points, transforming the curved boundary into a polygonal boundary composed of 10-20 vertices, resulting in an optimized cross-section boundary. The optimized cross-section boundary is then processed by mask generation. In the pixel coordinate system, the boundary line and the inner preset pixel width region are marked as 1, and the remaining regions are marked as 0, resulting in a two-dimensional boundary mask for the coal slime cross-section.

[0048] The details of point cloud data processing are as follows.

[0049] First, the point cloud data of the coal mine water sump is preprocessed to remove noise and distortion, resulting in a preprocessed point cloud. Preprocessing includes outlier filtering and / or distortion correction. The outlier filtering operation involves obtaining the standard deviation of the distance between each point cloud and its neighboring point clouds, and removing points with a standard deviation greater than a threshold. This removes false points caused by dust scattering or water vapor reflection, resulting in denoised point cloud data.

[0050] Next, by combining point cloud geometric features and reflection features, the point cloud label of each point cloud in the preprocessed point cloud of the water tank is obtained. The point cloud geometric features include normal vector, curvature, and neighborhood point density.

[0051] Among them, point clouds whose normal vector is less than the first angle threshold (nearly perpendicular), whose curvature is less than the first curvature threshold, whose neighborhood point density is less than the first neighborhood point density threshold, and whose reflection intensity (reflection feature) is greater than the first reflection intensity are labeled as water bodies; that is, point clouds whose normal vector is nearly perpendicular, whose water curvature is small, whose neighborhood point density is low, whose point clouds are sparse and uniform, and whose reflection intensity is high are labeled as water bodies.

[0052] For point clouds where the angle between the normal vector and the vertical line is greater than the second angle threshold (the second angle threshold is greater than the first angle threshold), and the curvature is greater than the second curvature threshold (the second curvature threshold is greater than the first curvature threshold), and the neighborhood point density is not less than the first neighborhood point density threshold, and the reflection intensity (reflection feature) is not greater than a certain threshold, the point cloud is labeled as coal slime; that is, point clouds with a large angle between the normal vector and the vertical line, a large surface curvature, high neighborhood point density, dense point cloud, and low reflection intensity are labeled as coal slime.

[0053] The remaining point cloud data is labeled "Other".

[0054] Finally, the point cloud labels are added to the water tank preprocessed point cloud to obtain the water tank labeled point cloud.

[0055] S2. Map the two-dimensional boundary mask of the cross section onto the water tank label point cloud to obtain the water tank fusion point cloud; process the water tank fusion point cloud with a semantic classification model to obtain semantic labels; based on the semantic labels, fit the coal slime cross section in the water tank fusion point cloud to construct a three-dimensional model of the coal slime cross section.

[0056] Mapping the two-dimensional boundary mask of the cross-section onto the water tank label point cloud not only preserves the accurate outline of the two-dimensional boundary but also integrates the spatial information of the three-dimensional point cloud, making the boundary between coal slime and non-coal slime areas clearer in three-dimensional space. The resulting fused point cloud of the water tank, after being processed by a semantic classification model, further refines the classification of coal slime and water, reducing cross-class confusion. Based on this, a three-dimensional model of the coal slime cross-section is fitted and constructed, fully presenting the spatial morphology of the coal slime cross-section, and providing accurate spatial constraints for subsequent path planning of the dredging robot.

[0057] First, the two-dimensional boundary mask of the cross section is mapped onto the water tank label point cloud to obtain the water tank fused point cloud.

[0058] The specific steps to obtain the water tank fusion point cloud model are as follows: Based on the camera intrinsic parameters (focal length and principal point coordinates), the pixel coordinates of the cross-section 2D boundary mask are transformed to 2D homogeneous coordinates in the camera coordinate system to obtain the image coordinate system boundary points; based on the camera extrinsic parameters (rotation matrix, translation vector), the image coordinate system boundary points are...

[0059] Projecting to the world coordinate system yields the three-dimensional boundary points of the coal slime cross-section. The three-dimensional boundary points of the coal slime cross-section and the water tank label point cloud are spatially registered (which can be achieved through the ICP algorithm) and the labels are corrected. The three-dimensional boundary points are used as dividing lines, with the inner boundary points retaining the "coal slime" label and the outer boundary points retaining the "water body" label, resulting in the coal slime cross-section aligned point cloud. The overlapping point clouds in the coal slime cross-section aligned point cloud with boundary constraints are then processed by a point cloud fusion algorithm (which can be achieved through weighted average filtering) to obtain the water tank fused point cloud.

[0060] Then, because the boundary points of the coal slime section are prone to appear, a point cloud may have a double confusion label of both water and coal slime. In order to clarify the label of the coal slime section point cloud, the water tank fused point cloud is processed by a semantic classification model to update the label and obtain the semantic label.

[0061] Specifically, the spatial features (coordinate position), geometric features, reflection features, point cloud labels, and image labels of the water reservoir fusion point cloud (preferably the point cloud corresponding to the boundary of the coal slime section in the water reservoir fusion point cloud) are embedded and then spliced ​​to obtain a point cloud feature vector. The point cloud feature vector is then processed by a point cloud deep learning model (the point cloud deep learning model includes but is not limited to PointNet++ network and Swin3D network) to update the labels and obtain semantic labels for the point cloud containing only water or coal slime.

[0062] Finally, based on semantic labels, the coal slime cross-section in the fused point cloud model of the water tank is fitted to construct a three-dimensional model of the coal slime cross-section. Specifically, the point cloud with the semantic label of coal slime in the fused point cloud of the water tank is fitted with the surface contour using the Alpha Shapes algorithm, including but not limited to, and the three-dimensional boundary candidate line is corrected to obtain the three-dimensional model of the coal slime cross-section.

[0063] S3. Based on the cross-sectional depth and / or slope and / or area information of the coal slime cross-section 3D model, plan the movement path of the dredging robot; based on the movement path of the dredging robot, control the robot to perform dredging.

[0064] Based on the depth, slope, and area information of the 3D cross-section of coal slime, the path of the dredging robot can be planned, which can make the path conform to the actual siltation situation, and is used to control the robot to carry out dredging work, thereby improving the quality and efficiency of the work.

[0065] The steps for planning the movement path of the dredging robot are as follows.

[0066] Step 1: Based on the cross-sectional depth (depth value of each point in the 3D model of the coal slime cross-section, i.e., the vertical distance of the point cloud relative to the bottom of the water tank), slope (angle between the cross-sectional surface and the horizontal plane), and / or area information (area of ​​the coal slime accumulation area divided by depth interval), form a cross-sectional parameter set containing spatial coordinates and physical attributes; based on the cross-sectional parameter set and the performance parameters of the dredging robot, construct path constraints.

[0067] The performance parameters of the dredging robot mentioned above include maximum climbing angle, operating radius, and minimum turning radius.

[0068] The above path constraints include: areas with a cross-sectional depth greater than the first depth threshold (0.5m) are designated as priority dredging areas; areas with a cross-sectional slope greater than the first slope threshold (30º) are designated as obstacle avoidance areas; and the operating radius coverage area is used as the path node spacing threshold.

[0069] Step 2: Based on path constraints, the 3D model of the coal slime cross-section is rasterized to obtain an environmental cost map. Specifically, the 3D model of the coal slime cross-section is divided into grids of a preset size, and each grid is assigned a corresponding cost based on path constraints. The environmental cost map is then processed using the A* path planning algorithm along with preset start and end coordinates (the dredging start point is the edge of the cross-section, and the end point is the deepest point of the coal slime accumulation, i.e., the maximum depth of the cross-section) to obtain the initial movement path.

[0070] Step 3: The initial movement path and preset slope smoothing constraints are processed by a path optimization algorithm (including but not limited to the Bézier curve fitting method). In the process, curve fitting is performed on the inflection points of the broken line to eliminate acute angle turning, so that the radius of curvature of the path is greater than the minimum turning radius of the robot, thereby reducing the energy consumption of the path and obtaining the movement path of the dredging robot.

[0071] Finally, based on the dredging robot's movement path, the movement path is converted into a signal and transmitted to the robot's central control equipment to control the robot's path movements during the dredging process in the water tank.

[0072] This embodiment also provides a robotic sludge removal system for sensing and recognizing coal slime cross-sections, used to implement the above-mentioned robotic sludge removal method for sensing and recognizing coal slime cross-sections, including:

[0073] A module for generating 2D boundary and point cloud labels for cross-sections is used to acquire infrared images and point cloud data of coal mine water sumps at different exposure levels. The infrared images at different exposure levels are fused to obtain an infrared fused image of the water sump. The infrared fused image is then denoised and defogged to obtain an enhanced image of the water sump. The coal slime cross-section in the enhanced image of the water sump is identified, and a 2D boundary mask for the cross-section is constructed. The point cloud data of the coal mine water sump is preprocessed to obtain a preprocessed point cloud of the water sump. Based on the geometric and reflection features of the point cloud, the point cloud label of each point cloud in the preprocessed point cloud of the water sump is obtained and marked in the preprocessed point cloud of the water sump to obtain a labeled point cloud of the water sump.

[0074] The coal slime cross-section 3D model generation module is used to map the cross-section 2D boundary mask onto the water tank label point cloud to obtain the water tank fusion point cloud; the water tank fusion point cloud is processed by a semantic classification model to obtain semantic labels; based on the semantic labels, the coal slime cross-section in the water tank fusion point cloud is fitted to construct the coal slime cross-section 3D model.

[0075] The dredging robot movement path generation and control module is used to plan the dredging robot's movement path based on the cross-sectional depth and / or slope and / or area information of the three-dimensional model of the coal slime cross-section; and to control the robot to perform dredging based on the dredging robot's movement path.

[0076] This embodiment also provides a robotic sludge removal device for sensing and recognizing coal sludge cross-sections, including a processor and a memory, wherein the processor executes the computer program stored in the memory to implement the above-mentioned robotic sludge removal method for sensing and recognizing coal sludge cross-sections.

[0077] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described robotic sludge removal method for sensing and recognizing coal sludge cross-sections.

[0078] This embodiment provides a robotic dredging method based on coal slime cross-section perception and recognition. First, infrared images and point cloud data of coal mine water tanks at different exposure levels are acquired. The infrared images at different exposure levels are then fused to enhance image details. Noise reduction and defogging processes are performed to eliminate noise and fog interference, resulting in an enhanced water tank image that accurately locates the two-dimensional boundary of the coal slime cross-section based on texture details. Simultaneously, point cloud data is preprocessed to obtain point cloud labels. Spatial depth and morphological features of the coal slime cross-section are analyzed based on three-dimensional structural information, providing multimodal data support of "high-fidelity two-dimensional boundary + accurate three-dimensional label" for subsequent coal slime cross-section recognition. Then, the two-dimensional boundary mask of the cross-section is mapped onto the water tank labels. Point clouds retain the precise outlines of two-dimensional boundaries while incorporating spatial information from three-dimensional point clouds, making the boundaries between coal slime and non-coal slime areas clearer in three-dimensional space. The resulting fused point cloud of the water tank, after processing by a semantic classification model, further refines the classification of coal slime and water, reducing cross-class confusion. Based on this, a three-dimensional model of the coal slime cross-section is fitted and constructed, fully presenting the spatial morphology of the coal slime cross-section, providing precise spatial constraints for subsequent path planning of the dredging robot. Finally, based on the depth, slope, and area information of the three-dimensional model of the coal slime cross-section, the path of the dredging robot is planned, allowing the path to conform to the actual siltation situation, which is used to control the robot to carry out dredging work, improving the work quality and efficiency of the dredging robot.

Claims

1. A robotic dewatering method for identifying a coal seam cross section, characterized by, The method comprises the following steps: S1, obtaining infrared images and point cloud data of different exposure degrees of a coal mine sump; The infrared images of different exposure degrees are fused to obtain an infrared fusion image of the sump; specifically, the infrared images of different exposure degrees are processed by a trained full convolutional neural network model for multiple rounds until the illumination gradient of the exposure fusion image output in the current round is greater than an illumination gradient threshold, and the exposure fusion image output in the current round is taken as the infrared fusion image of the sump; The operation of training the full convolutional neural network model is as follows: the infrared images of different exposure degrees form an initial infrared exposure image set; In the initial infrared exposure image set, each image is respectively subjected to feature extraction and weighted fusion processing to obtain a first exposure fusion image; the first exposure fusion image and the initial infrared exposure image set form a first detail enhancement image set; each image in the first detail enhancement image set is respectively subjected to feature extraction and weighted fusion processing to obtain a second exposure fusion image; the second exposure fusion image and the first detail enhancement image set form a second detail enhancement image set; each image in the second detail enhancement image set is respectively subjected to feature extraction and weighted fusion processing to obtain a third exposure fusion image; and the process is repeated until the loss value of the exposure fusion image in the current round and the exposure fusion image in the previous round is less than a loss threshold, and the exposure fusion image in the current round is taken as the infrared fusion image of the sump; The infrared fusion image is subjected to noise reduction and defogging processing to obtain a sump enhancement image; A coal slime section in the sump enhancement image is recognized to construct a section two-dimensional boundary mask; The point cloud data of the coal mine sump is preprocessed to obtain sump preprocessed point cloud; Based on the geometric features and reflection features of the point cloud, the point cloud labels of each point cloud in the sump preprocessed point cloud are obtained and marked in the sump preprocessed point cloud to obtain a sump labeled point cloud; S2, mapping the section two-dimensional boundary mask to the sump labeled point cloud to obtain a sump fusion point cloud; processing the sump fusion point cloud by a semantic classification model to obtain a semantic label; based on the semantic label, fitting the coal slime section in the sump fusion point cloud to construct a coal slime section three-dimensional model; S3, planning a moving path of a dredging robot based on the section depth, or / and the slope, or / and the area information of the coal slime section three-dimensional model; and controlling the robot to dredge based on the moving path of the dredging robot.

2. The robotic dewatering method of claim 1, wherein, The operation of constructing the section two-dimensional boundary mask in S1 is as follows: The sump enhancement image is subjected to gray threshold segmentation processing to obtain a coal slime region segmentation image; the coal slime region segmentation image is subjected to edge detection processing to obtain an initial section boundary; the initial section boundary is subjected to morphological processing to obtain a continuous section boundary; the continuous section boundary is subjected to polygon approximation processing to obtain an optimized section boundary; and the optimized section boundary is subjected to mask generation processing to obtain a coal slime section two-dimensional boundary mask.

3. The robotic dewatering method of claim 1, wherein, The point cloud geometric features in S1 include normal vector, curvature and neighborhood point density.

4. The robotic dewatering method of claim 1, wherein, The operation of obtaining the sump fusion point cloud in S2 is as follows: The pixel coordinates of the cross-section two-dimensional boundary mask are converted to the camera coordinate system to obtain image coordinate system boundary points; the image coordinate system boundary points are projected to the world coordinate system to obtain coal slime cross-section three-dimensional boundary points; the coal slime cross-section three-dimensional boundary points are spatially registered with the sump label point cloud, and label correction is performed to obtain coal slime cross-section alignment point cloud; overlapping point clouds in the coal slime cross-section alignment point cloud are fused to obtain a sump fusion point cloud.

5. The robotic dewatering method of claim 1, wherein, In S2, the operation of obtaining the semantic label is: The spatial features, geometric features, reflection features, point cloud labels and image labels of the point cloud in the sump fusion point cloud are respectively embedded and spliced to obtain a point cloud feature vector; The point cloud feature vector is processed by a trained point cloud deep learning model to obtain a semantic label of the point cloud.

6. A robot dewatering system for coal slime cross section recognition for implementing the coal slime cross section recognition robot dewatering method of claim 1, characterized in that, It comprises: A cross-section two-dimensional boundary and point cloud label generation module is configured to obtain infrared images and point cloud data of different exposure degrees of a coal mine sump; The infrared images of different exposure degrees are fused to obtain an infrared fusion image of the sump; the infrared fusion image is denoised and defogged to obtain an enhanced image of the sump; A coal slime cross-section in the enhanced image of the sump is identified to construct a cross-section two-dimensional boundary mask; the point cloud data of the coal mine sump is preprocessed to obtain preprocessed point cloud of the sump; Based on the geometric features and reflection features of the point cloud, the point cloud label of each point cloud in the preprocessed point cloud of the sump is obtained and marked in the preprocessed point cloud of the sump to obtain a sump label point cloud; A coal slime cross-section three-dimensional model generation module is configured to map the cross-section two-dimensional boundary mask to the sump label point cloud to obtain a sump fusion point cloud; the sump fusion point cloud is processed by a semantic classification model to obtain a semantic label; based on the semantic label, the coal slime cross-section in the sump fusion point cloud is fitted to construct a coal slime cross-section three-dimensional model; A dredging robot movement path generation and control module is configured to plan a dredging robot movement path based on the cross-section depth, or / and the slope, or / and the area information in the coal slime cross-section three-dimensional model; and control the robot to dredge based on the dredging robot movement path.

7. A robot dredging apparatus for identifying a coal seam cross section, characterised in that, It comprises a processor and a memory, wherein the processor executes a computer program stored in the memory to realize the coal slime cross-section sensing and knowledge recognition robot dredging method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It is used for storing a computer program, wherein the computer program is executed by a processor to realize the coal slime cross-section sensing and knowledge recognition robot dredging method of any one of claims 1-5.