Coal mine drilling image splicing method based on deep learning

Abstract

The invention discloses a coal mine drilling image splicing method based on deep learning. The method comprises the following steps: framing and preprocessing a drilling video; performing polar coordinate transformation to cylindrically unfold the image of the inner wall of the drill hole into a two-dimensional image; splicing of multiple frames of images is completed based on a deep learning-based feature extraction algorithm, and a complete inner wall panorama is generated; and meanwhile, real-time display of unfolded and spliced images and user interaction operation are supported. The method can efficiently and accurately reconstruct the drill hole inner wall expansion drawing, and provides a visual and reliable scheme for coal mine geologic structure analysis, safety evaluation and mining scheme optimization.

Description

Technical Field

[0001] The present invention belongs to the field of coal mine image processing and computer vision, and relates to a processing and visualization technology for coal mine borehole inner wall images, and specifically to a coal mine borehole image splicing method based on deep learning. Background Art

[0002] Coal mine borehole imaging technology utilizes specialized borehole imaging equipment to capture high-definition images or videos within underground coal mine environments, visually demonstrating the geological structure within the coal rock mass. These images provide crucial information for coal mine exploration, mining planning, and safety management. However, due to the complexity of underground coal mine environments, existing image processing technologies face the following challenges:

[0003] Drilling videos are difficult to analyze: The video information collected by drilling cameras is usually complex, and it is difficult to obtain accurate information about the inner wall of the borehole directly from the video. Especially when there are complex geological structures such as cracks and joints, the details in the video are easily ignored or blurred.

[0004] Insufficient stitching accuracy: Existing borehole image expansion and stitching technologies often struggle to maintain geometric accuracy and visual continuity when processing complex geological structures. This is especially true when the borehole walls contain unique geological structures such as cracks and joints. This can lead to deviations in the stitching effect, affecting the accuracy of the final visualization.

[0005] Low automation: Although some image stitching and expansion methods exist, existing technologies generally lack the ability to adapt to different borehole sizes, materials, and environmental conditions, and are unable to provide efficient and accurate automated expansion and stitching. This limits the widespread promotion and application of image processing in actual coal mining applications. Summary of the Invention

[0006] To address the shortcomings of the existing technology, the present invention aims to provide a deep learning-based coal mine borehole image stitching method. By efficiently expanding and precisely stitching borehole inner wall images, a panoramic view of the coal mine borehole is constructed. This technology can effectively enhance the visualization of the coal mine borehole inner wall, helping geological engineers more clearly and intuitively observe the geological structure within the borehole, thereby optimizing various tasks such as coal mine exploration, resource assessment, and safety monitoring. Through this invention, the automation, accuracy, and real-time performance of coal mine borehole inner wall image processing are significantly improved, providing an efficient and accurate technical solution for the coal mining industry.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A coal mine drilling image stitching method based on deep learning, comprising the following steps:

[0009] S1, drilling video framing and image preprocessing: extract key frames from the drilling video, and perform denoising, enhancement, and correction on the extracted borehole inner wall images to improve the accuracy and robustness of subsequent image processing;

[0010] S2, cylindrical expansion of the borehole inner wall image: Based on polar coordinate transformation, the borehole inner wall image processed by S1 is expanded into a two-dimensional plane to show the geometric structure of the inner wall;

[0011] S3, after the borehole inner wall image is expanded, multiple frames of expanded images are stitched together to construct a complete panoramic image: a feature point extraction algorithm based on deep learning is used to automatically learn the correspondence between feature descriptors between borehole images;

[0012] S4, real-time display and user interaction of image expansion and stitching results: real-time display of the image results after expansion and stitching, supporting user interactive operations.

[0013] The present invention also includes the following technical features:

[0014] Specifically, the S1 includes:

[0015] First, the panoramic video data of the coal mine drilling is collected by drilling camera equipment, and the continuous frame images in the panoramic video data are divided into frames to extract image frames as input data;

[0016] For the separated image frames, Gaussian filter and adaptive median filter algorithm are used to remove the noise, including sensor noise and environmental interference;

[0017] Use adaptive histogram equalization to correct local illumination of the image and enhance the contrast of the image;

[0018] By calibrating the intrinsic and extrinsic parameters of the camera, a mathematical model for distortion correction is established to eliminate barrel distortion or pincushion distortion.

[0019] Specifically, the mathematical expression of the Gaussian filter is:

[0020]

[0021] Among them, I(x,y) is the original image, I ′ (x,y) is the denoised image, and G(i,j) is the Gaussian kernel function.

[0022] Specifically, the mathematical model of the distortion correction is as follows:

[0023] x′=x(1+k1r 2 +k2r 4 +k3r 6)+2p1xy+p2(r 2 +2x 2 ) (2)

[0024] y′=y(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2y 2 )+2p2xy (3)

[0025] Among them, x, y are the undistorted image coordinates, x ′ ,y ′ are the coordinates after distortion correction, r is the distance from the point to the image center, k1, k2, k3 are radial distortion parameters, and p1, p2 are tangential distortion parameters.

[0026] Specifically, the S2 includes:

[0027] The center point of the drill hole is detected based on the center point detection model. The location information of the drill hole center point is used to calculate the radius and angle of the inner wall in each frame of the image. Assuming that the radius of the drill hole inner wall is r, the position of each pixel point is represented by the angle θ and radius r in the polar coordinate system. The pixels of the cylindrical drill hole inner wall image are converted from the polar coordinate system to the Cartesian coordinate system to generate the expanded rectangular image. The specific mapping relationship is as follows:

[0028] x=r·cos(θ) (4)

[0029] y=r·sin(θ) (5)

[0030] Where r is the radius of the inner wall, θ is the angle, which is the polar coordinate angle with the center of the circle as the origin, and x and y are the coordinates mapped to the two-dimensional plane.

[0031] Specifically, the bicubic interpolation algorithm is used to fill the missing pixels that occur during the expansion process to maintain the continuity and details of the image; the specific interpolation formula is:

[0032]

[0033] Among them, B i (x) and B j (y) is the bicubic interpolation basis function, P i,j is the value of the corresponding pixel.

[0034] Specifically, the S3 includes:

[0035] A deep learning-based feature point extraction and matching algorithm is used to extract key feature points from each frame of a multi-frame sequence of unfolded images of the inner wall of a borehole, and a correspondence between unfolded images of different frames is established. The unfolded images are trained end-to-end using a convolutional neural network to learn more complex and robust feature descriptors from a large amount of image data. The deep learning model can automatically adapt to feature representations of different scales, rotations, perspectives, and lighting changes.

[0036] Specifically, in S3, it is assumed that the feature point extracted by the deep learning model SuperPoint is p i =(x i ,y i ) and p i ′ =(x i ′ ,y i ′ ), where p i Represents the feature points in a frame of expanded image, p i ′ Represents the corresponding feature points of the expanded image in another frame; the feature point descriptors obtained by training the deep learning model can adapt to changes in lighting and geometric transformations; the goal of feature matching is to minimize the transformation matrix H corresponding to the feature points between two different frame images. The transformation matrix is ​​obtained by minimizing the matching error:

[0037]

[0038] Among them, p i and p i ′ They are the feature points or corresponding points in different frame images, and N is the number of feature points. By optimizing this error function, accurate image registration results are obtained.

[0039] A weighted average fusion algorithm is used in the stitching area to smoothly transition the stitching area. The algorithm assigns weights based on pixel distance to make the image stitching transition more natural. The weighted average formula is:

[0040]

[0041] Among them, w i is the weight of image i at pixel (x, y), I i (x,y) is the pixel value of image i, I f (x, y) is the pixel value of the spliced ​​image; weight w i Calculation is performed based on the distance or similarity between pixels, making the transition of the seam area smoother and more natural.

[0042] Specifically, in S4, the user can dynamically adjust the image viewing angle through a mouse or touch screen to view different areas of the borehole inner wall; adjust the image stitching accuracy as needed; save the stitching results and export them as a standard image format file for subsequent analysis and report generation.

[0043] Compared with the prior art, the present invention has the following technical effects:

[0044] (1) The present invention significantly improves the efficiency of coal mine drilling image processing by automating image framing, preprocessing, center point positioning and splicing, and reduces manual intervention and operation time.

[0045] (2) By utilizing the feature point descriptor extraction technology based on deep learning, the present invention ensures the stable extraction of feature points of the borehole inner wall expansion diagram, thereby ensuring the geometric accuracy and visual continuity during the splicing process and reducing the matching deviation in traditional methods.

[0046] (3) The present invention can adapt to the size, material and environmental conditions of the inner wall of different types of boreholes, provide higher adaptability and robustness, and can process image data in complex geological environments.

[0047] (4) Users can dynamically adjust the image display effect through real-time display and interactive operations to obtain the best image visualization effect and improve decision-making analysis efficiency.

[0048] (5) The precise expansion map generated by the present invention provides an important auxiliary tool for geological structure analysis, safety assessment, mining planning, etc. of coal mines, helping engineers to identify and evaluate potential risks more intuitively and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a visualization framework diagram of the interior of a coal mine borehole based on the image unfolding method.

[0050] Figure 2 Schematic diagram of the selection of inner and outer diameters for the unfolded image of the internal borehole and the effective unfolded area clipped based on the inner and outer diameters.

[0051] Figure 3 Schematic diagram of the transformation from polar coordinate system to Cartesian coordinate system during the unfolding process.

[0052] Figure 4 Schematic diagram of key point detection and matching of drilling images for front and back frames. DETAILED DESCRIPTION

[0053] Specific embodiments of the present invention are given below. It should be noted that the present invention is not limited to the following specific embodiments, and all equivalent modifications made on the basis of the technical solution of this application fall within the protection scope of the present invention.

[0054] Example 1:

[0055] This embodiment provides a coal mine drilling image stitching method based on deep learning, such as Figure 1 As shown, the following steps are included:

[0056] S1, drilling video framing and image preprocessing: extract key frames from the drilling video and perform denoising, enhancement, and correction on the extracted borehole inner wall images to improve the accuracy and robustness of subsequent image processing. Specifically, S1 includes:

[0057] First, panoramic video data of coal mine drilling is collected through drilling camera equipment. Since the video contains a large number of continuous frame images, in order to improve processing efficiency and effect, the continuous frame images in the panoramic video data are framed and extracted. Different frames per second can be extracted as input data according to user settings;

[0058] For the separated image frames, Gaussian filter and adaptive median filter algorithm are used to remove the noise, including sensor noise and environmental interference. Specifically, the mathematical representation of Gaussian filter is:

[0059]

[0060] Among them, I(x,y) is the original image, I ′ (x,y) is the denoised image, and G(i,j) is the Gaussian kernel function.

[0061] Due to the uneven lighting conditions in the coal mine drilling environment, adaptive histogram equalization (CLAHE) is used to perform local illumination correction on the image to enhance the contrast of the image and make the micro geological features such as cracks and joints more obvious;

[0062] By calibrating the camera's intrinsic and extrinsic parameters, a mathematical model for distortion correction is established to eliminate barrel distortion or pincushion distortion. Specifically, the mathematical model for distortion correction is as follows:

[0063] x′=x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 ) (2)

[0064] y′=y(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2y 2 )+2p2xy (3)

[0065] Among them, x, y are the undistorted image coordinates, x ′ ,y ′ are the coordinates after distortion correction, r is the distance from the point to the image center, k1, k2, k3 are radial distortion parameters, and p1, p2 are tangential distortion parameters.

[0066] After the image preprocessing is completed, the next key step is to expand the borehole inner wall image based on the polar coordinate transformation method, and expand the cylindrical inner wall image into a two-dimensional plane image.

[0067] S2, cylindrical expansion of the borehole inner wall image: Based on polar coordinate transformation, the borehole inner wall image processed by S1 is expanded into a two-dimensional plane to accurately present the geometric structure of the inner wall, avoiding image distortion caused by distortion or inaccurate coordinate mapping in traditional methods. Specifically, S2 includes:

[0068] Based on the center point detection model, the drilling image is first processed by the deep learning network to accurately detect the center point of the drilling hole. The model uses YOLO (or other appropriate detection algorithms) to locate the drilling center point. The model input is the pre-processed drilling key frame image (such as the image obtained in step S1), and the output is the center coordinate of the drilling hole. After the center point is detected, the position information of the drilling center point is used to calculate the radius and angle of the inner wall on each frame of the image, such as Figure 2 As shown in the figure, the inner (green) and outer (red) diameters of the expanded borehole image are selected (left), and the effective expanded area is clipped based on the inner and outer diameters (right). Assuming that the radius of the borehole inner wall is r, the position of each pixel point is represented by the angle θ and radius r in the polar coordinate system. The pixel points of the cylindrical borehole inner wall image are converted from the polar coordinate system to the Cartesian coordinate system to generate the expanded rectangular image, as shown in the figure. Figure 3 As shown; the specific mapping relationship is as follows:

[0069] x=r·cos(θ) (4)

[0070] y=r·sin(θ) (5)

[0071] Where r is the radius of the inner wall, θ is the angle, which is the polar coordinate angle with the center of the circle as the origin, and x and y are the coordinates mapped to the two-dimensional plane.

[0072] In order to ensure the accuracy of the expanded image, the bicubic interpolation algorithm is used to fill the missing pixels that occur during the expansion process to maintain the continuity and details of the image; the specific interpolation formula is:

[0073]

[0074] Among them, B i (x) and B j (y) is the bicubic interpolation basis function, Pi,j is the value of the corresponding pixel.

[0075] After the inner wall images are unfolded, multiple frames need to be stitched together to create a complete panorama. To ensure superior geometric accuracy and visual continuity in the stitched result, this paper employs deep learning-based feature point extraction and matching technology, thereby improving the robustness and adaptability of traditional stitching methods, especially in complex geological conditions and with varying lighting conditions.

[0076] S3, after the borehole inner wall image processed by S1 is expanded, multiple frames of expanded images are stitched together to construct a complete panoramic image. A deep learning-based feature point extraction algorithm is used to automatically learn the correspondence between feature descriptors between borehole images, overcoming the shortcomings of traditional feature matching methods in coping with complex environments and lighting changes. Specifically, S3 includes:

[0077] like Figure 4 As shown in the figure, first, a feature point extraction and matching algorithm based on deep learning is used to extract key feature points from each frame of a multi-frame borehole inner wall unfolded image sequence, and establish a correspondence between unfolded images of different frames; these algorithms perform end-to-end training on the unfolded images through convolutional neural networks, and can learn more complex and robust feature descriptors from a large amount of image data; the deep learning model can automatically adapt to feature representations of different scales, rotations, perspectives and illumination changes, greatly surpassing traditional feature extraction methods such as SIFT and ORB.

[0078] Assume that the feature points extracted by the deep learning model SuperPoint are p i =(x i ,y i ) and p i ′ =(x i ′ ,y i ′ ), where p i Represents the feature points in a frame of expanded image, p i ′ Represents the corresponding feature points of the expanded image in another frame; the feature point descriptors obtained by training the deep learning model can adapt to changes in lighting and geometric transformations, thereby reducing mismatching; the goal of feature matching is to minimize the transformation matrix H corresponding to the feature points between two different frame images. The transformation matrix is ​​obtained by minimizing the matching error:

[0079]

[0080] Among them, p i and p i ′are the feature points or corresponding points in different frame images, and N is the number of feature points. By optimizing this error function, accurate image registration results can be obtained.

[0081] A weighted average fusion algorithm is used in the stitching area to smoothly transition the stitching area and avoid the visual effect of obvious image seams. The algorithm assigns weights according to pixel distance to make the image stitching transition more natural. The weighted average formula is:

[0082]

[0083] Among them, w i is the weight of image i at pixel (x, y), I i (x,y) is the pixel value of image i, I f (x, y) is the pixel value of the spliced ​​image; weight w i It is usually calculated based on the distance or similarity between pixels, making the transition in the seam area smoother and more natural.

[0084] S4, real-time display and user interaction of image expansion and stitching results: The expanded and stitched image results are displayed in real time, supporting user interactive operations such as adjusting the stitching speed and accuracy. Users can dynamically adjust the image viewing angle using a mouse or touch screen to view different areas of the borehole wall. The accuracy of image stitching can be adjusted as needed, such as by selecting different interpolation algorithms or optimizing parameters. Users can choose to save the stitching results and export them to standard image format files for subsequent analysis and report generation.

[0085] Through the technical solution of the present invention, an expanded diagram of the inner wall of a borehole can be generated efficiently and accurately, providing an intuitive and reliable tool for coal mine geological structure analysis, safety assessment and mining plan optimization.

Claims

1. A coal mine drilling image stitching method based on deep learning, characterized in that: The following steps are involved: S1, drilling video framing and image preprocessing: extract key frames from the drilling video, and perform denoising, enhancement, and correction on the extracted borehole inner wall images to improve the accuracy and robustness of subsequent image processing; S2, cylindrical expansion of the borehole inner wall image: Based on polar coordinate transformation, the borehole inner wall image processed by S1 is expanded into a two-dimensional plane to show the geometric structure of the inner wall; S3, after the borehole inner wall image is expanded, multiple frames of expanded images are stitched together to construct a complete panoramic image: a feature point extraction algorithm based on deep learning is used to automatically learn the correspondence between feature descriptors between borehole images; S4, real-time display and user interaction of image expansion and stitching results: real-time display of the image results after expansion and stitching, supporting user interactive operations.

2. The coal mine drilling image stitching method based on deep learning according to claim 1, characterized in that: Said S1 comprises: First, the panoramic video data of the coal mine drilling is collected by drilling camera equipment, and the continuous frame images in the panoramic video data are divided into frames to extract image frames as input data; For the separated image frames, Gaussian filter and adaptive median filter algorithm are used to remove the noise, including sensor noise and environmental interference; Use adaptive histogram equalization to correct local illumination of the image and enhance the contrast of the image; By calibrating the intrinsic and extrinsic parameters of the camera, a mathematical model for distortion correction is established to eliminate barrel distortion or pincushion distortion.

3. The coal mine drilling image stitching method based on deep learning according to claim 2, characterized in that: The mathematical representation of the Gaussian filter is: Among them, I(x,y) is the original image, I ′ (x,y) is the denoised image, and G(i,j) is the Gaussian kernel function.

4. The coal mine drilling image stitching method based on deep learning according to claim 2, characterized in that: The mathematical model of the distortion correction is as follows: x′=x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 ) (2) <h2 style=";text-align:left;direction:ltr">y′=y(1+k1r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +k2r<h2 style=";text-align:left;direction:ltr"> 4 <h2 style=";text-align:left;direction:ltr"> +k3r<h2 style=";text-align:left;direction:ltr"> 6 <h2 style=";text-align:left;direction:ltr"> )+p1(r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +2y<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> )+2p2xy (3) Among them, x, y are the undistorted image coordinates, x ′ ,y ′ are the coordinates after distortion correction, r is the distance from the point to the image center, k1, k2, k3 are radial distortion parameters, and p1, p2 are tangential distortion parameters.

5. The coal mine drilling image stitching method based on deep learning according to claim 1, characterized in that: The S2 includes: The center point of the drill hole is detected based on the center point detection model. The location information of the drill hole center point is used to calculate the radius and angle of the inner wall in each frame of the image. Assuming that the radius of the drill hole inner wall is r, the position of each pixel point is represented by the angle θ and radius r in the polar coordinate system. The pixels of the cylindrical drill hole inner wall image are converted from the polar coordinate system to the Cartesian coordinate system to generate the expanded rectangular image. The specific mapping relationship is as follows: x=r·cos(θ) (4) y=r·sin(θ) (5) Where r is the radius of the inner wall, θ is the angle, which is the polar coordinate angle with the center of the circle as the origin, and x and y are the coordinates mapped to the two-dimensional plane.

6. The coal mine drilling image stitching method based on deep learning according to claim 5, characterized in that: The bicubic interpolation algorithm is used to fill the missing pixels that occur during the expansion process to maintain the continuity and details of the image. The specific interpolation formula is: Among them, B i (x) and B j (y) is the bicubic interpolation basis function, P i,j is the value of the corresponding pixel.

7. The coal mine drilling image stitching method based on deep learning according to claim 1, characterized in that: The S3 includes: A deep learning-based feature point extraction and matching algorithm is used to extract key feature points from each frame of a multi-frame sequence of unfolded images of the inner wall of a borehole, and a correspondence between unfolded images of different frames is established. The unfolded images are trained end-to-end using a convolutional neural network to learn more complex and robust feature descriptors from a large amount of image data. The deep learning model can automatically adapt to feature representations of different scales, rotations, perspectives, and lighting changes.

8. The coal mine drilling image stitching method based on deep learning according to claim 7, characterized in that: In S3, it is assumed that the feature points extracted by the deep learning model SuperPoint are p i =(x i ,y i ) and p i ′ =(x i ′ ,y i ′ ), where p i Represents the feature points in a frame of expanded image, p i ′ Indicates the corresponding feature points of the expanded image in another frame; The feature point descriptors obtained through training of the deep learning model can adapt to changing lighting and geometric transformations; The goal of feature matching is to minimize the transformation matrix H corresponding to the feature points between two different frame images. The transformation matrix is ​​obtained by minimizing the matching error: Among them, p i and p i ′ They are the feature points or corresponding points in different frame images, and N is the number of feature points. By optimizing this error function, accurate image registration results are obtained. A weighted average fusion algorithm is used in the stitching area to smoothly transition the stitching area. The algorithm assigns weights based on pixel distance to make the image stitching transition more natural. The weighted average formula is: Among them, w i is the weight of image i at pixel (x, y), I i (x,y) is the pixel value of image i, I f (x, y) is the pixel value of the spliced ​​image; weight w i Calculation is performed based on the distance or similarity between pixels, making the transition of the seam area smoother and more natural.

9. The coal mine drilling image stitching method based on deep learning according to claim 1, characterized in that: In said S4, the user can dynamically adjust the viewing angle of the image by using a mouse or a touch screen to view different areas of the borehole inner wall; Adjust the image stitching accuracy as needed; save the stitching results and export them to standard image format files for subsequent analysis and report generation.

Images