Coal and rock fracture image recognition method and system based on deep learning and post-processing

Abstract

The present invention discloses a coal rock fracture image recognition method and system based on deep learning and post-processing. A coal rock fracture image recognition model is used to identify and classify spliced coal rock fracture images. The model includes an improved YOLOv5 target detection network, an image segmentation network, and a classifier connected in sequence. After the image segmentation network outputs a binary image, disparity depth is calculated for fractures and non-fractures in fracture images at different angles, and fractures are determined based on the disparity depth. After the fractures are determined, the image features of the binary image are digitized to extract fracture image feature parameters. The fractal dimension of the fracture is calculated, and a preliminary judgment is made based on the fractal dimension whether the fracture can be characterized. If so, the fracture image feature parameters are input into the classifier to output the type of coal rock fracture. If not, the specific fracture type is output. Identification and classification are performed by integrating disparity depth, fractal dimension, and image features, effectively reducing the problems of misidentification and missed identification.

Description

Technical Field

[0001] The present invention belongs to the technical field of coal rock fracture identification, and specifically relates to a coal rock fracture image recognition method and system based on deep learning and post-processing. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] The distribution and structural evolution of coal fractures are crucial to the safe mining and efficient utilization of coal resources. Fracture development can lead to anisotropic coal seam permeability, affecting the gas flow characteristics of coal seams, and thus impacting gas extraction and the production efficiency of coalbed methane wells. Furthermore, the development of coal fractures can reduce the load-bearing capacity of the coal rock mass, leading to reduced stability and potentially causing coal mine accidents. Rapidly locating, identifying, and classifying coal fractures is crucial for roadway support and gas extraction.

[0004] With the rapid development of deep learning and digital image processing technologies, image recognition and classification algorithms have been widely used for the identification and classification of coal and rock fractures. However, existing image recognition algorithms suffer from low recognition accuracy when faced with low light levels, noise, blur, and complex fracture types found in coal mines. This is particularly true when dealing with complex and diverse coal and rock fractures, which are prone to misidentification and omission. Furthermore, most existing target recognition models utilize a single algorithm, resulting in low precision and accuracy in identifying multiple fracture types. Furthermore, the need to accurately detect fractures in real time during coal mining places higher demands on image preprocessing and recognition algorithms. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a coal rock fracture image recognition method and system based on deep learning and post-processing, which combines the target detection algorithm with the image segmentation algorithm, and performs post-processing after the fracture image is segmented. It integrates multiple information, namely parallax depth, fractal dimension and image features to perform fracture identification and classification, which can effectively reduce the problems of misidentification and missed identification, and improve the accuracy of fracture identification and classification in complex environments.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] In a first aspect, the present invention provides a method for identifying coal and rock fracture images based on deep learning and post-processing, comprising:

[0008] Acquire coal rock fracture images at different angles and perform preprocessing to obtain preprocessed coal rock fracture images at different angles;

[0009] The pre-processed coal rock fracture images at different angles are stitched together to obtain a stitched coal rock fracture image, and the stitched coal rock fracture image is input into a trained coal rock fracture image recognition model for fracture identification and classification to obtain a fracture classification result; the coal rock fracture image recognition model includes an improved YOLOv5 target detection network, an image segmentation network, and a classifier connected in sequence;

[0010] Specifically, the image processing steps in the coal rock fracture image recognition model are as follows: input the spliced coal rock fracture image into the improved YOLOv5 target detection network for fracture boundary positioning, and output the fracture annotation image; input the fracture annotation image into the image segmentation network for image segmentation, and output the segmented binary image; perform disparity depth calculation on the fractures and non-fractures in the pre-processed coal rock fracture images at different angles, and determine the fractures based on the disparity depth; after determining the fractures, perform image feature digitization on the binary image, and extract fracture image feature parameters; calculate the fractal dimension of the fracture based on the binary image, and preliminarily judge whether the fracture can be characterized based on the fractal dimension. If so, input the fracture image feature parameters into the classifier, and output the type of coal rock fracture; otherwise, output the specific type of fracture.

[0011] As a further technical solution, the preprocessing process of the coal rock fracture image is specifically as follows:

[0012] Adjust the brightness, contrast and grayscale of the image;

[0013] Enhance image details and perform the first global noise reduction;

[0014] Perform threshold segmentation on the image;

[0015] Use Gaussian filtering to perform secondary noise reduction on cracks and stubborn noise points in the image;

[0016] Finally, the piecewise linear transformation method is used to enhance the contrast of the denoised coal rock fracture image to highlight the fracture characteristics.

[0017] A further technical solution is that the improved YOLOv5 target detection network introduces a deformable convolutional network in its feature extraction network, and adopts a weighted bidirectional feature pyramid network to replace the feature pyramid network in its feature fusion network.

[0018] A further technical solution is that in the variable convolution operator of the deformable convolutional network, by adding a learnable offset parameter to each element in the convolution kernel, the original fixed convolution kernel has the ability to adapt to the shape of the object. The formula is expressed as follows:

[0019]

[0020] in, Indicates the location The response on the feature map, Represents the target position, It is represented as the position index of an element in the convolution kernel, R is represented as the area covered by the convolution kernel, W is represented as the weight, and X is represented as the input feature map. Indicates the offset.

[0021] A further technical solution is to calculate the disparity depth of cracks and non-cracks in the pre-processed coal rock crack images at different angles. Specifically, matching feature points are found in the coal rock crack images at different angles, the disparity on the image plane is calculated for the matching feature points, and the disparity depth of the feature points is calculated using the disparity, internal parameters of the camera, and external parameters. The formula is as follows:

[0022]

[0023] in, Expressed as parallax depth, Expressed as the baseline distance, Expressed as focal length, Expressed as parallax.

[0024] A further technical solution is to calculate the fractal dimension using a box counting method, specifically: select boxes of different sizes to cover the crack network of the binary image, plot the number of boxes and the corresponding box side lengths in a double logarithmic coordinate system, perform linear fitting on the plotted points, and obtain the fractal dimension of the crack.

[0025] A further technical solution is to preliminarily judge whether the crack can be characterized based on the fractal dimension:

[0026] When the fractal dimension is greater than the set threshold, it is judged as no, the specific type of crack is output, and the structural features of the specific type of crack are extracted;

[0027] When the fractal dimension is within the set range, it is judged as yes, and the fracture image feature parameters are input into the classifier, which outputs the type of coal rock fracture and extracts the structural features of the corresponding type.

[0028] In a second aspect, the present invention provides a coal rock fracture image recognition system based on deep learning and post-processing, comprising:

[0029] An image acquisition module is configured to: acquire coal rock fracture images at different angles and perform preprocessing to obtain preprocessed coal rock fracture images at different angles;

[0030] A fracture identification and classification module is configured to: stitch pre-processed coal rock fracture images from different angles to obtain a stitched coal rock fracture image, and input the stitched coal rock fracture image into a trained coal rock fracture image recognition model to identify and classify fractures, thereby obtaining a fracture classification result; the coal rock fracture image recognition model includes an improved YOLOv5 target detection network, an image segmentation network, and a classifier connected in sequence;

[0031] Specifically, the image processing steps in the coal rock fracture image recognition model are as follows: input the spliced coal rock fracture image into the improved YOLOv5 target detection network for fracture boundary positioning, and output the fracture annotation image; input the fracture annotation image into the image segmentation network for image segmentation, and output the segmented binary image; perform disparity depth calculation on the fractures and non-fractures in the pre-processed coal rock fracture images at different angles, and determine the fractures based on the disparity depth; after determining the fractures, perform image feature digitization on the binary image, and extract fracture image feature parameters; calculate the fractal dimension of the fracture based on the binary image, and preliminarily judge whether the fracture can be characterized based on the fractal dimension. If so, input the fracture image feature parameters into the classifier, and output the type of coal rock fracture; otherwise, output the specific type of fracture.

[0032] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the coal rock fracture image recognition method based on deep learning and post-processing as described in the first aspect.

[0033] In a fourth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the coal rock fracture image recognition method based on deep learning and post-processing as described in the first aspect are implemented.

[0034] One or more of the above technical solutions have the following beneficial effects:

[0035] By acquiring coal rock fracture images from different angles and stitching them together, the present invention can comprehensively present the geometric morphology and distribution of fractures, enhance the fracture processing capability of the coal rock fracture image recognition model, and thus improve the accuracy of fracture identification and classification. An improved YOLOv5 target detection network is introduced into the coal rock fracture image recognition model for fracture boundary positioning, and then combined with an image segmentation network to accurately segment the fracture area, which helps to accurately extract the boundaries and regional features of the fractures and improve the accuracy of fracture classification.

[0036] Before classifying the identified cracks, the present invention designs two innovative judgment steps. First, the crack depth is obtained by calculating the parallax depth, and the parallax depth between the crack and the non-crack part is used to determine whether the identified object is a crack, rather than other objects such as stains. Secondly, after it is determined to be a crack, the fractal dimension of the crack is used to analyze the complexity of the crack, and the representability of the crack is judged, effectively distinguishing complex cracks from simple cracks, avoiding unnecessary further analysis of overly complex cracks, and inputting the crack image feature parameters that can be represented into the classifier for classification, further improving the accuracy and reliability of the classifier.

[0037] The coal rock fracture image recognition model in the present invention integrates the advantages of multiple deep learning algorithms and combines the target detection algorithm with the image segmentation algorithm. The target detection algorithm can effectively locate the boundaries of coal rock fractures and improve the existing YOLOv5 target detection network to provide preliminary positioning results for subsequent fine segmentation; image segmentation is good at capturing the details of the fracture edge, and the SegNet image segmentation algorithm is used to complete more refined coal rock fracture extraction tasks, thereby improving the accuracy of fracture classification.

[0038] The real-time wireless monitoring system for coal and rock fractures based on the coal and rock fracture image recognition model has the characteristics of high accuracy, good robustness and fast calculation speed, which helps to realize the intelligent identification, recording and sorting of coal and rock fractures and improve production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0040] Figure 1 This is a flow chart of a method for identifying coal rock fracture images according to an embodiment of the present invention;

[0041] Figure 2 is the linear crack and its related parameters in the embodiment of the present invention;

[0042] Figure 3 1 is a diagram showing the structure of the target detection network and image segmentation network of the model in an embodiment of the present invention;

[0043] Figure 4 3 is a structural diagram of a real-time wireless monitoring system for coal and rock fractures based on a coal and rock fracture image recognition model in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0046] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment discloses a coal rock fracture image recognition method based on deep learning and post-processing, which includes the following steps:

[0049] S1: Acquire coal rock fracture images at different angles and perform preprocessing to obtain preprocessed coal rock fracture images at different angles.

[0050] In this embodiment, two cameras are used to capture images of the coal and rock, capturing two images of the same object from different angles. The two cameras are placed on the same plane in front of the coal and rock being photographed, capturing images from different angles. The angle difference between the camera optical axes is preferably between 25° and 35°.

[0051] A coal rock fracture image database containing over 300 images was established. Due to insufficient underground lighting, the coal and rock masses themselves are dark in color, and the fracture types are complex and varied. Furthermore, the images lack complete information on fracture morphology and occurrence. Therefore, specialized personnel are required to identify coal rock images and select images with typical fracture characteristics for training. This ensures the accuracy of the coal rock fracture recognition model in distinguishing fracture features through image analysis. After the images are screened, they undergo preprocessing.

[0052] The specific preprocessing process of coal rock fracture images is as follows:

[0053] S101: Adjust the brightness, contrast and grayscale of the image to increase the image brightness and preliminarily remove useless color information from the image to make the cracks more obvious;

[0054] S102: Enhances image details and improves image resolution through Topaz Gigapixel AI;

[0055] S103: Use Topaz DeNoise AI software based on convolutional neural network principles to perform global noise reduction on the image, reducing noise and improving crack clarity;

[0056] S104: Perform threshold segmentation on the image to filter out all cracks and a small amount of stubborn noise points. The threshold segmentation principle formula is as follows:

[0057] (1)

[0058] in, It's a pixel The adaptive threshold, is the local neighborhood window, C is the constant adjustment value, and are the height and width of the image, is the coordinate of a pixel on the image, is the grayscale value.

[0059] S105: Perform secondary noise reduction on cracks and stubborn noise points in the image using Gaussian filtering;

[0060] S106: The crack image after noise reduction tends to be smooth, so the piecewise linear transformation method is finally used to enhance the contrast of the coal rock crack image after noise reduction to highlight the characteristics of the crack area. The piecewise linear transformation formula is as follows:

[0061] (2)

[0062] in, is the pixel value of the original image, is the pixel value of the image after piecewise linear transformation, a and b are the pixel nodes of the original image, c and d are the pixel nodes of the image after piecewise linear transformation, 、 are the maximum pixel values of the original image and the image after piecewise linear transformation, respectively. All preprocessed images are annotated with Labelimg for cracks.

[0063] S2: The pre-processed coal rock fracture images at different angles are stitched together to obtain a stitched coal rock fracture image, which is then input into a trained coal rock fracture image recognition model for fracture identification and classification to obtain a fracture identification result.

[0064] In this example, two images of the same coal rock captured from different angles are spliced together to produce a spliced coal rock fracture image. This spliced coal rock fracture image is then fed into a coal rock fracture image recognition model for fracture identification and classification. The coal rock fracture image recognition model comprises a target detection network, an image segmentation network, and a classifier, all connected in sequence. The target detection network is a modified YOLOv5 target detection network, and the image segmentation network is a SegNet network. This indicates that the coal rock fracture image recognition model framework integrates target detection and image segmentation algorithms.

[0065] S201: Input the spliced coal rock fracture image into the improved YOLOv5 target detection network to locate the fracture boundary and output a fracture annotation image; that is, the detected fracture boundary box is superimposed on the spliced original image, thereby marking the specific location and range of the fracture.

[0066] The boundaries of coal rock fractures are located based on the improved YOLOv5 target detection network. The feature extraction effect of coal rock fractures is enhanced by reducing the segmentation targets, providing preliminary positioning results for subsequent fine segmentation.

[0067] The YOLOv5 object detection network consists of a sequentially connected input layer, a feature extraction network (Backbone), a feature fusion network (Neck), and a detection head (YOLO Head). The input layer feeds fractured coal rock images with fracture development characteristics into the image recognition model; the feature extraction network (Backbone) utilizes the Focus and C3 architectures. Coal rock fracture images typically feature dense coal rock, complex backgrounds, irregular shapes, and small objects. To address the task of recognizing coal rock fracture images, the YOLOv5 network was modified to produce an improved YOLOv5 object detection network. Specifically:

[0068] 1) In the original YOLOv5 model, the feature extraction network, Backbone, primarily consists of traditional convolutional layers, batch normalization layers, and activation functions. During feature extraction, because coal and rock fractures typically vary in size and contour, a fixed-size convolution kernel struggles to accurately extract the object's boundary information, thus hindering the model's ability to extract fracture features. Therefore, a deformable convolutional network (DCN) was introduced in Backbone. In the variable convolution operator, a learnable offset parameter, called offset, is added to each element in the convolution kernel, allowing the fixed convolution kernel to adapt to the object's shape. The calculation formula is as follows:

[0069] (3)

[0070] in, Indicates the location The response on the feature map, Represents the target position, It is represented as the position index of an element in the convolution kernel, R is represented as the area covered by the convolution kernel, W is represented as the weight, and X is represented as the input feature map. Represents the offset. Since the position of variable convolution sampling is irregular, the offset is generally a decimal, so formula (4) is implemented using bilinear interpolation as follows:

[0071] (4)

[0072] Among them, X represents the input feature map, represents any position in the area corresponding to the deformable convolution, It is a feature map The pixel value corresponding to a sampling point is Represents a two-dimensional bilinear interpolation kernel.

[0073] 2) For crack identification in dense, complex coal rock backgrounds, the key challenge is how to effectively combine, process, and fuse multi-scale features. Coal rock cracks are small, typically occupying less than 10% of the original image. The original YOLOv5 feature fusion network (Neck) uses an FPN+PAN structure to sample feature images of different sizes and process them to the same size. However, different input features have different resolutions, so when performing upsampling and downsampling and tensor concatenation, the weights of the output fused features vary. This requires optimizing the YOLOv5 network architecture and introducing learnable weights to distinguish the importance of different input features, thereby enhancing the learning ability for small objects.

[0074] The weighted bidirectional feature pyramid network (BiFPN) is used to replace the feature pyramid network FPN to adapt to the backbone network of small target detection, and efficient weighted feature fusion is achieved by using bidirectional cross connection and fast normalization. The fast normalization fusion is shown in formula (5):

[0075] (5)

[0076] in, It is represented as the output feature after fusion, and E represents a very small additional value (to keep numerical stability), Represented as all feature maps involved in fusion, Indicates the input A feature map. , represents the total number of feature maps, Represents the weight of the input feature, and the activation function makes , the weight is generally in the range of 0 to 1, and the calculation formula of a feature layer of BiFPN is shown in formula (6) and formula (7):

[0077] (6)

[0078] (7)

[0079] in, Indicates the top-down path The intermediate features of the layer, Indicates bottom-up The output features of the layer, Represented as the top-down path The input features of the layer, Resize() means adjusting the spatial size of the feature map, Represented as the top-down path The input features of the layer, is a very small additional value used to avoid the denominator being 0. In the feature fusion stage, the model uses depthwise separable convolution and adds batch normalization and activation function after each convolution to fuse multi-scale features and improve computational efficiency.

[0080] 3) In target detection, the boundary loss function used by the YOLOv5 network in the detection head is IoU, which has limitations in identifying irregularly shaped coal and rock fractures. The improved YOLOv5 target detection network uses Inner-CIoU loss to judge the target boundary. During the model training process, when dealing with high IoU coal and rock fractures, more focus is placed on fine-tuning the position of the prediction box to improve positioning accuracy. When dealing with low IoU coal and rock fractures, a larger-scale auxiliary box is used to effectively cover the target area and improve the model's generalization ability. In addition, Inner-CIoU focuses on the overlapping part inside the bounding box, improving the accuracy of bounding box regression in target detection, and can improve the detection accuracy and efficiency of occluded or overlapping coal and rock targets. The Inner-CIoU formula is as follows:

[0081] (8)

[0082] in, for loss function, for value, Represents the intersection-over-union ratio, ranging from 0 to 1.

[0083] It should be noted that due to the above improvements, the network's coal and rock fracture image positioning capability can be enhanced with a small increase in the amount of calculation.

[0084] For images where coal fractures are detected, the improved YOLOv5 object detection network outputs the fracture bounding box (i.e., the fracture annotation image). The image is cropped before proceeding to the next step. The bounding box includes the offset of the center point relative to the upper left corner of the grid cell, as well as its width and height. For images where no coal fractures are detected, the number of images where no fractures were detected is calculated and stored.

[0085] S202: Inputting the crack-annotated image into an image segmentation network for image segmentation, and outputting a segmented binary image;

[0086] After being detected and cropped by the object detection network, the semantic information of coal rock fracture images is relatively simple, making it difficult to capture edge details of the fractures. This lacks the high-level semantics of the image, making further research on the fracture structure impossible. Therefore, the SegNet image segmentation algorithm is used to achieve more refined coal rock fracture extraction. SegNet is a fully convolutional neural network based on a convolutional neural network (CNN). Compared to other image segmentation networks, convolutional neural networks offer higher accuracy and flexibility.

[0087] The image segmentation network, or SegNet, consists of an encoder and a decoder. The encoder is based on the VGG-16 network and contains several convolutional layers and max pooling layers. Each convolutional layer performs a two-dimensional convolution operation to extract local features of the image, which are used to extract deep features of the input coal and rock images. The decoder restores the spatial resolution of the image through a gradual upsampling operation and combines it with the feature map output by the encoder to achieve pixel-by-pixel segmentation. The specific steps are:

[0088] 1) Input the object detection network to locate the cropped image.

[0089] 2) The encoder extracts deep features of the coal rock image through convolution and pooling operations, gradually reducing the image size to capture key information such as cracks and interfaces in the coal rock. The convolution layer is used to extract local features of the coal rock image. The convolution operation formula is as follows:

[0090] (9)

[0091] in, Denote as input image, Represented as the convolution kernel, is the integration variable, Represents convolution.

[0092] The maximum pooling layer is used to reduce the spatial dimension of the feature map and retain important features. The size of the pooling window is 2×2, and the formula for the maximum pooling operation is:

[0093] (10)

[0094] in, is the output after pooling, is the pixel value in the input feature map.

[0095] 3) The decoder uses nonlinear upsampling to restore the spatial resolution of the image. SegNet records the pooling index and uses it to achieve accurate unpooling during the decoding phase. The formula is as follows:

[0096] (11)

[0097] in, is the pixel value after upsampling, is the maximum value retained by the pooling operation in the input feature map, is the maximum position index recorded during maximum pooling, Represented as the coordinates of an element in the input feature map.

[0098] 4) Output is a segmented binary image, where each pixel is labeled as a crack and background. This allows for the final determination of whether the coal rock has cracks and the precise location and structure of the cracks.

[0099] S203: performing parallax depth calculation on cracks and non-cracks in the pre-processed coal rock crack images at different angles, and determining cracks based on the parallax depths; after determining the cracks, performing image feature digitization on the binarized images, and extracting crack image feature parameters;

[0100] S20301: performing parallax depth calculation on the fracture and non-fracture parts in the pre-processed coal rock fracture images at different angles.

[0101] Specifically, matching feature points are found in images from two perspectives (i.e., coal rock fracture images from different angles). For this pair of matching feature points, their lateral distance (i.e., parallax) on the image plane is calculated. The parallax depth of the feature points is calculated using the parallax, the camera's internal parameters (focal length, principal point, etc.), and the external parameters (camera position and orientation). The formula is as follows:

[0102] (12)

[0103] in, is the parallax depth, is the baseline distance (i.e. the distance between the optical centers of the two cameras), is the focal length, The difference in parallax depth between the crack and the coal rock surface is the crack depth.

[0104] The present invention can obtain the crack depth by calculating the parallax depth, so that more crack features can be obtained and recorded by post-processing the crack image; it can also detect the accuracy of crack recognition. The parallax depth of the crack part and the non-crack part (i.e., the coal rock surface) are different. Through this step, it can be determined that the identified crack is not other objects such as stains, and the crack can be classified after it is determined.

[0105] S20302: Using a gray-level co-occurrence matrix to digitize the image features of the binary image, and extracting crack image feature parameters.

[0106] The gray-level co-occurrence matrix is used to digitize the image features of the crack high-level semantics (i.e., binary image) after image segmentation processing, which is used as the input data type for crack classification.

[0107] The gray-level co-occurrence matrix is the combination probability between the data gray levels The matrix composed of .

[0108] in, represents the grayscale direction, represents the grayscale level, Represents the row number of the gray-level co-occurrence matrix, Represents the column number of the gray-level co-occurrence matrix. Probability The gray level is point As the starting point, at a distance of , direction is When the gray value is point probability. The calculation formula is as follows:

[0109] (13)

[0110] in, As the starting point, is the point where gray value appears, Choose from 4 directions: 0°, 45°, 90°, and 135°. For distance, .

[0111] Through the above steps, the crack image characteristic parameters, namely contrast, homogeneity and entropy, are extracted.

[0112] S204: Calculate the fractal dimension of the crack based on the binary image, and preliminarily determine whether the crack can be characterized based on the fractal dimension. If so, input the characteristic parameters of the crack image into the classifier and output the type of coal rock crack. Otherwise, output the specific type of crack.

[0113] The box counting method is used to calculate the fractal dimension of the crack. Boxes of different sizes are selected to cover the crack network of the binary image. The number of boxes and the corresponding box side lengths are plotted in a double logarithmic coordinate system. The plotted points are linearly fitted to obtain the fractal dimension of the crack.

[0114] The specific steps of the box counting method are as follows: select a series of "boxes" of different sizes and overlay them on the fracture network. For each grid size, count how many grids contain at least part of the fracture. Plot the number of boxes and the corresponding box side lengths in a double logarithmic coordinate system. Perform a linear fit on the plotted points, and the slope of the resulting straight line is the fractal dimension of the fracture. The fractal dimension is a non-integer quantity that describes how a fractal object fills in space. For the fracture network in the image, the fractal dimension is usually between 1 and 2, and its value reflects the density and complexity of the fracture network.

[0115] Based on the obtained fractal dimension, it is judged whether the crack can be characterized, specifically:

[0116] When the fractal dimension of the crack is greater than the set threshold (the threshold in this embodiment is set to 1.5), the crack network is very dense, there are many connections between the cracks, and they cover a larger space. The crack is classified as a broken zone. At this time, further characterization is more difficult, and the judgment is no, and the specific type of the crack, namely the broken zone, is output, and its key structural features are extracted for the broken zone. In this embodiment, for the broken zone, the position of the broken area is determined according to the position of the pixel points of the broken area on the coal rock image. In order to accurately describe the degree of brokenness, the structural feature of the broken rate is defined. The broken rate refers to the ratio of the number of broken pixels in the broken area to the total number of pixels in the area. The breakage rate can be expressed as:

[0117] (14)

[0118] Among them, B represents the breakage rate, is the total number of image pixels, and D is the number of fractured pixels in the fractured area.

[0119] If the fracture dimension is within a set range (in this example, the range is set to [1, 1.5], that is, greater than 1 and less than 1.5), further characterization is possible, and the judgment is yes. The fracture image feature parameters are then input into a classifier, which outputs the type of coal rock fracture. The classifier uses a supervised learning model. The input is the fracture image feature parameters, namely contrast, homogeneity, and entropy parameters. The classifier outputs the coal rock fracture type, which includes two types: curved fractures and linear fractures, and extracts the structural features of the corresponding type. The classifier can use a commonly used supervised machine learning classification model, which is not specifically limited here.

[0120] The coal rock fracture image recognition model is trained in advance using labeled data and applied to fracture identification and classification tasks.

[0121] For the classified curved and linear cracks, their key structural features are extracted. Specifically, for linear cracks, their position, length, and apparent inclination angle are calculated. Figure 2is a parameter related to linear cracks. The position of linear cracks can be calculated by the following formula:

[0122] (15)

[0123] (16)

[0124] in, L 1 and L 2 represent the starting and ending positions of the crack, H is the length of coal rock, and are the starting and ending coordinates of the linear crack, Indicates the number of vertical pixels of the coal rock image.

[0125] The apparent inclination angle can be calculated by the following formula:

[0126] (17)

[0127] in, is the coal rock width, and Respectively Dot and The horizontal coordinate of the point.

[0128] True length of linear cracks It can be expressed as:

[0129] (18)

[0130] in, L 1 and L 2 represent the starting and ending positions of the crack, is the apparent inclination angle.

[0131] For curved cracks, calculate their length and curvature (i.e., overall arc). The length can be estimated using the following formula:

[0132] (19)

[0133] in, is the slope of a point on the curve, and are the starting and ending coordinates of the crack, respectively.

[0134] The overall curvature of the curved crack It can be calculated by integrating the local radians, as follows:

[0135] (20)

[0136] in, is the overall arc, is the slope of a point on the curve, and are the starting and ending coordinates of the crack, respectively.

[0137] The present invention determines whether a crack can be characterized based on its fractal dimension, and can make more accurate classifications under different levels of crack complexity, effectively distinguishing complex cracks from simple cracks, and avoiding unnecessary further analysis of overly complex cracks. It can also reduce the amount of calculation required by the classifier when classifying cracks, and improve the processing efficiency of the model. In particular, in the processing of large-scale coal and rock crack images, this judgment and simplification method can effectively save computing resources and time.

[0138] The above S202 and S203 post-process the crack image after positioning and annotation by the improved YOLOv5 target detection network, further determine the identified cracks, and classify them to obtain the corresponding crack types; for the corresponding crack types, their key structural features are extracted and stored.

[0139] The number of images with no detected fractures, as well as the fracture types and structural features in images with detected fractures, are stored in a local file system, such as a CSV or JSON file. To ensure the security and reliability of coal and rock fracture data, a data backup mechanism is established. This storage mechanism provides important data support for subsequent fracture identification, analysis, and model training.

[0140] Example 2

[0141] This embodiment provides a coal and rock fracture image recognition system based on deep learning and post-processing, including:

[0142] An image acquisition module is configured to: acquire coal rock fracture images at different angles and perform preprocessing to obtain preprocessed coal rock fracture images at different angles;

[0143] A fracture identification and classification module is configured to: stitch pre-processed coal rock fracture images from different angles to obtain a stitched coal rock fracture image, and input the stitched coal rock fracture image into a trained coal rock fracture image recognition model to identify and classify fractures, thereby obtaining a fracture classification result; the coal rock fracture image recognition model includes an improved YOLOv5 target detection network, an image segmentation network, and a classifier connected in sequence;

[0144] Specifically, the image processing steps in the coal rock fracture image recognition model are as follows: input the spliced coal rock fracture image into the improved YOLOv5 target detection network for fracture boundary positioning, and output the fracture annotation image; input the fracture annotation image into the image segmentation network for image segmentation, and output the segmented binary image; perform disparity depth calculation on the fractures and non-fractures in the pre-processed coal rock fracture images at different angles, and determine the fractures based on the disparity depth; after determining the fractures, perform image feature digitization on the binary image, and extract fracture image feature parameters; calculate the fractal dimension of the fracture based on the binary image, and preliminarily judge whether the fracture can be characterized based on the fractal dimension. If so, input the fracture image feature parameters into the classifier, and output the type of coal rock fracture; otherwise, output the specific type of fracture.

[0145] like Figure 4 As shown in the figure, a real-time wireless monitoring system for coal and rock fractures is established based on the above-mentioned coal and rock fracture image recognition system. First, a coal and rock fracture image recognition model is trained on a server. The trained model is then deployed to the Nvidia Jetson Xavier development board. A camera is used to capture real-time on-site images and transmit them wirelessly to the Jetson Xavier. The Jetson Xavier uses the trained coal and rock fracture image recognition model to detect whether the images contain fractured coal and rock. It then classifies the identified fractures and sends the classification results to the embedded STM32 terminal via a serial port for fracture data storage and sorting.

[0146] Example 3

[0147] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of embodiment 1 when executing the program.

[0148] Example 4

[0149] The purpose of this embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, which performs the steps of the method of embodiment 1 when executed by a processor.

[0150] The steps involved in the apparatuses of Examples 3 and 4 above correspond to those of Method Example 1. For detailed implementation, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any of the methods of the present invention.

[0151] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0152] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

[0153] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A coal rock fracture image recognition method based on deep learning and post-processing, characterized in that: include: Acquire coal rock fracture images at different angles and perform preprocessing to obtain preprocessed coal rock fracture images at different angles; The pre-processed coal rock fracture images at different angles are stitched together to obtain a stitched coal rock fracture image, and the stitched coal rock fracture image is input into a trained coal rock fracture image recognition model for fracture identification and classification to obtain a fracture classification result; the coal rock fracture image recognition model includes an improved YOLOv5 target detection network, an image segmentation network, and a classifier connected in sequence; Specifically, the image processing steps in the coal rock fracture image recognition model are as follows: the spliced coal rock fracture image is input into the improved YOLOv5 target detection network to locate the fracture boundary and output the fracture annotation image; Inputting the crack annotated image into an image segmentation network for image segmentation, and outputting a segmented binary image; performing disparity depth calculation on cracks and non-cracks in the pre-processed coal rock crack images at different angles, and determining the cracks based on the disparity depth; After the cracks are determined, the image features of the binary image are digitized to extract the crack image feature parameters; Calculating the fractal dimension of the crack based on the binary image, preliminarily determining whether the crack can be characterized based on the fractal dimension, and if so, inputting the characteristic parameters of the crack image into a classifier to output the type of coal rock crack, and if not, outputting the specific type of crack as a broken zone; The preprocessing process of the coal rock fracture image is specifically as follows: Adjust the brightness, contrast and grayscale of the image; Enhance image details and perform the first global noise reduction; Perform threshold segmentation on the image; Use Gaussian filtering to perform secondary noise reduction on cracks and stubborn noise points in the image; Finally, the piecewise linear transformation method is used to enhance the contrast of the coal rock fracture image after noise reduction to highlight the fracture features. The improved YOLOv5 target detection network introduces a deformable convolutional network in its feature extraction network, replaces the feature pyramid network with a weighted bidirectional feature pyramid network in its feature fusion network, and uses the Inner-CIoU loss to judge the target boundary. Specifically: In the variable convolution operator, by adding a learnable offset parameter to each element in the convolution kernel, the original fixed convolution kernel has the ability to adapt to the shape of the object. The calculation formula is as follows: in, Indicates the location The response on the feature map, Represents the target position, It is represented as the position index of an element in the convolution kernel, R is represented as the area covered by the convolution kernel, W is represented as the weight, and X is represented as the input feature map. Represents the offset; due to the irregularity of the position of variable convolution sampling, the offset is generally a decimal, so bilinear interpolation is used as follows: Among them, X represents the input feature map, represents any position in the area corresponding to the deformable convolution, It is a feature map The pixel value corresponding to a sampling point is Represents a two-dimensional bilinear interpolation kernel; The weighted bidirectional feature pyramid network is used to replace the feature pyramid network FPN to adapt to the backbone network of small target detection, and bidirectional cross connection and fast normalization are used to achieve efficient weighted feature fusion; the fast normalization fusion is shown in the following formula: in, Expressed as the output feature after fusion, E represents a very small additional value, Represented as all feature maps involved in fusion, Indicates the input feature maps, , represents the total number of feature maps, Represents the weight of the input feature; The Inner-CIoU loss formula is as follows: in, for loss function, for value, represents the intersection and union ratio; The image segmentation network is a SegNet network, which consists of two parts: an encoder and a decoder. The encoder is based on the VGG-16 network and contains several convolutional layers and maximum pooling layers. Each convolutional layer performs a two-dimensional convolution operation to extract local features of the image, which is used to extract the depth features of the input coal and rock images. The decoder restores the spatial resolution of the image through a step-by-step upsampling operation and combines the feature map output by the encoder to achieve pixel-by-pixel segmentation. The specific steps are as follows: 1) Input the cropped image to the target detection network; 2) The encoder extracts deep features of the coal rock image through convolution and pooling operations, gradually reducing the image size to capture key information about cracks and interfaces in the coal rock. The convolution layer is used to extract local features of the coal rock image. The convolution operation formula is as follows: in, Denote as input image, Represented as the convolution kernel, is the integration variable, represents convolution; The maximum pooling layer is used to reduce the spatial dimension of the feature map and retain important features; the size of the pooling window is 2×2, and the formula for the maximum pooling operation is: in, is the output after pooling, is the pixel value in the input feature map; 3) The decoder uses nonlinear upsampling to restore the spatial resolution of the image; SegNet records the pooling index and uses these indexes to achieve accurate de-pooling in the decoding stage. The formula is as follows: ； in, is the pixel value after upsampling, is the maximum value retained by the pooling operation in the input feature map, is the maximum position index recorded during maximum pooling, Represented as the coordinates of an element in the input feature map; 4) Output the segmented binary image, where each pixel is labeled as crack and background; The disparity depth calculation is performed on the cracks and non-cracks in the pre-processed coal rock crack images at different angles. Specifically, matching feature points are found in the coal rock crack images at different angles, the disparity on the image plane is calculated for the matching feature points, and the disparity depth of the feature points is calculated using the disparity, the internal parameters of the camera, and the external parameters. The formula is expressed as follows: in, Expressed as parallax depth, Expressed as the baseline distance, Expressed as focal length, Expressed as parallax; After the cracks are determined, the image features of the binary image are digitized to extract the crack image feature parameters, specifically: the image features of the binary image are digitized using a gray level co-occurrence matrix to extract the crack image feature parameters; The gray-level co-occurrence matrix is the combination probability between the data gray levels The matrix composed of ; in, represents the grayscale direction, represents the grayscale level, Represents the row number of the gray-level co-occurrence matrix, Represents the column number of the gray-level co-occurrence matrix; probability The gray level is point As the starting point, at a distance of , direction is When the gray value is point probability; The calculation formula is as follows: in, As the starting point, is the point where the gray value appears, Choose from 4 directions: 0°, 45°, 90°, and 135°. For distance, ; Through the above steps, the crack image characteristic parameters, namely contrast, homogeneity and entropy, are extracted; Based on the fractal dimension, it is preliminarily judged whether the crack can be characterized specifically as follows: When the fractal dimension is greater than the set threshold, it is judged as no, the specific type of fracture is output as a broken zone, and the structural characteristics of the specific type of fracture as a broken zone are extracted; the broken rate is defined, and the breakage rate can be expressed as: Among them, B represents the breakage rate, is the total number of image pixels, and D represents the number of fractured pixels in the fractured area; When the fractal dimension is within the set range, it is judged as yes, and the fracture image feature parameters are input into the classifier. The classifier outputs the type of coal rock fracture, including curved fracture and linear fracture, and extracts the structural features of the corresponding type; For linear cracks, calculate their position, length and apparent inclination. The position of linear cracks can be calculated by the following formula: in, L 1 and L 2 represent the starting and ending positions of the crack, H is the length of coal rock, and are the starting and ending coordinates of the linear crack, Indicates the number of vertical pixels of the coal rock image; The apparent inclination angle can be calculated by the following formula: in, is the coal rock width, and The abscissas represent the end and starting points of the linear crack respectively; True length of linear cracks It can be expressed as: in, L 1 and L 2 represent the starting and ending positions of the crack, is the apparent inclination angle; For curved cracks, calculate their length and curvature. The length can be estimated by the following formula: in, is the slope of a point on the curve, and are the starting and ending coordinates of the crack respectively; The overall curvature of the curved crack It can be calculated by integrating the local radians, as follows: in, is the overall arc, is the slope of a point on the curve, and are the starting and ending coordinates of the crack, respectively.

2. The coal rock fracture image recognition method based on deep learning and post-processing according to claim 1, characterized in that: The fractal dimension is calculated using a box counting method, specifically: boxes of different sizes are selected to cover the crack network of the binary image, the number of boxes and the corresponding box side lengths are plotted in a double logarithmic coordinate system, and a linear fit is performed on the plotted points to obtain the fractal dimension of the crack.

3. A coal-rock fracture image recognition system based on deep learning and post-processing, which adopts the coal-rock fracture image recognition method based on deep learning and post-processing according to any one of claims 1-2, characterized in that: include: An image acquisition module is configured to: acquire coal rock fracture images at different angles and perform preprocessing to obtain preprocessed coal rock fracture images at different angles; A fracture identification and classification module is configured to: stitch pre-processed coal rock fracture images from different angles to obtain a stitched coal rock fracture image, and input the stitched coal rock fracture image into a trained coal rock fracture image recognition model to identify and classify fractures, thereby obtaining a fracture classification result; the coal rock fracture image recognition model includes an improved YOLOv5 target detection network, an image segmentation network, and a classifier connected in sequence; Specifically, the image processing steps in the coal rock fracture image recognition model are as follows: the spliced coal rock fracture image is input into the improved YOLOv5 target detection network to locate the fracture boundary and output the fracture annotation image; Inputting the crack annotated image into an image segmentation network for image segmentation, and outputting a segmented binary image; performing disparity depth calculation on cracks and non-cracks in the pre-processed coal rock crack images at different angles, and determining the cracks based on the disparity depth; After the cracks are determined, the image features of the binary image are digitized to extract the crack image feature parameters; Calculating the fractal dimension of the crack based on the binary image, preliminarily determining whether the crack can be characterized based on the fractal dimension, and if so, inputting the characteristic parameters of the crack image into a classifier to output the type of coal rock crack, and if not, outputting the specific type of crack as a broken zone; The preprocessing process of the coal rock fracture image is specifically as follows: Adjust the brightness, contrast and grayscale of the image; Enhance image details and perform the first global noise reduction; Perform threshold segmentation on the image; Use Gaussian filtering to perform secondary noise reduction on cracks and stubborn noise points in the image; Finally, the piecewise linear transformation method is used to enhance the contrast of the coal rock fracture image after noise reduction to highlight the fracture features. The improved YOLOv5 target detection network introduces a deformable convolutional network into its feature extraction network, adopts a weighted bidirectional feature pyramid network to replace the feature pyramid network in its feature fusion network, and uses the Inner-CIoU loss to judge the target boundary; The disparity depth calculation is performed on the cracks and non-cracks in the pre-processed coal rock crack images at different angles. Specifically, matching feature points are found in the coal rock crack images at different angles, the disparity on the image plane is calculated for the matching feature points, and the disparity depth of the feature points is calculated using the disparity, the internal parameters of the camera, and the external parameters. The formula is expressed as follows: in, Expressed as parallax depth, Expressed as the baseline distance, Expressed as focal length, Expressed as parallax; Based on the fractal dimension, it is preliminarily judged whether the crack can be characterized specifically as follows: When the fractal dimension is greater than the set threshold, it is judged as no, the specific type of fracture is output as a broken zone, and the structural features of the specific type of fracture as a broken zone are extracted; When the fractal dimension is within the set range, it is judged as yes, and the fracture image feature parameters are input into the classifier, which outputs the type of coal rock fracture and extracts the structural features of the corresponding type.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the coal rock fracture image recognition method based on deep learning and post-processing as described in any one of claims 1-2 are implemented.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the coal rock fracture image recognition method based on deep learning and post-processing are implemented as described in any one of claims 1-2.

Images