Coal body damage type image recognition method combining texture features for decision fusion

Abstract

The present application provides a coal body damage type image recognition method combining texture features for decision fusion, uses a high-definition explosion-proof camera to shoot and obtain coal body images underground, carries out image processing and marking, constructs a coal body dataset, and creates a corresponding texture feature dataset according to the coal body original image; a classification model is constructed based on decision fusion, and the coal body original image dataset and the corresponding texture feature dataset are used to train the classification model; the coal body original image to be identified and the corresponding texture feature image are classified and tested in the trained classification model, and the damage type classification result of the coal body image is obtained. The present application can significantly improve the recognition efficiency of the underground coal body damage type by machine recognition of the damaged coal image, and provide a reference for the prediction of coal and gas outburst accidents, greatly improving the intelligence and safety of coal mine production.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a coal body damage type image recognition method combining texture features for decision fusion. BACKGROUND

[0002] In order to study the relationship between the damage degree of coal body and gas outburst, the tectonic coal is qualitatively classified in terms of the damage degree of coal body. Generally, the tectonic coal is classified into five types, i.e. non-damaged coal, damaged coal, strongly damaged coal, pulverized coal and full pulverized coal, according to the damage degree of coal, which is widely accepted and adopted by researchers. The adsorption capacity of different damage types of coal to gas is different, and the physical structure and mechanical properties are quite different, so the identification of the damage type of coal is of great significance to the prevention and control of coal and rock dynamic disasters and the assessment of gas outburst danger in complex tectonic mining areas. Since the entire environment of coal is very harsh and complex in the entire process of coal mining, transportation, storage, utilization and processing and recycling, how to identify and accurately distinguish the damage type of tectonic coal in such an environment is very important to ensure the safety of coal production, promote the rational use of resources and environmental protection.

[0003] At present, the most commonly used method for identifying the damage type of coal body is still the method of empirical macroscopic physical observation and identification, which plays a key role in daily production inspection, but lacks objective and quantitative classification standards. Therefore, this classification method mainly based on manual observation has the disadvantages of low efficiency, unstable accuracy, complex influencing factors and the like, which may cause certain safety hazards and has low reliability.

[0004] Therefore, in the face of the rapid development of machine intelligence recognition and classification technology for images today, through the study of automatic identification of fine-grained images of coal body damage type, the identification efficiency and accuracy can be greatly improved, the occurrence of coal and gas outburst accidents can be effectively prevented, and the safety of coal production can be ensured. SUMMARY

[0005] Therefore, the present application provides a coal body damage type image recognition method combining texture features for decision fusion, which aims to improve the identification efficiency of the damage type of coal body in the underground mine, increase the accuracy of identification, and effectively prevent the occurrence of coal and gas outburst accidents, thereby improving the safety of coal production.

[0006] To this end, the present application provides the following technical solutions:

[0007] The present application provides a coal body damage type image recognition method combining texture features for decision fusion, which comprises:

[0008] The coal body image is obtained by shooting with a high-definition explosion-proof camera in a well, image processing and marking are performed, a coal body dataset is constructed, and a corresponding texture feature image dataset is created according to the coal body original image;

[0009] A classification model based on decision fusion is constructed, and the model is trained based on the coal body image dataset and the corresponding texture feature image dataset; wherein the classification model includes two sub-classifiers with the same structure but different parameters; each sub-classifier includes a note suggestion sub-network and a classification sub-network with two different functions; the original image and the texture feature image are input into the classification sub-network in the two classifiers for learning and training, and the output features are compressed into a five-dimensional vector after bilinear pooling; the average pooling layer in the two classification sub-networks is connected, a three-layer convolutional neural network for adaptive learning is constructed as the note suggestion sub-network for learning, different channel weights are obtained, the learned weights are multiplied by the feature outputs of the original image and the texture feature image respectively, and then summed to obtain the fusion classification result.

[0010] The coal body original image and the corresponding texture feature image to be identified are input into the trained classification model based on decision fusion to obtain the classification result of the coal body original image.

[0011] Further, the sub-classifier is improved and constructed based on a bilinear neural network B-CNN and a ResNet-18.

[0012] Further, the corresponding texture feature dataset is created according to the coal body original image, including:

[0013] According to the method of extracting texture features from a gray level co-occurrence matrix, typical feature parameters of five types of images are extracted.

[0014] The difference degrees of images of each category on different features are compared, the feature with the largest difference is selected as the overall feature of the image, and the texture feature dataset of the image is created.

[0015] Further, the typical feature parameters include mean, variance, contrast, dissimilarity, homogeneity, ASM, entropy and correlation.

[0016] Further, the difference degrees of images of each category on different features are determined by a structural similarity index value.

[0017] Further, the feature with the largest difference is the mean value.

[0018] The application further provides a computer readable storage medium, which stores a computer instruction set, and the computer instruction set is executed by a processor to realize the coal body damage type image recognition method of decision fusion combined with texture features.

[0019] Advantages and positive effects of the application: the application collects a large number of real underground coal body damage images of five different types and performs image preprocessing, constructs a damage coal image data set containing five types of damage coal, namely non-damaged coal, damaged coal, strongly damaged coal, crushed coal and full-powder coal, according to the idea of multi-modal feature fusion, constructs a sub-classification model based on a bilinear neural network B-CNN and simultaneously constructs a double-line attention sub-model based on deep learning to realize optimization of original image and texture image classification results, so as to comprehensively learn and predict. The application constructs a damage coal recognition model based on deep learning technology, can internalize the model in actual production equipment of a coal mine, realizes real-time recognition of underground damage coal, achieves efficient and accurate recognition effect, and has the ability to solve actual problems underground. Compared with the traditional image recognition technology, the application is more efficient and has stronger self-adaptability. On the other hand, the application can solve the current underground recognition mainly by artificial recognition, overcome the disadvantages of unstable artificial recognition effect and low efficiency, and the like.

[0020] In addition, the application uses deep learning technology to construct an image recognition model of coal body damage type, can be installed in a portable safety device underground, prevents coal and gas outburst accidents, contributes to automation and safety of the underground coal mining process, helps to realize whole-process monitoring of working quality, improves the informatization degree of the coal mine and speeds up the process of intelligent production and safe production. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0022] Figure 1 The total process of the coal body damage type image recognition method combined with texture features for decision fusion in the embodiment of the application;

[0023] Figure 2 The five types of damage coal image example diagram after preprocessing in the embodiment of the application;

[0024] Figure 3 The texture feature image calculation process of all images in the embodiment of the application;

[0025] Figure 4A comparison chart for the difference of five types of images in eight characteristics in the embodiment of the present application is shown in the following table:

[0026] Figure 5 The sub-classifier model of the improved B-CNN in the embodiment of the present application is shown in the following table:

[0027] Figure 6 The classification model based on decision fusion in the embodiment of the present application is shown in the following table: DETAILED DESCRIPTION

[0028] In order to enable the person skilled in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should be within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] The present application is based on the original data set CoalsData processed from the collected various types of coal body images, and proposes a bilinear neural network model combined with traditional texture features, which is used for identifying the coal body damage type, and the overall process is as shown in the following figure: Figure 1 The specific steps include the following:

[0031] S1, use a high-definition explosion-proof camera to shoot and obtain coal body images underground, perform image processing and marking, construct a coal body data set, and create a corresponding texture feature image data set according to the coal body original image.

[0032] In a specific implementation, the underground damaged coal images are obtained by shooting with a high-definition anti-explosion camera carried by a person in a mine. When shooting, the images with an original pixel of 3000*4000 are shot from different directions and the requirements of high definition, no occlusion and obvious features are achieved as much as possible. However, due to poor lighting and complex environment in the mine, the original images still inevitably have problems such as serious light reflection, pipeline and protective net occlusion, and problems such as mixed coal body damage types and unbalanced data samples in natural conditions. Therefore, further processing and adjustment are performed on all images. Under the premise of ensuring the quality and quantity of the images, image cutting, image brightness adjustment, contrast processing and color processing are used for image correction, and image cutting, flipping and symmetry are used for data enhancement and expansion, and the image category labels are manually marked. An example of the five types of preprocessed damaged coal images is shown in Figure 2

[0033] According to the method of extracting texture features based on the gray level co-occurrence matrix (GLCM), the typical feature parameters of the five types of images, i.e., non-damaged coal, damaged coal, severely damaged coal, pulverized coal and fully pulverized coal, are extracted. The differences of the images of the five categories in different features are compared, the most different feature is selected as the overall feature of the image, and the texture feature data set of the image is created.

[0034] The main purpose of creating the texture feature data set of the image is to obtain the texture feature with the maximum overall difference and difference balance by calculating the eight texture features of the five types of images. The feature is selected as the texture feature of all images, and the texture images of all images under the feature are output. The GLCM method, one of the most classic texture extraction methods, is used to generate a gray level co-occurrence matrix to obtain the regularity of the gray value and spatial variation in the image and to describe the texture features of the image. Learning the typical texture features of the five types of coal body damage type images, according to the correlation of different structural features with the five types, the most different structural feature is found to be learned, which will make it easier to learn the difference of the five types of damage types. As shown in Figure 3

[0035] S11, converting each coal body original image in the coal body original image data set into a gray image;

[0036] S12, extracting pictures by type from the gray image data set;

[0037] The coal body original image includes five types, and N pictures are extracted for each type (1-1, 1-2, …, 1-N; 2-1, 2-2, …, 2-N; …; 5-1, 5-2, …, 5-N).

[0038] S13, calculating the gray level co-occurrence matrix for each image of each type extracted.​​

[0039] S14、According to the gray level co-occurrence matrix, the texture feature parameter corresponding to the feature picture of each image is calculated.

[0040] The texture feature parameters are 8, and are mean, variance, contrast, dissimilarity, homogeneity, ASM, entropy and correlation. The calculated pictures are 5xNx8.

[0041] S15, A certain feature texture picture of the N pictures of each type of sample is compared with the blank picture, and N SSIM (Structural Similarity Index Measurement) values are obtained, and then the average value is obtained.

[0042] Thus, the average SSIM values of the five types of sample pictures on the eight feature values and the blank image can be finally obtained, that is, 5x8 SSIM values.

[0043] S16, A line chart is established, and the difference significance and dispersion balance of the five types of pictures on the eight feature values are observed, and the results can be obtained through the performance of data on the image and the variance.

[0044] S17, Select one of the texture feature parameters with the best difference and balance performance as the texture extraction feature parameter of all images.

[0045] The selection method of the difference feature is that N images are randomly sampled from each of the five types of coal body damage type images, and the image outputs of these images on the above eight feature values are calculated. Then, an evaluation coordinate system with a blank image as a reference is established for each type of image sampling and each feature picture, such as Figure 4 The SSIM values of each type of sample image and the blank image are calculated, then the SSIM values of the same type of sample pictures on the eight feature values are averaged, and finally the SSIM values of the five types of images on the eight feature values are compared, and the feature with the largest difference is selected as the texture feature extraction feature. The mean value is used as the extraction feature in the embodiment of the application.

[0046] S18, The corresponding texture feature picture of the selected texture feature parameter is calculated for the gray scale picture of all images.

[0047] S19, The texture feature data set of all images is constructed.

[0048] S2, construct a classification model based on decision fusion, and train the classification model based on decision fusion based on the coal body original map data set and the texture feature data set corresponding to the coal body original map.

[0049] Considering the destructiveness and environmental darkness of the coal body image, how to quickly extract significant feature information from it is one of the key points of the fine-grained recognition algorithm. Under the same neural network structure, the feature dimension extracted is unchanged, and the key lies in how much difference information the neural network extracts. The more the difference information, the higher the accuracy of the recognition and classification result. The idea of decision fusion is to learn the importance and correlation of the features of the multi-branch model, which can give different attention to different channels in image recognition, and optimally combine the multi-angle typical features.

[0050] The decision fusion classification model constructed in the embodiment of the application can be divided into two sub-classifiers, and the structure of each sub-classifier is the same, but the parameters are different. As shown in Figure 5 Two same subnetwork models based on bilinear neural network (B-CNN) and ResNet-18 are constructed, and two classifiers are obtained by training the original map and the texture feature map to extract two levels of features.

[0051] As shown in Figure 6 Each subnetwork can be divided into an attention suggestion subnetwork and a classification subnetwork with two different functional subnetwork structures. The original image and the texture feature image are input into the improved bilinear classification subnetwork for learning and training, and the output is a five-dimensional feature vector. At the same time, the output feature maps of the average pooling layers of the left and right connected two classification subnetworks are connected, and two self-adaptive learning three-layer convolutional neural networks are constructed as the attention suggestion subnetwork to learn the above spliced output features, and the output is two five-dimensional channel weights. The learned weights are multiplied by the feature outputs of the original map and the texture feature map respectively, and then summed to obtain the fusion classification result of the original map and the texture map.

[0052] S3, input the coal body original map to be recognized and the corresponding texture feature map into the trained classification model based on decision fusion to obtain the classification result of the coal body original map.

[0053] The input flow of the data stream is as follows. First, the feature map of the last convolutional layer in the bilinear neural network is input into the average pooling layer (AVG Pooling), and HxW pixels in each channel are compressed into 1 real number. The output of this layer is X c The outputs X origin and X feature of the original map and the texture map are connected on the left and right, and the size is 512x1x1.

[0054]

[0055] Then the connection can be expressed as X in , the feature map size is 1024x1x1.

[0056] X in = X origin + X feature ;

[0057] X in is sent into two structure same attention suggestion sub-networks, trained by convolution neural network containing 2 fully connected layers (FC); finally, normalized by excitation layer Sigmoid layer into a set of real numbers between 0-1 W origin and W feature output, representing the importance of each channel. The channel weight is multiplied by the feature map classification output of the bilinear neural network on the corresponding channel, expressed as:

[0058] F1=F origin x W origin ;

[0059] F2=F feature x W feature ;

[0060] According to the above criteria and experiments, the final fusion feature classifier output is:

[0061] F M =F1+F2;

[0062] Where F M is the final fusion recognition classification vector, F1 and F2 are the original image classification function output vector and texture image classification function output vector respectively, and W origin and W feature are the weights assigned to the original image and texture image respectively.

[0063] In the image recognition problem of coal body, considering the destructiveness of coal body and the dimness of environment, how to quickly extract significant feature information from it is one of the key points of fine-grained recognition algorithm. Therefore, the B-CNN model with advantages in texture recognition field is improved by using Resnet-18 as the basic network, so that the learning of image information can be increased while the influence of image pixel position information is ignored. On the other hand, since the feature dimension extracted under the same neural network structure is certain, the key of recognition effect is the amount of difference information extracted by the neural network. The more the difference information is, the higher the recognition classification accuracy is, therefore, the difference of structural similarity of five kinds of images on different texture feature parameters is compared, the texture feature parameter with the maximum variance and the maximum difference is selected as the extraction feature of the texture graph, and the texture image data set is constructed. In the experiment, after the comparison of the texture, the mean feature value is selected as the texture image feature, and the missing information of the original graph is supplemented and enhanced. Finally, based on the decision fusion method of multi-modal feature fusion and the strong correlation of texture features in coal body image, a model based on deep learning algorithm is proposed, a three-layer attention suggestion neural network is constructed, the classification features of the original image and the texture feature image are adaptively fused, and the optimal fusion classification result is obtained.

[0064] The present application collects five kinds of underground images of damaged coal, constructs a coal body damage type data set through image processing and picture marking, constructs a coal body damage type recognition model by using deep learning technology, and experiments on the image recognition of coal body damage type, and the recognition accuracy reaches 99.52%, which has obvious advantages compared with the prior art. At the same time, the model of the present application is experimented on the public data set, and the recognition accuracy of the KTH-TIPS data set and the UIUC data set related to texture reaches 98.5% and 99.5% respectively, which confirms that the model of the present application has strong generalization ability and high stability on the data set related to texture.

[0065] The embodiment of the present application also discloses a computer readable storage medium, which stores a computer instruction set, and the computer instruction set is executed by a processor to realize a coal body damage type image recognition method for decision fusion combined with texture features as provided in any of the above embodiments.

[0066] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other manners. Among them, the above-described device embodiments are only illustrative, for example, the division of the units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0067] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on multiple units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0068] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0069] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0070] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for coal seam damage type image recognition by combining texture features for decision fusion, characterized in that, The method includes: High-definition explosion-proof cameras are used to capture coal body images underground. Image processing and labeling are then performed to construct a coal body dataset. A corresponding texture feature image dataset is created based on the original coal body images. This includes: extracting texture features using a gray-level co-occurrence matrix method to extract typical feature parameters for five image categories; comparing the differences between different texture feature parameters for each category of images, selecting the feature with the greatest difference as the overall feature of the image, and creating a texture feature dataset; the feature with the greatest difference is the mean value. A classification model based on decision fusion is constructed and trained on a coal image dataset and a corresponding texture feature image dataset. The classification model includes two sub-classifiers with identical structures but different parameters. Each sub-classifier comprises two sub-network structures with different functions: an attention proposal sub-network and a classification sub-network. The original image and texture feature image are input into the classification sub-networks of the two classifiers for learning and training. The output features are compressed into a five-dimensional vector after bilinear pooling. Simultaneously, the average pooling layers in the two classification sub-networks are connected to construct an adaptive learning three-layer convolutional neural network as the attention proposal sub-network to learn different channel weights. The learned weights are multiplied by the feature outputs of the original image and the texture feature image, and then summed to obtain the fused classification result. The classification sub-networks are constructed based on bilinear neural networks B-CNN and ResNet-18. The original image of the coal body to be identified and the corresponding texture feature map are input into the trained decision fusion-based classification model to obtain the classification result of the original coal body image.

2. The method for coal seam damage type image recognition by combining texture features for decision fusion according to claim 1, characterized in that, The typical characteristic parameters used for comparison include eight types: mean, variance, contrast, dissimilarity, homogeneity, ASM, entropy, and correlation.

3. The method for coal seam damage type image recognition based on texture feature fusion for decision-making according to claim 1, characterized in that, The degree of difference between images of different categories on different features is determined by the structural similarity index value.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer instruction set, which, when executed by a processor, implements a coal body damage type image recognition method as described in any one of claims 1 to 3, which combines texture features for decision fusion.

Images