An image semantic segmentation method in a coal mine underground

Abstract

The application discloses a kind of coal mine underground image semantic segmentation methods, belong to computer vision field.First, the underground scene image of acquisition is preprocessed to generate dataset;Then construct the feature extraction network with ResNet-101 as skeleton, and use different scale images in each stage input to enhance the extracted features;Then construct fusion attention module to fuse features in each stage, and use global attention module to enhance global information and obtain long-range dependency relationship;Finally, the obtained features are input into the classifier to generate semantic map, and the semantic segmentation of image is completed.The calculation amount and complexity of the method are greatly reduced, the attention mechanism is used for complex scene, the semantic information of target area is highlighted, and the image segmentation effect is improved.

Description

Technical Field

[0001] The present invention relates to an image semantic segmentation method, in particular to an underground coal mine image semantic segmentation method suitable for use in underground coal mines, and belongs to the field of computer vision. Background Art

[0002] Researching the structural features and restoration methods of visual scenes in underground tunnels is crucial for analyzing the structured imagery of complex underground tunnel scenes. This study addresses the complex environments of underground coal mines, characterized by strong direct light, dim light, dust, mist, and smoke. Traditional image analysis methods include the Lucas-Kanade algorithm, matching methods, energy methods, and phase methods. In complex underground scenes, sudden changes in lighting conditions can cause brightness to fluctuate, and even small movements within a confined space can cause significant positional shifts. Traditional methods are prone to producing erroneous results in complex underground scenes. Therefore, further research is needed to analyze complex underground scenes.

[0003] For problems such as sudden changes in illumination, shadows, and position offsets encountered in the analysis of complex underground scenes, semantic analysis methods based on deep learning image segmentation theory can be very effective solutions. Deep learning image segmentation models can approximate nonlinear models with extremely high accuracy. However, research on structured scenes mostly focuses on indoor scenes of buildings, and there is no relevant work conducted in underground coal mine tunnel scenes. Therefore, considering the large aspect ratio of coal mine tunnels, and in order to ensure the accuracy and speed of segmentation, the present invention proposes a semantic analysis method based on multi-level feature fusion image segmentation theory. Former researchers have done a lot of work on image segmentation tasks in other scenes.

[0004] Patent (Qin Yinghua, Xu Yi. Glass defect detection method based on frequency domain and space combination of segmentation network [P]. Jiangsu Province: CN112686896B, 2021-07-06.) Processes the image by frequency domain combined with spatial analysis to obtain the frequency characteristics of the image, and then performs deconvolution on the processed image to extract high-dimensional features in the image as feature points, and uses a convolutional neural network to perform segmentation training on the feature points in the image to be detected to obtain the detection results. Patent (Yu Shuang, Ji Wei, Ma Kai, Zheng Yefeng. Image segmentation method, device, computer equipment and storage medium [P]. Guangdong Province: CN112598686B, 2021-06-04.) Uses a priori knowledge vector to encode the image to obtain a target feature map, and then decodes it to obtain a first segmentation map, and then reconstructs the first segmentation map according to the prior knowledge vector to obtain multiple labeled segmented images. Based on this, the target feature map is processed to obtain a second segmentation map, so that it is integrated into multiple annotation results, thereby improving the accuracy of image segmentation. Patent (Tang Yunwei, Jing Linhai, Chen Fulun, Wan Haoming, Liu Yanzhen. Fine extraction method of remote sensing image information based on surface elements [P]. Beijing: CN112434685B, 2021-05-28.) The internal spectral histogram of the statistical remote sensing image is used as the first classification feature, and the curve matching algorithm is used to supervise the classification of the remote sensing image. According to the preliminary results, the spatial correlation between adjacent image targets is extracted as the second feature. Combined with the spectral histogram and spatial correlation, the curve matching algorithm is used again to segment the image. Patent (Han Jing, Chen Xiaoyu, Li Duanyang, Zhang Quan, Teng Zhijie, Wei Chiheng, Li Yiran. A method for infrared road scene segmentation based on category prototype regression [P]. Jiangsu Province: CN112381101B, 2021-05-28.) Use category prototype regression on the data set to obtain category prototype features, cluster network deep features, make global category features tighter, amplify the differences between categories, and construct relationship matrices and attention modules accordingly to make the overall features tighter and improve the final image segmentation accuracy. Patent (He Jinlong. Image segmentation method and device [P]. Beijing: CN112101369B, 2021-02-05.) Utilize the logical relationship between two target areas in the image and the position information of their respective vertices, wherein the two target areas at least partially overlap, and according to the position information of the respective vertices of the two target areas, determine the position information of the intersection between the respective boundaries of the two target areas, and according to the position information of the intersection and the logical relationship, segment the two different objects from the image to be processed.The paper (Chen C, Deng J, N Lv. Illegal Constructions Detection in Remote Sensing Images based on Multi-scale Semantic Segmentation [C] / / 2020 IEEE International Conference on Smart Internet of Things (SmartIoT). IEEE, 2020.) uses a multi-scale parallel structure instead of the traditional multi-layer convolutional layer. Based on this, a semantic segmentation network with a new encoder-decoder structure is proposed. Conditional random fields are also used to constrain the segmentation results, resulting in higher segmentation accuracy. The paper (Zhang F, Chen Y, Li Z, et al. ACFNet: Attentional Class Feature Network for Semantic Segmentation [C] / / International Conference on Computer Vision (ICCV), IEEE, 2019.) proposes the concept of class centers, which extracts global context from a classification perspective. This class-level context describes the overall representation of each class in the image. Furthermore, an attention class feature module is proposed. This module calculates and adaptively combines different class centers for each pixel, resulting in a coarse-to-fine attention class feature segmentation network. This improves image segmentation accuracy. Summary of the Invention

[0005] Aiming at the shortcomings of the existing technology, a semantic segmentation method for coal mine underground images is provided, which has simple steps, good segmentation effect and strong robustness in describing scene features.

[0006] In order to address the deficiencies of the prior art, the present invention provides a method for semantic segmentation of coal mine images, the steps of which are as follows:

[0007] Step 1: Collect downhole images, perform annotation preprocessing on the image data, and divide the annotated preprocessed image data into training sample and test sample data sets.

[0008] Step 2: Input the training sample dataset into the feature extraction network to extract input image features. The feature extraction network includes an improved ResNet-101 network. The improved ResNet-101 network structure deletes the downsampling operations in the fourth and fifth stages of the conventional ResNet-101 network, and retains the other contents of the fourth and fifth stages.

[0009] Step 3: In the fourth stage of the improved ResNet-101 network, the feature map output by the third stage and the additional input feature map are simultaneously input through multi-scale input to output a low-level feature map; in the fifth stage, the feature map output by the fourth stage and the additional input feature map are simultaneously input through multi-scale input to output a high-level feature map; the additional input feature map is obtained by processing the input image through the residual unit, and the additional input feature map is obtained by compressing the original input image so that it has the same size as the output feature map of the previous stage;

[0010] Step 4: Build a fusion attention module after the fifth stage of the improved ResNet-101 network, use the fusion attention module to fuse the low-level feature map and the high-level feature map, and output a new feature map containing global context semantic information;

[0011] Step 5: Build a global context enhancement module after the fusion attention module to enhance the global representation of the new feature map, thereby obtaining the long-range dependency between pixels in the feature map and obtaining the final fused feature map;

[0012] Step 6: Input the final fused feature map into the pre-trained classifier to generate a semantic map. Then, the performance of the generated semantic map is tested using the test sample data set to verify the performance of the feature extraction network. If the performance meets the standard, it can be used to perform semantic segmentation on coal mine photos. If it does not meet the standard, retraining is required.

[0013] Step 7: Use the trained feature extraction network to perform semantic segmentation on the input coal mine underground image.

[0014] The specific process of step 1) is as follows:

[0015] Step 11) Use an underground explosion-proof camera to obtain a clear image.

[0016] Step 12) Manually perform semantic segmentation and annotation on the obtained image, that is, classify each pixel in the image; different regions in the image are segmented from each other, and each region is defined by semantic information;

[0017] Step 13) Randomly construct training sample sets and test sample sets from the labeled images in a ratio of 4:1.

[0018] The specific process of step 2) is as follows:

[0019] Step 21) Based on the original ResNet-101 network, the improved ResNet-101 network is divided into five stages to extract the features of the input image, thereby obtaining output feature maps of different levels;

[0020] Step 22) Each of the five stages of the improved ResNet-101 network contains multiple channels, and the importance of the information contained in each channel for semantic segmentation is also different. Therefore, a channel attention mechanism is added to each stage, and the importance of different channels is represented by assigning a 0-1 weight to each channel;

[0021] Step 23) To enrich the detail information, the downsampling operations in the fourth and fifth stages are deleted, thereby preventing the receptive fields of the feature maps in the fourth and fifth stages of the conventional ResNet-101 from gradually increasing with the convolution and downsampling process, while the detail information of small objects in the feature maps is gradually lost;

[0022] Step 24) Use dilated convolution to save the output feature maps of the fourth and fifth stages, so that the feature maps of the third, fourth, and fifth stages are the same size, which is 1 / 8 the size of the input image.

[0023] The specific process of step 3) is as follows:

[0024] Step 31) Since the receptive field gradually increases with the process of convolution and downsampling, the detail information of small targets is gradually lost. In order to obtain more detail information, multi-scale input is adopted. Basic residual units are added to the input ends of the fourth and fifth stages of the improved ResNet-101 network respectively, and the additional 1 / 8 size input image is directly input to the basic residual unit to obtain the additional input feature maps of the fourth and fifth stages. The additional input feature maps obtained in this step are subjected to one feature extraction and are low-level feature maps. In the improved ResNet-101 network, the input images of each stage except the first stage are the output feature maps of the previous stage. The inputs of the fourth and fifth stages are high-level feature maps, which contain less detail information than the low-level feature maps.

[0025] Step 32) The additional input feature maps processed by the basic residual unit in the fourth and fifth stages are fused with the normal input feature maps in the fourth and fifth stages respectively, making full use of the shallow feature maps to enrich the information of small objects in the deep feature maps;

[0026] Step 33) Use multi-scale input to enhance the feature map of 1 / 8 size of the input image, where the process of multi-scale input is: Assume that the ResNet-101 network contains L i layer convolution, then the j-th layer convolution can be defined as y j =M j (x j ), where y j is the output tensor of the jth layer, M j Contains convolution, ReLU activation function and regularization operation, the input image x of the i-th stage iThe size of (N,H j ,W j ,W j ), N represents the batch size, H i and W i Represents the height and width of the input feature map, C i is the number of channels; the output feature map F of the i-th stage i It can be expressed as:

[0027]

[0028] Step 34) I i It represents the additional input of the i-th stage, whose resolution is the same as the output tensor of the i-1-th stage, and its feature map after feature extraction is:

[0029]

[0030] Step 35) The fusion input of stage i is expressed as:

[0031]

[0032] Where, F i represents the output tensor of the i-th stage, Represents a channel splicing operation;

[0033] Step 36) The fifth stage outputs a high-level feature map x h , the fourth stage low-level feature map χ l .

[0034] The specific process of step 4) is as follows:

[0035] Step 41) Construct the fusion attention module: The fusion attention module contains two inputs, the high-level feature map output from the fifth stage of step 36) And the low-level feature map output by the fourth stage H h ×W h is the high-level feature map χ h The number of spatial locations, H l ×W l is the low-level feature map χ l The number of spatial locations of C h and C l They are high-level feature maps χ h and low-level feature maps χ l The number of channels, 1×1 convolution W θ Used to transform the low-level feature map χ l The feature conversion of in is the number of channels of the converted feature, R is a real number, εl is the low-level feature map χ l The result after feature conversion is shown in formula (4):

[0036] ε l =W θ (χ l ) (4)

[0037] Step 42) Convert the feature conversion result ε l After regularization with the softmax function, we get f(ε l );

[0038] Step 43) Use bottleneck feature transformation to process f(ε l ), obtain channel dependency, 1×1 convolution W γ1 and W γ2 Will be used for χ h The feature transformation of attention output is obtained The result is as shown in formula (5):

[0039] O F =W γ2 ReLU(LN(W γ1 (f(ε l )))) (5)

[0040] Output F Reflects χ l χ h compensation, which is derived from χ l Selected from all locations,

[0041] Step 44) The final output fusion feature map Y F for:

[0042] Y F =cat(O F , χ h ) (6).

[0043] The specific process of step 5) is as follows:

[0044] Step 51) Build a global attention module after the fifth stage of the ResNet-101 network to obtain long-range dependencies that are crucial for semantic segmentation. Let the input feature X∈R C×H×W , C, H, W are the number of channels, spatial height and width respectively, 1×1 convolution W θ Used to transform feature X:

[0045] θ=W θ (X) (7)

[0046] in is the number of channels of the converted feature;

[0047] Step 52) After regularization with the softmax function, the similarity matrix is ​​obtained

[0048] Step 53) The output of the attention module is processed by a 1×1 convolution W γ1 and W γ2 And the middle normalization and ReLU function are used to calculate the result, as shown in formula (8):

[0049]

[0050] Step 54) The final output feature map Y G ∈R C×H×W The expression is:

[0051] Y G =cat(O G , X) (9).

[0052] The specific process of step 6) is as follows:

[0053] Step 61) The final output fusion feature map Y obtained in step 5) is G Input into the classifier to generate channel semantic segmentation feature map;

[0054] Step 62) Comparing the generated feature map with the real label image annotated in step 1) to supervise the training of the feature extraction network parameters, thereby obtaining a trained network model; inputting the test sample dataset obtained in step 1) as an input image into the trained network model to test the performance of the network model;

[0055] Step 63) Load the trained model parameters and perform scene semantic analysis on the next batch of photos taken underground.

[0056] Beneficial effects:

[0057] The present invention adopts an attention mechanism to address the complex scenes in underground coal mine images, highlights the semantic information of the target area, and improves the image segmentation effect. Compared with other segmentation methods, it takes into account both the accuracy and speed of image segmentation, making it more robust.

[0058] This method enhances the extracted features by constructing a multi-scale input network; constructs a fusion attention module to fuse the features extracted at each stage; and constructs a global attention module to enhance global information and obtain long-range dependencies. Finally, a classifier is used to generate a semantic map to complete the semantic segmentation of the image, ensuring the accuracy of the segmentation and improving the robustness of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a schematic diagram of the basic residual network unit of the coal mine underground image semantic segmentation method of the present invention.

[0060] Figure 2 Schematic diagram of the attention fusion module of the coal mine underground image semantic segmentation method of the present invention.

[0061] Figure 3 Schematic diagram of the global attention module of the semantic segmentation method for coal mine underground images of the present invention.

[0062] Figure 4 It is a schematic diagram of the network framework of the multi-feature fusion image segmentation method in the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below with reference to the accompanying drawings.

[0064] The present invention provides a method for semantic segmentation of underground coal mine images. The method uses an underground explosion-proof camera to capture images of underground scenes, which are then preprocessed to generate a dataset. The dataset is then input, a feature extraction network is selected to extract features from the images, and a multi-scale input module is constructed to enhance the extracted feature map. A fusion attention module is then constructed to fuse the extracted features from each stage. A global attention module is constructed to enhance global information and obtain long-range dependencies. Finally, a classifier is used to generate a semantic map to complete the semantic segmentation of the image. This method has advantages over other semantic segmentation methods: the algorithm's computational complexity and complexity are greatly reduced. The attention mechanism is used to address complex scenes, highlighting the semantic information of the target area, improving the image segmentation effect, and greatly enhancing the robustness of the algorithm.

[0065] like Figure 4 As shown, the semantic segmentation method of an underground coal mine image of the present invention comprises the following steps:

[0066] Step 1) Collect downhole images, perform annotation preprocessing on the image data, and divide the annotated preprocessed image data into training sample and test sample data sets.

[0067] The specific process is:

[0068] Step 11) Use an underground explosion-proof camera to obtain a clear image.

[0069] Step 12) semantic segmentation and annotation of the obtained image, that is, each pixel in the image is classified; different regions in the image are segmented from each other, and each region is defined by semantic information.

[0070] Step 13) Randomly construct training sample sets and test sample sets from the labeled images in a ratio of 4:1.

[0071] Step 2) Input the training sample dataset obtained in step 1) into a feature extraction network with ResNet-101 as the backbone to extract input image features; delete the downsampling operations of the fourth and fifth stages of the five feature extraction stages in ResNet-101, retain the other contents of the fourth and fifth stages, and make its feature map 1 / 8 the size of the input image.

[0072] The specific process is:

[0073] Step 21) Use ResNet-101 as the backbone network for feature extraction. ResNet-101 is divided into five stages, each of which is composed of a basic residual convolution unit (RCU) to extract features of the input image and obtain output feature maps at different levels.

[0074] Step 22) Each of the five stages of the feature extraction network ResNet-101 contains multiple channels, and the importance of the information contained in each channel for semantic segmentation is also different. Therefore, a channel attention mechanism channel is added to each stage, and a 0-1 weight is assigned to each channel to indicate the importance of different channels.

[0075] Step 23) Delete the downsampling operations in the fourth and fifth stages. The receptive fields of the feature maps of the existing ResNet-101 in the fourth and fifth stages gradually increase with the process of convolution and downsampling, and the detail information of small objects is gradually lost. In order to enrich the detail information, the downsampling operations in the fourth and fifth stages are deleted in this step 23).

[0076] Step 24) Use dilated convolution to save the output feature maps of the fourth and fifth stages, so that the feature maps of the third, fourth and fifth stages are of the same size, which is 1 / 8 the size of the input image.

[0077] Step 3) After deleting the fourth and fifth stages of the downsampling operation, a multi-scale input is used to enhance the feature map of the input image size extracted in step 2), and the feature map is output.

[0078] The specific process is:

[0079] Step 31) The receptive field gradually increases with the process of convolution and downsampling, and the detail information of small targets is gradually lost. In order to obtain more detail information, multi-scale input is used, and the additional input image is input into the basic residual unit (RCU) to obtain the additional input feature maps of the fourth and fifth stages. The basic residual unit structure is as follows: Figure 1As shown in the figure, the additional input feature map obtained in this step undergoes feature extraction and is a low-level feature map. In the ResNet-101 network, the input image of each stage except the first stage is the output feature map of the previous stage. The input of the fourth and fifth stages is a high-level feature map, which contains less detailed information than the lower-level feature maps.

[0080] In step 32), the fourth and fifth stage additional input feature maps obtained in step 31) are fused with the fourth and fifth stage input feature maps of ResNet-101, respectively, so as to make full use of the shallow feature maps to enrich the information of small targets in the deep feature maps.

[0081] Step 33) The process of multi-scale input is: Assume that the ResNet-101 network contains L i layer convolution, then the j-th layer convolution can be defined as y j =M j (x j ), where y j is the output tensor of the jth layer, M j Contains convolution, ReLU activation function and regularization operation, the input image x of the i-th stage i The size of (N,H j ,W j ,W j ), N represents the batch size, H i and W i Represents the height and width of the input feature map, C i is the number of channels. The output feature map F of the i-th stage i It can be expressed as:

[0082]

[0083] Step 34) I i It represents the additional input of stage i, and its resolution is the same as the output tensor of stage i-1. Its feature map after feature extraction is:

[0084]

[0085] Step 35) The fusion input of stage i can be expressed as:

[0086]

[0087] Where, F i represents the output tensor of the i-th stage, Represents a channel splicing operation.

[0088] Step 36) The fifth stage outputs the high-level feature map χ h , the fourth stage low-level feature map χl .

[0089] Step 4) Construct a fusion attention module, which fuses the feature maps obtained in the fourth and fifth stages of step 3) with a size of 1 / 8 of the input image, and outputs a new feature map containing global context semantic information. Figure 2 As shown;

[0090] The specific process is:

[0091] Step 41) Construct the fusion attention module: The fusion attention module contains two inputs, the high-level feature map output from the fifth stage of step 36) And the low-level feature map output by the fourth stage H h ×W h is the high-level feature map χ h The number of spatial locations, H l ×W l is the low-level feature map χ l The number of spatial locations of C h and C l They are high-level feature maps χ h and low-level feature maps χ l The number of channels, 1×1 convolution W θ Used to transform the low-level feature map χ l The feature conversion of in is the number of channels of the converted feature, R is a real number, ε l is the low-level feature map χ l The result after feature conversion is shown in formula (4):

[0092] ε l =W θ (χ l ) (4)

[0093] Step 42) Convert the feature conversion result ε l After regularization with the softmax function, we get f(ε l ).

[0094] Step 43) Use bottleneck feature transformation to process f(ε l ), obtain channel dependency, 1×1 convolution W γ1 and W γ2 Will be used for χ h The feature transformation of attention output is obtained The result is as shown in formula (5):

[0095] O F =W γ2 ReLU(LN(W γ1 (f(εl )))) (5)

[0096] Output F Reflects χ l χ h compensation, which is derived from χ l Selected from all locations.

[0097] Step 44) The final output fusion feature map Y F for:

[0098] Y F =cat(O F , χ h ) (6)

[0099] Step 5) Build a global attention module after the fifth stage of the ResNet-101 network, as follows Figure 3 As shown in FIG, the global representation of the new feature map obtained in step 4) is enhanced, the long-range dependency relationship between features at different levels is obtained, and the final fused feature map is obtained.

[0100] The specific process is:

[0101] Step 51) Construct a global attention enhancement block to obtain long-range dependencies that are crucial for semantic segmentation. Let the input feature X∈R C×H×W , C, H, W are the number of channels, spatial height and width respectively, 1×1 convolution W θ Used to transform feature X:

[0102] θ=W θ (X) (7)

[0103] in is the number of channels of the converted features.

[0104] Step 52) After regularization with the softmax function, the similarity matrix is ​​obtained

[0105] Step 53) The output of the attention module is processed by a 1×1 convolution W γ1 and W γ2 And the middle normalization and ReLU function are used to calculate the result, as shown in formula (8):

[0106]

[0107] Step 54) The final output feature map Y G ∈R C×H×W It can be expressed as follows:

[0108] Y G =cat(OG ,X) (9)

[0109] Step 6) Input the fused output feature map obtained in step 5) into the pre-trained classifier to generate a semantic map. Then, input the test sample dataset obtained in step 1) into the trained network to test the network's performance.

[0110] Step 61) The final output fusion feature map Y obtained in step 5) is G Input into the classifier to generate channel semantic segmentation feature map.

[0111] In step 62), the generated feature map is compared with the real label image annotated in step 1) to supervise the training of the network model parameters, thereby obtaining a trained network model; the test sample data set obtained in step 1) is input as an input image into the trained network model to test the performance of the network model.

[0112] Step 63) loads the model parameters trained in step 62) and performs scene semantic analysis on the next batch of photos taken underground.

Claims

1. A method for semantic segmentation of coal mine images, characterized in that Here are the steps: Step 1: Collect downhole images, perform annotation preprocessing on the image data, and divide the annotated preprocessed image data into training sample and test sample data sets; Step 2: Input the training sample dataset into the feature extraction network to extract input image features. The feature extraction network includes an improved ResNet-101 network. The improved ResNet-101 network deletes the original downsampling operations in the fourth and fifth stages and retains the rest of the fourth and fifth stages. Step 3: In the fourth stage of the improved ResNet-101 network, the feature map output by the third stage and the additional input feature map are simultaneously input through multi-scale input to output a low-level feature map; The fifth stage simultaneously inputs the feature map output by the fourth stage and the additional input feature map through multi-scale input, and outputs a high-level feature map; the additional input feature map is obtained by processing the input image through the basic residual unit, and the additional input feature map is obtained by compressing the original input image to make it the same size as the output feature map of the previous stage; Step 31) Since the receptive field gradually increases with the process of convolution and downsampling, the detail information of small targets is gradually lost. In order to obtain more detail information, multi-scale input is adopted. Basic residual units are added to the input ends of the fourth and fifth stages of the improved ResNet-101 network respectively, and the additional 1 / 8 size input image is directly input to the basic residual unit to obtain the additional input feature maps of the fourth and fifth stages. The additional input feature maps obtained in this step are subjected to one feature extraction and are low-level feature maps. In the improved ResNet-101 network, the input images of each stage except the first stage are the output feature maps of the previous stage. The inputs of the fourth and fifth stages are high-level feature maps, which contain less detail information than the low-level feature maps. Step 32) The additional input feature maps processed by the basic residual unit in the fourth and fifth stages are fused with the normal input feature maps in the fourth and fifth stages respectively, making full use of the shallow feature maps to enrich the information of small objects in the deep feature maps; Step 33) Use multi-scale input to enhance the feature map of 1 / 8 size of the input image, where the process of multi-scale input is: Assume that the ResNet-101 network contains L i layer convolution, then the j-th layer convolution can be defined as y j =M j (x i ), where y j is the output feature map of the jth layer, M j Contains convolution, ReLU activation function and regularization operation, the input image x of the i-th stage i The size of (N,H i ,W i ,C i ), N represents the batch size, H i and W i Represents the height and width of the input feature map, C i is the number of channels; the output feature map F of the i-th stage i It can be expressed as: Step 34) I i represents the additional input feature map of the i-th stage, whose resolution is the same as the output feature map resolution of the i-1-th stage; Step 35) The fusion input of stage i is expressed as: Where, F i-1 represents the output feature map of the i-1th stage, Represents a channel splicing operation; Step 36) The fifth stage outputs a high-level feature map x h , the fourth stage low-level feature map χ l ; Step 4: Build a fusion attention module after the fifth stage of the improved ResNet-101 network, use the fusion attention module to fuse the low-level feature map and the high-level feature map, and output a new feature map containing global context semantic information; Step 41) Construct the fusion attention module: The fusion attention module contains two inputs, the high-level feature map output from the fifth stage of step 36) And the low-level feature map output by the fourth stage H h ×W h is the high-level feature map χ h The number of spatial locations, H l ×W l is the low-level feature map χ l The number of spatial locations of C h and C l They are high-level feature maps χ h and low-level feature maps χ l The number of channels, 1×1 convolution W θ Used to transform the low-level feature map χ l The feature conversion of in is the number of channels of the converted feature, R is a real number, ε l is the low-level feature map χ l The result after feature conversion is shown in formula (3): e l =W θ (x l ) (3) Step 42) Convert the feature conversion result ε l After regularization with the softmax function, we get f(ε l ); Step 43) Use bottleneck feature transformation to process f(ε l ), obtain channel dependency, 1×1 convolution W γ1 and W γ2 Will be used for χ h The feature transformation of attention output is obtained The result is as shown in formula (4): YOU ARE F JW γ2 ReLU(LN(W). γ1 (f(ε l )))) (4) Output F Reflects χ l χ h compensation, which is derived from χ l Selected from all locations, Step 44) The final output fusion feature map Y F for: Y F =cat(O F ,χ h ) (5) Step 5: Build a global context enhancement module after the fusion attention module to enhance the global representation of the new feature map, thereby obtaining the long-range dependency between pixels in the feature map and obtaining the final fused feature map; Step 51) Build a global attention module after the fifth stage of the ResNet-101 network to obtain long-range dependencies that are crucial for semantic segmentation. Let the input feature X∈R C×H×W , C, H, W are the number of channels, spatial height and width respectively, 1×1 convolution W θ Used to transform feature X: θ=W θ (X) (6) in is the number of channels of the converted feature; Step 52) After regularization with the softmax function, the similarity matrix is ​​obtained Step 53) The output of the attention module is processed by a 1×1 convolution W γ1 and W γ2 And the middle normalization and ReLU function are used to calculate the result as shown in formula (7): Step 54) The final output feature map Y G ∈R C×H×W The expression is: Y G =cat(O G ,X) (8); Step 6: Input the final fused feature map into the pre-trained classifier to generate a semantic map. Then, the performance of the generated semantic map is tested using the test sample data set to verify the performance of the feature extraction network. If the performance meets the standard, it can be used to perform semantic segmentation on coal mine photos. If it does not meet the standard, retraining is required. Step 7: Use the trained feature extraction network to perform semantic segmentation on the input coal mine underground image.

2. A method for semantic segmentation of coal mine images according to claim 1, characterized in that: The specific process of step 1 is: Step 11) using an underground explosion-proof camera to obtain a clear image; Step 12) Manually perform semantic segmentation and annotation on the obtained image, that is, classify each pixel in the image; different regions in the image are segmented from each other, and each region is defined by semantic information; Step 13) Randomly construct training sample sets and test sample sets from the labeled images in a ratio of 4:

1.

3. A method for semantic segmentation of coal mine images according to claim 1, characterized in that: The specific process of step 2 is as follows: Step 21) Based on the original ResNet-101 network, the improved ResNet-101 network is divided into five stages to extract the features of the input image, thereby obtaining output feature maps of different levels; Step 22) Each of the five stages of the improved ResNet-101 network contains multiple channels, and the importance of the information contained in each channel for semantic segmentation is also different. Therefore, a channel attention mechanism is added to each stage, and the importance of different channels is represented by assigning a 0-1 weight to each channel; Step 23) To enrich the detail information, the downsampling operations in the fourth and fifth stages are deleted, thereby preventing the receptive fields of the feature maps in the fourth and fifth stages of the conventional ResNet-101 from gradually increasing with the convolution and downsampling process, while the detail information of small objects in the feature maps is gradually lost; Step 24) Use dilated convolution to save the output feature maps of the fourth and fifth stages, so that the feature maps of the third, fourth, and fifth stages are the same size, which is 1 / 8 the size of the input image.

4. A method for semantic segmentation of coal mine images according to claim 1, characterized in that: The specific process of step 6 is as follows: Step 61) The final output fusion feature map Y obtained in step 5 is G Input into the classifier to generate channel semantic segmentation feature map; Step 62) Compare the generated feature map with the real label image annotated in step 1 to supervise the training of the feature extraction network parameters, thereby obtaining a trained network model; input the test sample dataset obtained in step 1 as an input image into the trained network model to test the performance of the network model; Step 63) Load the trained model parameters and perform scene semantic analysis on the next batch of photos taken underground.

Images