A coal particle image segmentation method, device, medium and equipment

Abstract

The application discloses a coal particle image segmentation method and device, medium and equipment, and relates to the technical field of image segmentation. In view of the problems of missing segmentation and over-segmentation in the image segmentation process, an algorithm for simultaneously paying attention to local information and global information is provided, low-level features are extracted from a ResNet50 network, and the features are converted into sequences by using semantic serialization, and then become the input of a multi-branch encoder. Secondly, in order to strengthen the extraction of edge features, the input of the encoder is repeatedly used as an input of a multi-head self-attention mechanism. Finally, the output sequence of the encoder is reshaped into the output features of the ResNet50 network and sent to the compression excitation space pyramid pooling part. By using the multi-branch Transformer encoder and the compression excitation space pyramid pooling, the application not only pays attention to the global information, but also strengthens the edge features, and solves the problems of missing segmentation and over-segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, and in particular relates to a coal particle image segmentation method, device, medium and equipment. BACKGROUND

[0002] Coal particle size analysis refers to the process of measuring and evaluating the size and distribution of coal particles. In the coal industry, particle size analysis can help determine the quality, combustion characteristics, processing efficiency and market demand of coal. Coal particle segmentation is a key step in coal particle size analysis, and the core task of this process is to accurately identify and extract coal particles and effectively distinguish and isolate them from non-coal substances. This process can be achieved through image segmentation methods.

[0003] In the prior art, Chinese patent CN 111145188 B discloses an image segmentation method based on ResNet and UNet models. The output results of the first three layers of the ResNet feature extraction module are replaced with the output results of the third, fourth and fifth layers of the UNet, and a ResNet and UNet based image segmentation training model is obtained. The model structure is as shown in Figure 1 The method uses ResNet as an image feature extraction module and deepens the network layers to solve the problem of inaccurate feature extraction in the image segmentation process.

[0004] However, because the pixel representations between coal particles and between coal particles and non-coal substances in coal particle images can be very similar, the above method cannot focus on the complex semantic features of coal particles when segmenting coal particle images, resulting in improved coal particle segmentation results. SUMMARY

[0005] Therefore, in order to solve the technical problem of poor coal particle image segmentation in the prior art, the present specification provides a coal particle image segmentation method, device, medium and equipment.

[0006] The present specification adopts the following technical solutions:

[0007] The present specification provides a coal particle image segmentation method, comprising:

[0008] A multi-branch Transformer model is added between the output end of the original Unet network encoder and the input end of the decoder, the input end of each Transformer branch in the multi-branch Transformer model is connected to the output end of the previous Transformer branch and the output end of the encoder, the skip connection between each layer of the original Unet network encoder and decoder is replaced with a compressed excitation hollow space pyramid pooling module to construct an improved unet network, and the improved unet network is trained to obtain a coal particle image segmentation model.

[0009] The coal particle image to be segmented is input into the coal particle image segmentation model, and the feature extraction of the coal particle image to be segmented is performed through multiple downsampling layers of the encoder to obtain low-level pixel feature maps of the coal particle image to be segmented at multiple scales; the low-level pixel features related to the coal particles in the multiple low-level pixel feature maps are amplified through the compressed excitation void space pyramid pooling module to obtain multiple coal particle key pixel feature maps; the feature extraction of the minimum scale low-level pixel feature map is performed through multiple Transformer branches of the multi-branch Transformer model to obtain a multi-level coal particle semantic feature map; the multi-level coal particle semantic feature map is upsampled through multiple upsampling layers of the encoder to obtain the low-level pixel features related to the multiple scales. Figure 1 A plurality of corresponding up-sampling images are obtained by fusing a plurality of coal particle key pixel feature maps with the corresponding plurality of up-sampling images to obtain restoration images of multiple scales, and the segmented coal particle image to be segmented is output based on the restoration image of the largest scale.

[0010] Furthermore, the improved unet network also includes replacing the encoder of the original Unet network with the first three layers of the resnet50 residual network.

[0011] Furthermore, the multi-branch Transformer model includes 12 Transformer branches, and the Transformer branches process the output feature map of the last layer of the resnet50 residual network and the output of the previous Transformer branch through a multi-head self-attention mechanism.

[0012] Furthermore, the compression-excited dilated spatial pyramid pooling module uses four dilated convolutions with different dilation rates, specifically including:

[0013] X1=f 1×1 (X)

[0014] X2=f 3×3,r＝6 (x)

[0015] X3=f 3×3，r＝12 (X)

[0016] X4=f 3×3，r = 18 (X)

[0017] X5=GAP(X)

[0018] Y i =SENet(X i )

[0019] Y=Concat(Y1, Y2, Y3, Y4, Y5)

[0020] Among them, X represents the input of the compressed excitation void spatial pyramid pooling module, X1~X4 represent the output of 4 void convolutions, and f n×n (·) represents a dilated convolution with a kernel size of n, r represents the dilation rate of the dilated convolution, GAP(·) represents global average pooling, SENet(·) represents the compressed-excited attention mechanism, and Y represents the output of the compressed-excited dilated spatial pyramid pooling.

[0021] Furthermore, before extracting features from the minimum-scale low-level pixel feature map through multiple Transformer branches of the multi-branch Transformer model, the minimum-scale low-level pixel feature map is segmented into N image blocks, and the N image blocks are mapped to a space of dimension D through a linear layer for semantic embedding:

[0022]

[0023] in, Represents a two-dimensional feature sequence; Xi is the image block of the i-th channel in the minimum-scale low-level pixel feature map, and the size of each image block is P×P; is the projection matrix of the image block, C is the number of channels of the image block; represents the position encoding matrix.

[0024] Furthermore, before performing the upsampling operation on the multi-level coal particle semantic feature map through the multiple upsampling layers of the encoder, the size of the multi-level coal particle semantic feature map is adjusted to be the same as the size of the minimum-scale low-level pixel feature map.

[0025] This specification provides a coal particle image segmentation device, comprising:

[0026] A model building module is used to add a multi-branch Transformer model between the output of the original Unet network encoder and the input of the decoder, wherein the input of each Transformer branch in the multi-branch Transformer model is simultaneously connected to the output of the previous Transformer branch and the output of the encoder; the skip connections between the encoder and decoder layers of the original Unet network are replaced with a compressed excitation dilated spatial pyramid pooling module to construct an improved UNET network; and the improved UNET network is trained to obtain a coal particle image segmentation model.

[0027] The image segmentation module is used to input the coal particle image to be segmented into the coal particle image segmentation model, perform feature extraction on the coal particle image to be segmented through multiple downsampling layers of the encoder, and obtain low-level pixel feature maps of the coal particle image to be segmented at multiple scales; amplify the low-level pixel features related to coal particles in multiple low-level pixel feature maps through the compression-excitation void space pyramid pooling module to obtain multiple coal particle key pixel feature maps; perform feature extraction on the minimum scale low-level pixel feature map through multiple Transformer branches of the multi-branch Transformer model to obtain a multi-level coal particle semantic feature map; perform upsampling operations on the multi-level coal particle semantic feature map through multiple upsampling layers of the encoder to obtain low-level pixel features related to multiple scales. Figure 1 A plurality of corresponding up-sampling images are obtained by fusing a plurality of coal particle key pixel feature maps with the corresponding plurality of up-sampling images to obtain restoration images of multiple scales, and the segmented coal particle image to be segmented is output based on the restoration image of the largest scale.

[0028] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned coal particle image segmentation method is implemented.

[0029] This specification provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned coal particle image segmentation method is implemented.

[0030] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0031] The present invention firstly amplifies the low-level pixel features related to coal particles in the low-level pixel feature map through the compression-excitation cavity space pyramid pooling module, and obtains a coal particle key pixel feature map with higher expression ability and resolution; then the multi-branch Transformer model is used to extract high-level coal particle semantic features. Since each Transformer branch in the multi-branch Transformer model operates based on the minimum-scale low-level pixel feature map and the output of the previous Transformer branch, this layer-by-layer progressive approach makes the final multi-level coal particle semantic feature map have rich semantic information and multiple levels of abstract features; the coal particle key pixel feature map and the up-sampled map obtained based on the multi-level coal particle semantic feature map are fused, and the obtained grayscale image has stronger expression ability and resolution ability, and more accurately captures the subtle features and semantic information of the coal particle image, thereby obtaining a more accurate segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0033] Figure 1 The model structure diagram of patent CN 111145188 B provided for this specification;

[0034] Figure 2 A schematic flow chart of a coal particle image segmentation method provided in this specification;

[0035] Figure 3 This is a model framework diagram of a coal particle image segmentation method provided in this specification;

[0036] Figure 4 Transformer branch diagram provided for this manual;

[0037] Figure 5 The multi-branch Transformer encoder ablation experiment results provided in this manual, the first column is the original image, the second column is the detection result of the GT model, the third column is the detection result of the TransUNet model, the third column is the detection result of the MIET model, the first row is coal sample 2, the second row is coal sample 1, and the third row is coal sample 3;

[0038] Figure 6 The compression-excitation void space pyramid pooling ablation experiment results provided in this manual are as follows: the first column is the original image, the second column is the detection result of the GT model, the third column is the detection result of the TransUNet model, and the third column is the detection result of the SWASPP model. The first row is coal sample 5, the second row is coal sample 6, and the third row is coal sample 2;

[0039] Figure 7 The results of the particle size analysis of the divided coal particles provided in this manual are shown in the figure. DETAILED DESCRIPTION

[0040] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0041] Studying coal particle size is crucial for coal transportation and efficient storage, as well as for coalbed methane development. Therefore, coal particle size analysis has become a hot research area in the coal industry. Coal particle segmentation is a key step in coal processing and utilization, aiming to separate raw coal from impurities, thereby improving coal quality and utilization efficiency. The core task of this process is to accurately identify and extract coal particles, effectively distinguishing and isolating them from non-coal matter.

[0042] Traditional coal particle segmentation methods usually rely on manual operation and visual inspection, which is time-consuming, labor-intensive and easily affected by subjective factors. With the advancement of digital image processing technology, scholars have increasingly used image analysis technology to study coal particles. Despite this, current methods still need to improve accuracy and stability when dealing with diverse coal particle shapes, but with the rapid development of computer vision technology, deep learning-based image segmentation networks have shown excellent performance in analyzing complex coal particle images. These networks can efficiently segment coal particles and extract precise features, providing powerful tools for the analysis and processing of coal particles. In particular, models based on convolutional neural networks (CNNs), such as Mask R-CNN and U-Net, have been widely used in the field of coal particle segmentation and have provided new directions and ideas for the development of this image segmentation.

[0043] In the task of coal particle segmentation, coal particles can vary greatly in shape and size, making it difficult for the model to accurately capture coal particles of various sizes and shapes. Current U-shaped image segmentation networks overly focus on local information and ignore global information, leading to over-segmentation. Therefore, some researchers have introduced the Transformer into the segmentation network. However, the self-attention mechanism in the Transformer network focuses on global information while ignoring local information, leading to under-segmentation, making it difficult to strike a balance between the two.

[0044] In summary, the problems existing in the prior art are:

[0045] (1) Existing models struggle to maintain a balance between global and local information, often failing to achieve satisfactory performance when segmenting small objects such as coal particles. U-shaped networks, often found in segmentation networks, focus only on local information and ignore global information, leading to over-segmentation of coal particles.

[0046] (2) With the emergence of Transformer, researchers introduced the Transformer mechanism into the U-shaped segmentation network. Although the introduction of Transformer solved the problem of ignoring global information, it did not make full use of edge detail information, resulting in the problem of missed segmentation of coal particles.

[0047] Difficulty in solving the above technical problems: The main difficulty of the above technical problems lies in the difficulty in balancing global information and local information. Excessive attention to one of the two will lead to over-segmentation or under-segmentation of coal particles.

[0048] The significance of solving the above technical problems: Accurately segmenting coal particles and extracting relevant features are crucial for determining the particle size and distribution of coal particles. These data have a far-reaching impact on revealing the diffusion mechanism of methane gas and improving coal processing technology.

[0049] Based on this, this specification provides a coal particle segmentation method based on a combination of a multi-branch Transformer encoder and a U-shaped network. Specific implementation methods of each application embodiment of this specification are described in detail below with reference to the accompanying drawings.

[0050] Example 1

[0051] Figure 2 The process of the coal particle image segmentation method of the embodiment of the present disclosure is shown. Figure 2 The coal particle image segmentation method is described in detail, which specifically includes the following steps:

[0052] S101: Replace the encoder of the original Unet network with a resnet50 residual network module, add a multi-branch Transformer model between the output of the original Unet network encoder and the input of the decoder, and replace the jump connection between the encoder and decoder of the original Unet network with a compressed excitation atrous spatial pyramid pooling module to construct an improved unet network; train the improved unet network to obtain a coal particle image segmentation model.

[0053] In order to solve the problem of over-segmentation or under-segmentation caused by the imbalance between local information and global information in the segmentation process based on coal particle images, a segmentation method based on the combination of a multi-branch Transformer encoder and a U-shaped network is proposed. In addition, in order to further utilize the extracted multi-scale features, a compressed excitation hollow spatial pyramid pooling is proposed. Specifically, first, in the U-shaped network encoder part, the features of the coal particle image are extracted by using the three cascaded convolutional layers in the ResNet50 network. Second, the extracted features are mapped into the input of the multi-branch Transformer encoder through image serialization and semantic serialization, and the extraction of edge features is strengthened by the multi-branch encoder. Finally, the output of the multi-branch Transformer encoder is reshaped to the size of the original feature map, and then enters the decoder of the U-shaped network. In order to fully utilize the multi-scale information, a compressed excitation hollow spatial pyramid pooling is proposed instead of the simple skip connection in the U-shaped network. Through the multi-head self-attention mechanism in the multi-branch Transformer encoder, not only the global information is focused, but also the extraction of edge and detail features is strengthened. In the decoder part, the compressed excitation hollow spatial pyramid pooling is used instead of the skip connection, which further strengthens the multi-scale information, so that the coal particle segmentation effect is better.

[0054] The improved unet network constructed based on the above steps is shown in Figure 3 and contains three modules, namely an input module, a multi-branch Transformer encoder module and a U-shaped network decoder module. 1) The input module consists of two parts: a ResNet50 network for feature extraction, and an image serialization and semantic embedding part for further strengthening the feature representation of the input. 2) The multi-branch Transformer encoder module contains 12 layers of iterative Transformer layers. Through the multi-head self-attention mechanism, this module not only focuses on global information, but also strengthens the extraction of edge and detail features by recycling input information, which helps to reduce the under-segmentation and over-segmentation problems in the coal particle segmentation process. 3) The U-shaped network decoder module uses a compressed excitation hollow spatial pyramid pooling module to replace the traditional simple skip connection, realizing the more optimal fusion of multi-scale information. Before segmentation, a convolutional layer is used to adjust the channel number of the fused feature map to make it consistent with the number of classes in the coal particle image segmentation task. The specific steps of constructing the improved unet network are shown in Figure 3 .

[0055] First, in the encoder part of the U-shaped network, the first three convolutional layers of the ResNet50 network are used to extract three feature maps, which specifically include: inputting the input image into the ResNet50 network; obtaining three down-sampled feature maps with different resolutions by using the first 3 convolutional layers in the ResNet50 network.

[0056] The second step is to perform image serialization and semantic embedding on the third feature map extracted above, and map it as the input of the multi-branch Transformer encoder. Specifically, the third downsampled feature map is divided into multiple image blocks, each of size P×P, for a total of N blocks. In this way, the input data is reconstructed into a series of vector sequences of length N. In order to fully utilize the spatial characteristics of the feature map, the semantic sequence needs to be mapped into a D-dimensional space, and the spatial information of the image block needs to be positionally encoded so that it can be used as the input of the multi-branch Transformer encoder:

[0057]

[0058] in, Two-dimensional feature sequence, is the position encoding matrix, is the projection matrix of the image patch.

[0059] In the third step, the encoder in the multi-branch Transformer encoder is designed iteratively. The multi-head self-attention mechanism in each Transformer layer takes the mapped encoding vector as input, further enhancing the extraction of edge features. The multi-branch Transformer model uses an iterative design and includes 12 Transformer layers. Each Transformer layer processes the input vector through the multi-head self-attention mechanism and combines the input vector with the output of the previous Transformer layer. The iterative design focuses on the fact that each layer's multi-head self-attention mechanism uses the downsampled feature map from the last layer of the previous ResNet50 as one of its inputs. In other words, the multi-head self-attention mechanism module has two input vectors: one is the downsampled feature map from the last layer of ResNet50, and the other is the output of the previous Transformer layer.

[0060] The semantically embedded features and the output features of the Transformer layer are used together as the input of the multi-head self-attention mechanism to enhance the model's ability to extract edge features. In the third step, the mapped vector is input into a multi-branch Transformer encoder. The encoder adopts an iterative design and contains 12 Transformer layers. Each layer processes the input vector through the multi-head self-attention mechanism (MSA) and combines it with the output of the previous layer. This not only focuses on global information, but also enhances the ability to extract edge and detail features. Each Transformer layer consists of three main parts: multi-head self-attention mechanism, layer normalization (LN), and multi-layer perceptron (MLP). The specific design process is as follows:

[0061] Z' l = MSA(LN(Z l-1 ))+ Z l-1 (2)

[0062] Z l = MLP(LN(Z l ))+ MSA(LN(Z l-1 ))+ Z l-1 (3)

[0063] wherein Z l-1 represents the output feature of the (l-1)-th layer of the Transformer, Z l represents the output feature after the Transformer layer, and l e {1, 2, …, 12}.

[0064] The fourth step is to reshape the output of the multi-branch Transformer encoder to the feature map size matching the U-Net network decoder, so as to be input into the decoding stage of the U-shaped network. The fourth step adjusts and reshapes the output of the multi-branch Transformer encoder to adapt to the required feature map size of the U-shaped network decoder, so as to be input into the multi-scale feature fusion.

[0065] The fifth step is to replace the original simple skip connection with the compressed excitation hollow space pyramid pooling in the U-shaped network decoder part, so as to perform feature fusion, and finally use convolution to adjust the channel number of the feature map to the class number of segmentation, so as to realize accurate segmentation of coal particles.

[0066] In order to reconstruct the detailed structure of the image, the adjusted feature map is input into the decoder part of the U-shaped network. The adjusted feature map will be fused with the feature map obtained by downsampling through the ResNet50 network, and this process is realized by means of compressed excitation hollow space pyramid pooling. Finally, the channel number of the feature map is adjusted through convolution, so as to match the class number of the segmentation task, so as to generate the final segmentation result. The fifth step uses the compressed excitation hollow space pyramid pooling part to fully fuse the multi-scale information, further strengthens the attention to the context information and edge feature information, and finally adjusts the channel number of the fused feature map to the class number of segmentation through convolution. The design of the compressed excitation hollow space pyramid pooling adopts four different dilated rate hollow convolutions, which combines global average pooling and compressed excitation attention mechanism to realize. The specific design process is as follows:

[0067] X1 = f 1×1 (X) (4)

[0068] X2 = f 3×3，r＝6 (X) (5)

[0069] X3=f 3×3，r＝12 (X) (6)

[0070] X4=f 3×3 , r＝18 (X) (7)

[0071] X5=GAP(X) (8)

[0072] Y i =SENet(X i ) (9)

[0073] Y=Concat(Y1, Y2, Y3, Y4, Y5)(10)

[0074] Among them, X represents the input of the compressed excitation void spatial pyramid pooling module, X1~X4 represent the output of 4 void convolutions, and f n×n (·) represents a dilated convolution with a kernel size of n, r represents the dilation rate of the dilated convolution (the first r = 1), GAP(·) represents global average pooling, SENet(·) represents the compressed-excited attention mechanism, and Y represents the output of the compressed-excited dilated spatial pyramid pooling.

[0075] S102: Input the coal particle image to be segmented into the coal particle image segmentation model. The resnet network extracts features from the coal particle image to be segmented through its N layers to obtain M local feature maps with successively decreasing scales. The multi-branch transformer model extracts features from the local feature map with the smallest scale to obtain global features. The encoder upsamples the global features through its M layers to obtain local features with successively decreasing scales. Figure 1 A corresponding restoration image of M scales increasing in sequence; the restoration image of the Mth scale is obtained by splicing the restoration image of the M-1th scale and the key feature map of the M-1th scale and inputting them into the upper mining layer, and the key feature map of the M-1th scale is obtained by using the compressed excitation void space pyramid pooling module to focus on amplifying the features of the M-1th scale; based on the restoration image of the largest scale, the segmented coal particle image to be segmented is output.

[0076] The embodiment of the present application can obtain the characteristics of coal particles by accurately segmenting the coal particle image, and can analyze these coal particle characteristics by the method of coal particle size analysis to obtain the required coal particle size characteristics. The present application considers the extraction of features from both global information and local information. In the image feature extraction stage, the embodiment of the present application uses ResNet50 network as the feature extraction model, which can extract feature maps of different resolutions. At the same time, the image serialization and semantic embedding are used to convert the feature map into a vector as the input of the encoder. In the encoder stage, the embodiment of the present application uses 12 Transformer layers to form a multi-branch Transformer encoder in an iterative manner. The embodiment of the present application uses a compressed air cavity space pyramid pooling instead of the skip connection in the UNet network, which fully fuses multi-scale information. Finally, the channel number of the fused feature map is adjusted to the number of segmentation classes. In summary, the combination algorithm of the U-shaped network and the multi-branch Transformer encoder in the embodiment of the present application not only pays attention to the global information, but also strengthens the extraction of edge detail features, solves the problems of over-segmentation and under-segmentation in the coal particle segmentation process, and further lays a good foundation for coal particle size feature analysis.

[0077] Based on Figure 2 The coal particle image segmentation method based on the combination of the multi-branch Transformer encoder and the U-shaped network, the multi-branch Transformer encoder repeatedly uses the feature map extracted from the ResNet50 network, reshapes the output sequence of the encoder into a feature map size, and sends this feature map into the compressed excitation space pyramid pooling part to extract more important feature information, thereby solving the problems of coal particle under-segmentation and over-segmentation.

[0078] The embodiment also provides experimental verification of the above method, specifically including:

[0079] The technical effects of the present application are described in detail in combination with performance tests and experimental analysis. The existing image segmentation model is difficult to balance between global information and local information, and is not helpful for coal particle image segmentation. The embodiment of the present application aims to solve the problems of over-segmentation and under-segmentation of coal particle image segmentation. The embodiment of the present application uses a self-made coal particle image dataset. Compared with the existing algorithm, the results are shown in the following table:

[0080]

[0081]

[0082] At the same time, in order to further illustrate the superiority of the algorithm, the embodiment of the present application gives the effect diagram of the compared algorithm in the above table in the coal particle image segmentation, as shown in Figure 4 From the above table and the effect diagram, it can be seen that the embodiment of the present application has better performance in the coal particle image segmentation. Figure 4 It can be seen that the result of segmenting coal particles by the proposed algorithm is closer to the GT frame.

[0083] To further demonstrate the effectiveness of the present invention, an ablation experiment was conducted in this embodiment. This embodiment employed the TransUNet algorithm to verify the effectiveness of the multi-branch Transformer encoder, with the results shown in the table below. As can be seen from the table below, if the multi-branch Transformer encoder used in this embodiment is not used, and a conventional Transformer is used for training instead, the model accuracy will be reduced.

[0084] method Accuracy Average Intersection-Union Ratio Recall TransUNe 0.992 0.862 0.858 TransUNet 0.994 0.908 0.908 Proposed algorithm 0.995 0.928 0.928

[0085] In order to further verify the impact of the compression-excitation void space pyramid pooling module in the embodiment of the present application, the embodiment of the present application conducted a pooling module ablation experiment. As above, the embodiment of the present application uses the self-made coal particle data as the training set and directly trains it in TransUNet. The obtained results are used as the baseline model results. Then, the compression-excitation void space pyramid pooling proposed in the embodiment of the present application is used, and the same coal particle data set is selected as the training set. The results are as follows Figure 5 As shown, from Figure 5 It can be seen that the compression-excited dilated spatial pyramid pooling module of the algorithm proposed in the embodiment of the present application achieves a good segmentation effect.

[0086] Finally, in order to more clearly illustrate the accuracy of coal particle size characteristic measurement, Figure 6 The results of a set of experiments on coal particle size measurement are presented. The horizontal axis represents the size of the coal particles, divided into 10 intervals, each 0.2 mm wide. The blue bars represent the number of coal particles within each size interval, while the red curve shows the cumulative distribution. The experimental results show that the proportion of coal particles in the 1-3 mm range reaches a maximum of 97.49% and a minimum of 89.56%. This fully demonstrates the high accuracy of the proposed algorithm in measuring coal particle size.

[0087] This specification also provides a coal particle image segmentation device, comprising:

[0088] A model building module is used to replace the encoder of the original Unet network with a resnet50 residual network module, add a multi-branch Transformer model between the output of the original Unet network encoder and the input of the decoder, and replace the skip connection between the encoder and decoder of the original Unet network with a compressed excitation dilated spatial pyramid pooling module to build an improved unet network; the improved unet network is trained to obtain a coal particle image segmentation model;

[0089] The image segmentation module is used to input the coal particle image to be segmented into the coal particle image segmentation model. The resnet network extracts features from the coal particle image to be segmented through its N layers to obtain M local feature maps with successively decreasing scales; the multi-branch Transformer model extracts features from the local feature map with the smallest scale to obtain global features; the encoder performs upsampling operations on the global features through its M layers to obtain local features with successively decreasing scales. Figure 1 A corresponding restoration image of M scales increasing in sequence; the restoration image of the Mth scale is obtained by splicing the restoration image of the M-1th scale and the key feature map of the M-1th scale and inputting them into the upper mining layer, and the key feature map of the M-1th scale is obtained by using the compressed excitation void space pyramid pooling module to focus on amplifying the features of the M-1th scale; based on the restoration image of the largest scale, the segmented coal particle image to be segmented is output.

[0090] The specific definition of the coal particle image segmentation device can be found in the definition of the coal particle image segmentation method above, and will not be repeated here. The various modules in the above-mentioned coal particle image segmentation device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0091] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 2 The coal particle image segmentation method is provided.

[0092] This specification also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 2 The coal particle image segmentation method is provided.

[0093] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0094] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A coal particle image segmentation method, characterized in that: include: A multi-branch Transformer model is added between the output of the encoder and the input of the decoder of the original Unet network. The input of each Transformer branch in the multi-branch Transformer model is simultaneously connected to the output of the previous Transformer branch and the output of the encoder. The skip connections between the encoder and decoder layers of the original Unet network are replaced with a compressed excitation dilated spatial pyramid pooling module to construct an improved unet network. Each Transformer layer consists of three main parts: a multi-head self-attention mechanism, layer normalization, and a multi-layer perceptron. The design process is as follows: Z′ l =MSA(LN(Z l-1 ))+Z l-1 WITH l =MLP(LN(Z′ l ))+MSA(LN(Z l-1 ))+Z l-1 Among them, Z l-1 represents the output features of the l-1 branch Transformer, l∈{1, 2, …, 12}; The improved unet network also includes replacing the encoder of the original Unet network with the first three layers of the resnet50 residual network; The multi-branch Transformer model includes 12 Transformer branches, each of which processes the output feature map of the last layer of the resnet50 residual network and the output of the previous Transformer branch through a multi-head attention mechanism; The compression-excitation atrous spatial pyramid pooling module uses four types of atrous convolutions with different dilation rates, specifically: X1=f 1×1 (X) X2=f 3×3，r＝6 (X) X3=f 3×3，r＝12 (X) X4=f 3×3，r＝18 (X) X5=GAP(X) Y i =SENet(X i ) Y=Concat(Y1, Y2, Y3, Y4, Y5) Among them, X represents the input of the compressed excitation void spatial pyramid pooling module, X1-X4 represents the output of 4 void convolutions, and f n×n (.) represents a dilated convolution with a kernel size of n, r represents the dilation rate of the dilated convolution, GAP(·) represents global average pooling, SENet(·) represents the compressed excitation attention mechanism, and Y represents the output of the compressed excitation dilated spatial pyramid pooling; The improved unet network is trained to obtain the coal particle image segmentation model; The coal particle image to be segmented is input into the coal particle image segmentation model, and the features of the coal particle image to be segmented are extracted by improving the multiple downsampling layers of the encoder of the unet network to obtain low-level pixel feature maps of the coal particle image to be segmented at multiple scales; the low-level pixel features related to the coal particles in the multiple low-level pixel feature maps are amplified by the compressed excitation void spatial pyramid pooling module to obtain multiple coal particle key pixel feature maps; the minimum scale low-level pixel feature map is extracted by multiple Transformer branches of the multi-branch Transformer model to obtain a multi-level coal particle semantic feature map; the multi-level coal particle semantic feature map is upsampled by improving the multiple upsampling layers of the decoder of the unet network to obtain multiple upsampling maps corresponding to the low-level pixel feature maps at multiple scales; the multiple coal particle key pixel feature maps are fused with the corresponding multiple upsampling maps to obtain restoration maps at multiple scales, and the segmented coal particle image to be segmented is output based on the restoration map at the maximum scale.

2. The coal particle image segmentation method according to claim 1, characterized in that: Before extracting features from the minimum-scale low-level pixel feature map through multiple Transformer branches of the multi-branch Transformer model, the minimum-scale low-level pixel feature map is divided into N image blocks, and the N image blocks are mapped to a space of dimension D through a linear layer for semantic embedding: in, represents a two-dimensional feature sequence, X i is the image patch of the i-th channel in the minimum-scale low-level pixel feature map, and the size of each image patch is P×P; is the projection matrix of the image block, C is the number of channels of the image block; Represents the image position encoding matrix.

3. The coal particle image segmentation method according to claim 1, characterized in that: Before performing the upsampling operation on the multi-level coal particle semantic feature map through the multiple upsampling layers of the decoder of the improved unet network, the size of the multi-level coal particle semantic feature map is adjusted to make it the same as the size of the minimum-scale low-level pixel feature map.

4. A coal particle image segmentation device, characterized in that: include: A model building module is used to add a multi-branch Transformer model between the output of the original Unet network encoder and the input of the decoder, where the input of each Transformer branch in the multi-branch Transformer model is simultaneously connected to the output of the previous Transformer branch and the output of the encoder; the skip connections between the encoder and decoder layers of the original Unet network are replaced with a compressed excitation dilated spatial pyramid pooling module to construct an improved Unet network; Each Transformer layer consists of three main parts: a multi-head self-attention mechanism, layer normalization, and a multi-layer perceptron. The design process is as follows: Z′ l =MSA(LN(Z l-1 ))+Z l-1 WITH l =MLP(LN(Z′ l ))+MSA(LN(Z l-1 ))+Z l-1 Among them, Z l-1 represents the output feature of the l-1th branch Transformer, l∈{1, 2, …, 12}; The improved unet network also includes replacing the encoder of the original Unet network with the first three layers of the resnet50 residual network; The multi-branch Transformer model includes 12 Transformer branches, each of which processes the output feature map of the last layer of the resnet50 residual network and the output of the previous Transformer branch through a multi-head attention mechanism; The compression-excitation atrous spatial pyramid pooling module uses four types of atrous convolutions with different dilation rates, specifically: X1=f 1×1 (X) X2=f 3×3，r＝6 (X) X3=f 3×3，r＝12 (X) X4=f 3×3，r＝18 (X) X5=GAP(X) Y i =SENet(X i ) Y=Concat(Y1, Y2, Y3, Y4, Y5) Among them, X represents the input of the compressed excitation void spatial pyramid pooling module, X1-X4 represents the output of 4 void convolutions, and f n×n (.) represents a dilated convolution with a kernel size of n, r represents the dilation rate of the dilated convolution, GAP(·) represents global average pooling, SENet(·) represents the compressed excitation attention mechanism, and Y represents the output of the compressed excitation dilated spatial pyramid pooling; The improved unet network is trained to obtain the coal particle image segmentation model; The image segmentation module is used to input the coal particle image to be segmented into the coal particle image segmentation model, extract features of the coal particle image to be segmented through multiple downsampling layers of the encoder of the improved UNET network, and obtain low-level pixel feature maps of the coal particle image to be segmented at multiple scales; amplify the low-level pixel features related to coal particles in multiple low-level pixel feature maps through the compressed excitation void space pyramid pooling module to obtain multiple coal particle key pixel feature maps; extract features of the minimum scale low-level pixel feature map through multiple Transformer branches of the multi-branch Transformer model to obtain a multi-level coal particle semantic feature map; upsample the multi-level coal particle semantic feature map through multiple upsampling layers of the decoder of the improved UNET network to obtain multiple upsampling maps corresponding to the low-level pixel feature maps of multiple scales; obtain restoration maps of multiple scales by fusing multiple coal particle key pixel feature maps with the corresponding multiple upsampling maps, and output the segmented coal particle image to be segmented based on the restoration map of the largest scale.

5. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

6. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

Images