Coal particle image segmentation method and device, medium and equipment
By combining the multi-branch Transformer model and the improved UNet network in coal particle image segmentation, the compressed excitation hollow space pyramid pooling module is used to solve the problem of difficulty in balancing global information and local information in coal particle image segmentation, and achieve more accurate Coal particle segmentation and feature extraction.
Patent Information
- Application Number
- CN202410966971.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-18
AI Technical Summary
The prior art is difficult to balance global information with local information in coal particle image segmentation, resulting in over-segment or missing segmentation problems, and it is impossible to accurately capture the complex semantic characteristics of coal particles.
Using a multi-branch Transformer model combined with the improved UNet network, multi-branch Transformer model is added between the encoder and decoder of the UNet network, and replacing the jump connection as a compressed excitation hollow space pyramid pooling module, multi-scale features are extracted and performed Semantic embedding, enhanced edge and detail feature extraction.
It improves the accuracy and resolution of coal particle image segmentation, can capture the subtle features and semantic information of coal particles more accurately, reduces the problems of leakage and oversegment, and significantly improves the segmentation effect.
Smart Images

Figure CN118968055A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image segmentation, and in particular to a method, device, medium and equipment for segmenting coal particle images. Background Art
[0002] Coal particle size analysis refers to the process of measuring and evaluating the size and distribution of coal particles. In the coal industry, particle size analysis can help determine the quality, combustion characteristics, processing efficiency and market demand of coal. Coal particle segmentation is a key step in coal particle size analysis. The core task of this process is to accurately identify and extract coal particles and effectively distinguish and isolate them from non-coal materials. This process can be achieved through image segmentation methods.
[0003] In the prior art, Chinese patent CN 111145188 B discloses an image segmentation method based on ResNet and UNet models. The method replaces the output results of the third, fourth and fifth layers of UNet with the output results of the first three layers of the ResNet feature extraction module to obtain an image segmentation training model based on ResNet and UNet. The model structure is as follows: Figure 1 This method uses ResNet as the image feature extraction module and deepens the number of network layers to solve the problem of inaccurate feature extraction during image segmentation.
[0004] However, since the pixel representations between coal particles and between coal particles and non-coal materials in coal particle images may be very similar, the above method cannot pay attention to the complex semantic characteristics of coal particles when segmenting coal particle images, resulting in the need to improve the segmentation effect of coal particles. Summary of the invention
[0005] Based on this, in order to solve the technical problem of poor coal particle image segmentation effect in the prior art, this specification provides a coal particle image segmentation method, device, medium and equipment.
[0006] This manual adopts the following technical solutions:
[0007] This specification provides a coal particle image segmentation method, including:
[0008] A multi-branch Transformer model is added between the output end of the original Unet network encoder and the input end of the decoder, wherein the input end of each Transformer branch in the multi-branch Transformer model is simultaneously connected to the output end of the previous Transformer branch and the output end of the encoder; the jump connection between each layer of the encoder and decoder of the original Unet network is replaced with a compressed excitation void spatial pyramid pooling module to construct an improved unet network; the improved unet network is trained to obtain a coal particle image segmentation model;
[0009] The coal particle image to be segmented is input into the coal particle image segmentation model, and the feature extraction of the coal particle image to be segmented is performed through multiple downsampling layers of the encoder to obtain low-level pixel feature maps of the coal particle image to be segmented at multiple scales; the low-level pixel features related to coal particles in multiple low-level pixel feature maps are amplified through the compressed excitation void space pyramid pooling module to obtain multiple coal particle key pixel feature maps; the feature extraction of the minimum scale low-level pixel feature map is performed through multiple Transformer branches of the multi-branch Transformer model to obtain a multi-level coal particle semantic feature map; the multi-level coal particle semantic feature map is upsampled through multiple upsampling layers of the encoder to obtain the low-level pixel features related to multiple scales. Figure 1 A plurality of corresponding up-sampling images; by fusing a plurality of coal particle key pixel feature images with the corresponding plurality of up-sampling images, restoration images of a plurality of scales are obtained, and the segmented coal particle image to be segmented is output based on the restoration image of the maximum scale.
[0010] Furthermore, the improved unet network also includes replacing the encoder of the original Unet network with the first three layers of the resnet50 residual network.
[0011] Furthermore, the multi-branch Transformer model includes 12 Transformer branches, and the Transformer branches process the output feature map of the last layer of the resnet50 residual network and the output of the previous Transformer branch through a multi-head self-attention mechanism.
[0012] Furthermore, the compression-excited atrous spatial pyramid pooling module uses four atrous convolutions with different dilation rates, specifically including:
[0013] X1=f 1×1 (X)
[0014] X2=f 3×3,r=6 (x)
[0015] X3=f 3×3,r=12 (X)
[0016] X4=f 3×3,r = 18 (X)
[0017] X5=GAP(X)
[0018] Y i =SENet(X i )
[0019] Y=Concat(Y1, Y2, Y3, Y4, Y5)
[0020] Among them, X represents the input of the compressed excitation dilated spatial pyramid pooling module, X1~X4 represent the outputs of four dilated convolutions, and f n×n (·) represents a dilated convolution with a kernel size of n, r represents the dilation rate of the dilated convolution, GAP(·) represents global average pooling, SENet(·) represents the compressed-excited attention mechanism, and Y represents the output of compressed-excited dilated spatial pyramid pooling.
[0021] Furthermore, before extracting features from the minimum-scale low-level pixel feature map through multiple Transformer branches of the multi-branch Transformer model, the minimum-scale low-level pixel feature map is segmented into N image blocks, and the N image blocks are mapped to a space of dimension D through a linear layer for semantic embedding:
[0022]
[0023] in, Represents a two-dimensional feature sequence; Xi is the image block of the i-th channel in the minimum-scale low-level pixel feature map, and the size of each image block is P×P; is the projection matrix of the image block, and C is the number of channels of the image block; represents the position encoding matrix.
[0024] Furthermore, before performing the upsampling operation on the multi-level coal particle semantic feature map through the multiple upsampling layers of the encoder, the size of the multi-level coal particle semantic feature map is adjusted to be the same as the size of the minimum-scale low-level pixel feature map.
[0025] This specification provides a coal particle image segmentation device, comprising:
[0026] A model building module is used to add a multi-branch Transformer model between the output end of the original Unet network encoder and the input end of the decoder, wherein the input end of each Transformer branch in the multi-branch Transformer model is simultaneously connected to the output end of the previous Transformer branch and the output end of the encoder; the jump connection between each layer of the encoder and decoder of the original Unet network is replaced with a compressed excitation void spatial pyramid pooling module to construct an improved unet network; the improved unet network is trained to obtain a coal particle image segmentation model;
[0027] The image segmentation module is used to input the coal particle image to be segmented into the coal particle image segmentation model, perform feature extraction on the coal particle image to be segmented through multiple downsampling layers of the encoder, and obtain low-level pixel feature maps of the coal particle image to be segmented at multiple scales; amplify the low-level pixel features related to coal particles in multiple low-level pixel feature maps through the compression-excitation void space pyramid pooling module to obtain multiple coal particle key pixel feature maps; perform feature extraction on the minimum scale low-level pixel feature map through multiple Transformer branches of the multi-branch Transformer model to obtain a multi-level coal particle semantic feature map; perform upsampling operations on the multi-level coal particle semantic feature map through multiple upsampling layers of the encoder to obtain low-level pixel features related to multiple scales. Figure 1 A plurality of corresponding up-sampling images; by fusing a plurality of coal particle key pixel feature images with the corresponding plurality of up-sampling images, restoration images of a plurality of scales are obtained, and the segmented coal particle image to be segmented is output based on the restoration image of the maximum scale.
[0028] The present specification provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned coal particle image segmentation method is implemented.
[0029] This specification provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned coal particle image segmentation method when executing the program.
[0030] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:
[0031] The present invention firstly amplifies the low-level pixel features related to coal particles in the low-level pixel feature map through the compression excitation cavity space pyramid pooling module, and obtains a coal particle key pixel feature map with better expression ability and resolution; then a multi-branch Transformer model is used to extract high-level coal particle semantic features. Since each Transformer branch in the multi-branch Transformer model operates based on the minimum-scale low-level pixel feature map and the output of the previous Transformer branch, this layer-by-layer progressive approach makes the final multi-level coal particle semantic feature map have rich semantic information and multiple levels of abstract features; the coal particle key pixel feature map and the up-sampled map obtained based on the multi-level coal particle semantic feature map are fused, and the obtained grayscale image has stronger expression ability and resolution ability, and can more accurately capture the subtle features and semantic information of the coal particle image, thereby obtaining a more accurate segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0033] Figure 1 Patent CN 111145188 B model structure diagram provided for this specification;
[0034] Figure 2 A schematic diagram of a flow chart of a coal particle image segmentation method provided in this specification;
[0035] Figure 3 A model framework diagram of a coal particle image segmentation method provided in this specification;
[0036] Figure 4 Transformer branch diagram provided for this manual;
[0037] Figure 5 The multi-branch Transformer encoder ablation experiment result diagram provided in this manual, the first column is the original image, the second column is the detection result of the GT model, the third column is the detection result of the TransUNet model, the third column is the detection result of the MIET model, the first row is coal sample 2, the second row is coal sample 1, and the third row is coal sample 3;
[0038] Figure 6 The compression excitation void space pyramid pooling ablation experiment result diagram provided in this manual, the first column is the original image, the second column is the detection result of the GT model, the third column is the detection result of the TransUNet model, the third column is the detection result of the SWASPP model, the first row is coal sample 5, the second row is coal sample 6, and the third row is coal sample 2;
[0039] Figure 7 This is the result of particle size analysis of divided coal particles provided in this manual. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this application will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0041] Studying the particle size of coal particles has important reference value for the transportation and efficient storage of coal, as well as the development of coalbed methane. Therefore, coal particle size analysis has become a hot research field in the coal industry. Coal particle segmentation is a key step in the process of coal processing and utilization. Its purpose is to separate raw coal from impurities, thereby improving the quality and efficiency of coal. The core task of this process is to accurately identify and extract coal particles, and effectively distinguish and isolate them from non-coal substances.
[0042] Traditional coal particle segmentation methods usually rely on manual operation and visual inspection, which is time-consuming, labor-intensive and easily affected by subjective factors. With the advancement of digital image processing technology, scholars have increasingly used image analysis technology to study coal particles. Despite this, current methods still need to improve accuracy and stability when dealing with diverse coal particle shapes, but with the rapid development of computer vision technology, deep learning-based image segmentation networks have shown excellent performance in analyzing complex coal particle images. These networks can efficiently segment coal particles and extract precise features, providing powerful tools for the analysis and processing of coal particles. In particular, models based on convolutional neural networks (CNNs), such as Mask R-CNN and U-Net, have been widely used in the field of coal particle segmentation and have provided new directions and ideas for the development of this image segmentation.
[0043] In the coal particle segmentation task, coal particles may have large differences in shape and size, which may make it difficult for the model to accurately capture coal particles of various sizes and shapes. The current U-shaped image segmentation network focuses too much on local information and ignores global information, which will lead to over-segmentation problems; therefore, some scholars introduced Transformer into the segmentation network, but the self-attention mechanism in the Transformer network focuses on global information and ignores local information, which will lead to missed segmentation problems. It is difficult to strike a balance between the two.
[0044] In summary, the problems existing in the prior art are:
[0045] (1) Existing models have difficulty in maintaining a balance between global information and local information, and often fail to achieve satisfactory performance when segmenting objects with small particle sizes such as coal particles. The U-shaped network that often appears in segmentation networks only focuses on the local information of the features and ignores the global information, resulting in the problem of over-segmentation of coal particles.
[0046] (2) With the emergence of Transformer, researchers introduced the Transformer mechanism into the U-shaped segmentation network. Although the introduction of Transformer solved the problem of ignoring global information, it did not make full use of edge detail information, resulting in the problem of missed segmentation of coal particles.
[0047] Difficulty in solving the above technical problems: The main difficulty of the above technical problems lies in the difficulty in balancing global information and local information. Excessive attention to one of the two will lead to over-segmentation or under-segmentation of coal particles.
[0048] The significance of solving the above technical problems: Accurately segmenting coal particles and extracting relevant features are crucial to determining the particle size and distribution of coal particles. These data have a far-reaching impact on revealing the diffusion mechanism of methane gas and improving coal processing technology.
[0049] Based on this, this specification provides a coal particle segmentation method based on a combination of a multi-branch Transformer encoder and a U-shaped network. The specific implementation methods of each application embodiment of this specification are described in detail below in conjunction with the accompanying drawings.
[0050] Example 1
[0051] Figure 2 The process of the coal particle image segmentation method of the embodiment of the present disclosure is shown. Figure 2 The coal particle image segmentation method is described in detail, and specifically comprises the following steps:
[0052] S101: Replace the encoder of the original Unet network with a resnet50 residual network module, add a multi-branch Transformer model between the output of the original Unet network encoder and the input of the decoder, and replace the jump connection between the encoder and decoder of the original Unet network with a compressed excitation void spatial pyramid pooling module to construct an improved unet network; train the improved unet network to obtain a coal particle image segmentation model.
[0053] In the process of segmentation of coal particle images, due to the inability to balance local information and global information well, over-segmentation or under-segmentation problems may occur. Therefore, a segmentation method based on the combination of a multi-branch Transformer encoder and a U-type network is proposed. In addition, in order to further make better use of the extracted multi-scale features, compressed excitation void space pyramid pooling is proposed. Specifically, first, in the U-type network encoder part, three cascaded convolutions in the ResNet50 network are used to extract the features of the coal particle image. Secondly, the extracted information is mapped to the input of the multi-branch Transformer encoder through image serialization and semantic serialization, and the extraction of edge features is strengthened through the multi-branch encoder. Finally, the output of the multi-branch Transformer encoder is reshaped to the size of the original feature map, and then enters the decoder of the U-type network. In order to make full use of multi-scale information, compressed excitation void space pyramid pooling is proposed to replace the simple jump connection in the U-type network. The embodiment of the present application not only pays attention to the global information but also strengthens the extraction of edge detail features through the multi-head self-attention mechanism in the multi-branch Transformer encoder. In the decoder part, compressed excitation void space pyramid pooling is used instead of jump connection to further strengthen the multi-scale information, so that the effect of coal particle segmentation is better.
[0054] The improved unet network constructed based on the above steps is as follows Figure 3 As shown in the figure, it contains three modules, namely the input module, the multi-branch Transformer encoder module and the U-type network decoder module. 1) The input module consists of two parts: the ResNet50 network is used to extract features, and the image serialization and semantic embedding parts are used to further strengthen the feature representation of the input. 2) The multi-branch Transformer encoder module contains 12 layers of iterative Transformer layers. Through the multi-head self-attention mechanism, this module not only pays attention to the global information, but also strengthens the extraction of edge and detail features by recycling the input information, which helps to reduce the problems of missed segmentation and over-segmentation in the coal particle segmentation process. 3) The U-type network decoder module uses a compressed excitation void space pyramid pooling module to replace the traditional simple jump connection, thereby achieving better fusion of multi-scale information. Before segmentation, a convolutional layer is used to adjust the number of channels of the fused feature map to make it consistent with the number of categories in the coal particle image segmentation task. Construct as follows Figure 3 The specific steps shown to improve the unet network include:
[0055] In the first step, in the encoder part of the U-type network, the first three cascaded convolutional layers of the ResNet50 network are used to extract three feature maps, specifically including: sending the input image to the ResNet50 network; using the first three convolutional layers in the ResNet50 network to obtain three downsampled feature maps of different resolutions.
[0056] The second step is to perform image serialization and semantic embedding on the third feature map extracted above, and map it as the input of the multi-branch Transformer encoder, which specifically includes: dividing the third downsampled feature map into multiple image blocks, each block is of size P×P, and a total of N blocks. In this way, the input data is reconstructed into a series of vector sequences of length N; in order to make full use of the spatial features of the feature map, the semantic sequence needs to be mapped to the D-space, and the spatial information of the image block needs to be positionally encoded so that it can be used as the input of the multi-branch Transformer encoder:
[0057]
[0058] in, Two-dimensional feature sequence, is the position encoding matrix, is the projection matrix of the image patch.
[0059] In the third step, the encoder is designed in an iterative way in the multi-branch Transformer encoder. The multi-head self-attention mechanism in each Transformer layer takes the mapped encoding vector as input, further strengthening the extraction of edge features. The multi-branch Transformer model adopts an iterative design and includes 12 Transformer layers. Each Transformer layer processes the input vector through the multi-head self-attention mechanism and combines the input vector with the output of the previous Transformer layer. The iterative design is mainly that the multi-head self-attention mechanism of each layer must use the feature map of the last layer of ResNet50 downsampled as one of the inputs. That is, there are two input vectors for the multi-head self-attention mechanism module, one is the feature map of the last layer of ResNet50 downsampled, and the other input is the output of the previous Transformer layer.
[0060] The semantically embedded features and the output features of the Transformer layer are used together as the input of the multi-head self-attention mechanism to enhance the model's ability to extract edge features. In the third step, the mapped vector is input into a multi-branch Transformer encoder. The encoder adopts an iterative design and contains 12 Transformer layers. Each layer processes the input vector through a multi-head self-attention mechanism (MSA) and combines it with the output of the previous layer. This not only pays attention to global information, but also enhances the ability to extract edge and detail features. Each Transformer layer consists of three main parts: a multi-head self-attention mechanism, layer normalization (LN), and a multi-layer perceptron (MLP). The specific design process is as follows:
[0061] Z′ l =MSA(LN(Z l-1 ))+Z l-1 (2)
[0062] Z l =MLP(LN(Z′ l ))+MSA(LN(Z l-1 ))+Z l-1 (3)
[0063] Among them, Z l-1 Represents the output features of the l-1th layer Transformer, Z l Represents the output features after the Transformer layer, l∈{1, 2,…, 12}.
[0064] In the fourth step, the output of the multi-branch Transformer encoder is reshaped into a feature map size that matches the U-Net network decoder so that it can be input into the decoding stage of the U-net. In the fourth step, the output of the multi-branch Transformer encoder is adjusted and reshaped to adapt to the feature map size required by the U-net decoder so that it can be used as input for multi-scale feature fusion.
[0065] The fifth step is to use compressed excitation void spatial pyramid pooling to replace the original simple jump connection in the U-type network decoder to make full use of multi-scale information for feature fusion. Finally, convolution is used to adjust the number of channels of the feature map to the number of segmented categories, thus achieving accurate segmentation of coal particles.
[0066] In order to reconstruct the detailed structure of the image, the adjusted feature map is input into the decoder part of the U-type network. The adjusted feature map will be fused with the feature map obtained by downsampling the ResNet50 network, which is achieved with the help of compressed excitation dilated spatial pyramid pooling. Finally, the number of channels of the feature map is adjusted through convolution to match the number of categories of the segmentation task to generate the final segmentation result. In the fifth step, the multi-scale information is fully integrated using the compressed excitation dilated spatial pyramid pooling part, which further strengthens the attention to contextual information and edge feature information. Finally, the number of channels of the fused feature map is adjusted to the number of segmentation categories through convolution. The design of compressed excitation dilated spatial pyramid pooling adopts four kinds of dilated convolutions with different expansion rates, combined with global average pooling and compressed excitation attention mechanism. The specific design process is as follows:
[0067] X1=f 1×1 (X) (4)
[0068] X2=f 3×3,r=6 (X) (5)
[0069] X3=f 3×3,r=12 (X) (6)
[0070] X4=f 3×3 , r=18 (X) (7)
[0071] X5=GAP(X) (8)
[0072] Y i =SENet(X i ) (9)
[0073] Y=Concat(Y1, Y2, Y3, Y4, Y5)(10)
[0074] Among them, X represents the input of the compressed excitation dilated spatial pyramid pooling module, X1~X4 represent the outputs of four dilated convolutions, and f n×n (·) represents a dilated convolution with a kernel size of n, r represents the dilation rate of the dilated convolution (the first r = 1), GAP(·) represents global average pooling, SENet(·) represents the compressed-excited attention mechanism, and Y represents the output of compressed-excited dilated spatial pyramid pooling.
[0075] S102: Input the coal particle image to be segmented into the coal particle image segmentation model. The resnet network extracts features of the coal particle image to be segmented through its N layers to obtain M local feature maps with successively decreasing scales. The multi-branch Transformer model extracts features from the local feature map with the smallest scale to obtain global features. The encoder upsamples the global features through its M layers to obtain local features with successively decreasing scales. Figure 1 A corresponding restoration image of M scales increasing in sequence; the restoration image of the Mth scale is obtained by splicing the restoration image of the M-1th scale and the key feature map of the M-1th scale and inputting them into the upper mining layer, and the key feature map of the M-1th scale is obtained by focusing on amplifying the features of the M-1th scale using a compressed excitation void space pyramid pooling module; the coal particle image to be segmented after segmentation is output based on the restoration image of the largest scale.
[0076] The embodiment of the present application accurately segments the coal particle image to obtain the characteristics of the coal particles. These coal particle characteristics can be analyzed by the method of coal particle size analysis to obtain the required coal particle size characteristics. The present invention considers the extraction of features from both the global information and local information of the feature map. In the image feature extraction stage, the embodiment of the present application adopts the ResNet50 network as the feature extraction model, which can extract feature maps of different resolutions, and uses image serialization and semantic embedding to convert the feature map into a vector as the input of the encoder. In the encoder stage, the embodiment of the present application uses 12 Transformer layers to iteratively form a multi-branch Transformer encoder. The embodiment of the present application uses the hollow space pyramid pooling in the compressor to replace the jump connection in the UNet network, fully fuses the multi-scale information, and finally adjusts the number of channels of the fused feature map to the number of segmented categories. In summary, the embodiment of the present application not only pays attention to the global information through the combination algorithm of the U-type network and the multi-branch Transformer encoder, but also strengthens the extraction of edge detail features, solves the problem of over-segmentation and missed segmentation in the coal particle segmentation process, and further lays a good foundation for the analysis of coal particle size characteristics.
[0077] based on Figure 2 The coal particle image segmentation method shown in the figure is based on the coal particle image segmentation method combining a multi-branch Transformer encoder with a U-type network. The multi-branch Transformer encoder repeatedly reuses the feature map extracted from the ResNet50 network, reshapes the output sequence of the encoder into the size of the feature map, and sends the feature map to the compressed excitation space pyramid pooling part to extract more important feature information, thereby solving the problems of missed segmentation and over-segmentation of coal particles.
[0078] This embodiment also provides experimental verification of the above method, which specifically includes:
[0079] The technical effect of the present invention is described in detail in combination with performance testing and experimental analysis. The existing image segmentation model is difficult to balance between global information and local information, and is not very helpful for coal particle image segmentation. The embodiment of the present application aims to solve the problem of over-segmentation and under-segmentation of coal particle image segmentation. The embodiment of the present application uses a self-made coal particle image dataset. It is compared with the existing algorithm, and the results are shown in the following table:
[0080]
[0081]
[0082] At the same time, in order to further illustrate the superiority of the proposed algorithm, the embodiment of the present application provides the effect diagram of coal particle image segmentation of the compared algorithms in the above table, such as Figure 4 As shown. Figure 4 It can be seen that the result of segmenting coal particles by the proposed algorithm is closer to the GT frame.
[0083] In order to further demonstrate the effectiveness of the present invention, an ablation experiment was conducted in the embodiment of the present application. The embodiment of the present application uses the TransUNet algorithm to verify the effect of the multi-branch Transformer encoder, and the results are shown in the following table. As can be seen from the table below, if the multi-branch Transformer encoder used in the embodiment of the present application is not used, but the commonly used Transformer is used for training, the accuracy of the model will be reduced.
[0084] method Accuracy Average Intersection-Union Ratio Recall TransUNe 0.992 0.862 0.858 TransUNet 0.994 0.908 0.908 Proposed Algorithm 0.995 0.928 0.928
[0085] In order to further verify the impact of the compressed excitation void space pyramid pooling module in the embodiment of the present application, the embodiment of the present application conducted a pooling module ablation experiment. As above, the embodiment of the present application uses the self-made coal particle data as the training set and directly trains it in TransUNet. The obtained results are used as the baseline model results. Then, the compressed excitation void space pyramid pooling proposed in the embodiment of the present application is used, and the same coal particle data set is selected as the training set. The obtained results are shown in the figure below. Figure 5 As shown, from Figure 5 It can be seen that the compression-excited atrous spatial pyramid pooling module of the algorithm proposed in the embodiment of the present application achieves a good segmentation effect.
[0086] Finally, in order to more clearly illustrate the accuracy of coal particle size characteristic measurement, Figure 6 A set of experimental results of coal particle size measurement is shown. The horizontal axis in the figure represents the size of coal particles, which is divided into 10 intervals, and the width of each interval is 0.2mm. The blue bar represents the number of coal particles in each size interval, and the red curve shows the cumulative distribution. The experimental results show that the proportion of coal particles in the 1-3mm interval is as high as 97.49% and as low as 89.56%. This fully verifies the high accuracy of the proposed algorithm in measuring coal particle size.
[0087] This specification also provides a coal particle image segmentation device, comprising:
[0088] A model building module is used to replace the encoder of the original Unet network with a resnet50 residual network module, add a multi-branch Transformer model between the output end of the original Unet network encoder and the input end of the decoder, and replace the jump connection between the encoder and decoder of the original Unet network with a compressed excitation void spatial pyramid pooling module to build an improved unet network; train the improved unet network to obtain a coal particle image segmentation model;
[0089] The image segmentation module is used to input the coal particle image to be segmented into the coal particle image segmentation model. The resnet network extracts features from the coal particle image to be segmented through its N layers to obtain M local feature maps with successively decreasing scales; the multi-branch Transformer model extracts features from the local feature map with the smallest scale to obtain global features; the encoder upsamples the global features through its M layers to obtain local features with successively decreasing scales. Figure 1 A corresponding restoration image of M scales increasing in sequence; the restoration image of the Mth scale is obtained by splicing the restoration image of the M-1th scale and the key feature map of the M-1th scale and inputting them into the upper mining layer, and the key feature map of the M-1th scale is obtained by focusing on amplifying the features of the M-1th scale using a compressed excitation void space pyramid pooling module; the coal particle image to be segmented after segmentation is output based on the restoration image of the largest scale.
[0090] For the specific definition of the coal particle image segmentation device, please refer to the definition of the coal particle image segmentation method above, which will not be repeated here. Each module in the above-mentioned coal particle image segmentation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0091] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 2 The coal particle image segmentation method is provided.
[0092] This specification also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 2 The coal particle image segmentation method is provided.
[0093] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0094] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
Claims
1. A coal particle image segmentation method, characterized in that: include: A multi-branch Transformer model is added between the output end of the original Unet network encoder and the input end of the decoder, wherein the input end of each Transformer branch in the multi-branch Transformer model is simultaneously connected to the output end of the previous Transformer branch and the output end of the encoder; the jump connection between each layer of the encoder and decoder of the original Unet network is replaced with a compressed excitation void spatial pyramid pooling module to construct an improved unet network; the improved unet network is trained to obtain a coal particle image segmentation model; The coal particle image to be segmented is input into the coal particle image segmentation model, and the features of the coal particle image to be segmented are extracted through multiple downsampling layers of the encoder to obtain low-level pixel feature maps of the coal particle image to be segmented at multiple scales; the low-level pixel features related to the coal particles in the multiple low-level pixel feature maps are amplified through the compression-excitation void spatial pyramid pooling module to obtain multiple coal particle key pixel feature maps; the minimum scale low-level pixel feature map is feature extracted through multiple Transformer branches of the multi-branch Transformer model to obtain a multi-level coal particle semantic feature map; the multi-level coal particle semantic feature map is upsampled through multiple upsampling layers of the encoder to obtain multiple upsampling maps corresponding to the low-level pixel feature maps at multiple scales; the multiple coal particle key pixel feature maps are fused with the corresponding multiple upsampling maps to obtain restoration maps at multiple scales, and the segmented coal particle image to be segmented is output based on the restoration map at the maximum scale.
2. The coal particle image segmentation method according to claim 1, characterized in that: The improved unet network also includes replacing the encoder of the original Unet network with the first three layers of the resnet50 residual network.
3. The coal particle image segmentation method according to claim 2, characterized in that: The multi-branch Transformer model includes 12 Transformer branches, and the Transformer branches process the output feature map of the last layer of the resnet50 residual network and the output of the previous Transformer branch through a multi-head self-attention mechanism.
4. The coal particle image segmentation method according to claim 1, characterized in that: The compression-excitation atrous spatial pyramid pooling module uses four atrous convolutions with different dilation rates, specifically including: X1=f i×1 (X) X2=f 3x3,r=6 (X) X3=f 3×3,r=12 (X) X4=f 3×3,r=18 (X) X5=GAP(X) Y i =SENet(X i ) Y=Concat(Y1,Y2,Y3,Y4,Y5) Among them, X represents the input of the compressed excitation dilated spatial pyramid pooling module, X1~X4 represent the outputs of four dilated convolutions, and f n×n (·) represents a dilated convolution with a kernel size of n, r represents the dilation rate of the dilated convolution, GAP(·) represents global average pooling, SENet(·) represents the compressed-excited attention mechanism, and Y represents the output of compressed-excited dilated spatial pyramid pooling.
5. The coal particle image segmentation method according to claim 1, characterized in that: Before extracting features from the minimum-scale low-level pixel feature map through multiple Transformer branches of the multi-branch Transformer model, the minimum-scale low-level pixel feature map is segmented into N image blocks, and the N image blocks are mapped to a space of dimension D through a linear layer for semantic embedding: in, represents a two-dimensional feature sequence; X i is the image patch of the ith channel in the minimum-scale low-level pixel feature map, and the size of each image patch is P×P; is the projection matrix of the image block, and C is the number of channels of the image block; represents the position encoding matrix.
6. The coal particle image segmentation method according to claim 1, characterized in that: Before performing the upsampling operation on the multi-level coal particle semantic feature map through the multiple upsampling layers of the encoder, the size of the multi-level coal particle semantic feature map is adjusted to be the same as the size of the minimum scale low-level pixel feature map.
7. A coal particle image segmentation device, characterized in that: include: A model building module is used to add a multi-branch Transformer model between the output end of the original Unet network encoder and the input end of the decoder, wherein the input end of each Transformer branch in the multi-branch Transformer model is simultaneously connected to the output end of the previous Transformer branch and the output end of the encoder; the jump connection between each layer of the encoder and decoder of the original Unet network is replaced with a compressed excitation void spatial pyramid pooling module to construct an improved unet network; the improved unet network is trained to obtain a coal particle image segmentation model; The image segmentation module is used to input the coal particle image to be segmented into the coal particle image segmentation model, extract features of the coal particle image to be segmented through multiple downsampling layers of the encoder, and obtain low-level pixel feature maps of the coal particle image to be segmented at multiple scales; amplify the low-level pixel features related to coal particles in multiple low-level pixel feature maps through the compression-excitation void space pyramid pooling module to obtain multiple coal particle key pixel feature maps; extract features of the minimum-scale low-level pixel feature map through multiple Transformer branches of the multi-branch Transformer model to obtain a multi-level coal particle semantic feature map; upsample the multi-level coal particle semantic feature map through multiple upsampling layers of the encoder to obtain multiple upsampling maps corresponding to the low-level pixel feature maps at multiple scales; fuse multiple coal particle key pixel feature maps with the corresponding multiple upsampling maps to obtain restoration maps at multiple scales, and output the segmented coal particle image to be segmented based on the restoration map at the maximum scale.
8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Image segmentation method based on ResNet and UNet models
CN111145188A
Cervical cell segmentation network training method, cervical cell segmentation method and device
CN114240949A
Multi-resolution semantic segmentation method and device based on attention pyramid
CN114359297A
Remote sensing image semantic segmentation method based on double-branch feature fusion
CN115797931A
Transform and U-Net combined medical image liver segmentation method and system
CN115965633A
