A method for identifying a cutting image by improving a real-time semantic segmentation network

Abstract

The present application provides a kind of improved real-time semantic segmentation network's cutting image identification method. With the help of BiSeNetV1 real-time model, through the extended spatial path convolution, spatial pyramid attention mechanism (SPARM) and simplified attention feature fusion module (S-iAFF), a kind of real-time network for cutting image identification is designed. The extended spatial path structure obtains more rich spatial features, the context path uses the optimized spatial pyramid attention to refine the high-level semantic feature extraction, and the feature fusion uses the simplified attention feature fusion to strengthen the fusion degree. Compared with the original BiSeNetV1 segmentation precision, the method has greater improvement, and compared with the high-precision method DeepLabV3+ in the case of little difference in accuracy, the network inference speed is greatly improved. The present application can be widely used in oil and other underground energy exploitation, lunar energy exploration and other fields.

Description

Technical Field

[0001] This invention relates to the problem of real-time segmentation and recognition of rock cuttings images in the field of computer vision, and in particular to a real-time segmentation method for rock cuttings images using an improved BiSeNetV1 network. Background Technology

[0002] Rock cuttings image recognition is a crucial method for determining the oil content of geological formations. In recent years, with the improvement of living standards, human demand for energy has been increasing. Oil and other energy sources are important indicators of national economic development, making related technologies for their development and utilization increasingly important. In oil exploration technology, the analysis of geological parameters during drilling operations can quickly determine the oil content of a location. Workers collect initial rock cuttings samples from the wellhead, followed by a series of processing steps, including cleaning, to obtain rock cuttings samples suitable for identification. Finally, an image acquisition system is used to obtain rock cuttings images for subsequent identification. Currently, rock cuttings recognition technology has a wide range of applications, including coal, oil, natural gas underground energy extraction, and lunar energy exploration, where its application is essential.

[0003] With technological advancements, deep learning is gradually replacing traditional image processing algorithms in the classification and recognition of rock debris images. In particular, the application of semantic segmentation algorithms has significantly improved the classification results of rock debris images, reducing the time required for manual identification. Semantic segmentation is a typical research direction in computer vision. Its pixel-level classification method can effectively associate the label category of each pixel in different scenes, ultimately achieving pixel-level classification tasks through different annotation results. It is widely used in fields such as face recognition, autonomous driving, and medical image processing. These fields, while pursuing segmentation accuracy, also have high requirements for real-time processing. The development of semantic segmentation has been gradual, with increasingly higher segmentation accuracy in classic training scenarios. However, most of this improved training accuracy comes at the cost of increased parameters and training time. Especially after reaching a certain level of training accuracy, it often happens that the improved complex model only increases accuracy by 1% compared to the previous one, but the training cost (such as computation, time, and memory usage) increases exponentially. Based on this situation, the development of real-time semantic segmentation has gained attention in recent years. Considering certain application scenarios, research on real-time semantic segmentation mainly focuses on simplifying the number of parameters and memory usage in the training model while maintaining segmentation accuracy as much as possible.

[0004] In recent years, two common methods have been used to accelerate real-time segmentation algorithms: one is to reduce the size of the input image by cropping to decrease training complexity, a very simple method that destroys the spatial features of the image; the other is to reduce the number of channels in the network during training to reduce inference complexity, but this also weakens the network's spatial capabilities. While these methods effectively reduce model complexity, they sacrifice the network's spatial feature extraction ability, significantly reducing recognition accuracy. Based on this, the BiSeNet bilateral segmentation network proposed by Changqian Yu, Jingbo Wang, and others in 2018 effectively solves the above problems. It consists of two parts: a spatial path and a contextual path to address the loss of spatial information and the reduction in receptive field.

[0005] This invention, combining the characteristics of rock debris image recognition and relevant training data, reveals that BiSeNet achieves high matching accuracy for rock debris images. The BiSeNetV1 model is improved in three aspects: spatial path, contextual path, and feature fusion module. This improves upon the original lightweight model's superior feature extraction capabilities, resulting in a more suitable feature extraction and fusion model for rock debris images. Specifically, the spatial path uses five convolutional layers instead of three to extract richer spatial features. The contextual path uses an improved SPARM (spatial pyramid attentional refinement mechanism) instead of the ARM (attentional refinement mechanism) to extract contextual semantic features from rock debris images. Finally, the S-iAFF (Simplified-iterative attentional fusion module) fusion method replaces the FFM (Feature fusion module) to fuse the extracted spatial and contextual semantic features. Summary of the Invention

[0006] This invention proposes a real-time segmentation method for rock debris images based on an improved BiSeNetV1 network. The BiSeNetV1 network extracts low-level spatial information and high-level semantic information from rock debris images through spatial paths and contextual paths, respectively. The improved strategy first extracts richer spatial features from rock debris images by expanding the spatial path of BiSeNetV1. For the contextual path, an optimized Spatial Pyramid Attention (SPARM) mechanism is used to further refine the extraction of high-level semantic features. In the feature fusion stage, a simplified attention feature fusion (S-iAFF) is used to replace the original FFM fusion module to enhance the fusion degree of low-level spatial and high-level semantic features. Attached Figure Description

[0007] Figure 1Overall segmentation process for rock debris images.

[0008] Figure 2 An improved BiSeNetV1 network model for real-time segmentation of rock debris images.

[0009] Figure 3 Extended spatial path structure.

[0010] Figure 4 The structure of the space pyramid attention module (SPA).

[0011] Figure 5 An optimized Spatial Pyramid Attention Mechanism (SPARM) module.

[0012] Figure 6 Simplified Multi-scale Channel Attention Module (SMS-CAM).

[0013] Figure 7 S-iAFF feature fusion module. Detailed Implementation

[0014] The present invention will now be described in detail with reference to the accompanying drawings, including the following steps:

[0015] To capture sufficient spatial information and refine contextual information extraction from images, high-precision neural network methods often increase the receptive field during convolution, such as through pooling or dilated convolution. However, increasing the receptive field also increases training costs. Analysis reveals that the image features in the rock debris dataset differ from those in the classic city scene dataset, Cityscapes. Global spatial information is crucial in city scenes because the shapes of categories are diverse, and even within the same category, there are differences in color and shape. Different types of objects vary significantly in size, requiring sufficient global spatial information to aid in determining pixel information in a given area. In contrast, rock debris images, on the whole, show less difference in shape and size among the same type of rock debris. The similarity in shape, size, color, and texture details within the same category is high, and the edge information of overlapping rock debris types is complex. Therefore, it is necessary to expand the convolutional structure of spatial paths and refine the feature extraction capabilities of contextual paths to ensure that feature mapping focuses more on edge features.

[0016] According to the appendix Figure 1 The process of processing rock cuttings images is as follows:

[0017] (1) Each original rock debris image with a resolution of 4000×3750 was cropped in sequence (the boundaries of each cropped image overlapped) into 30 rock debris images with a resolution of 512×512. The cropped images were used to train and test the model, thereby improving the model's reasoning ability.

[0018] (2) As attached Figure 2 The input image is fed into an extended 5-layer spatial path convolutional structure to extract richer low-level spatial feature information. The extension method is as follows: while maintaining the spatial size of the original input image, the spatial structure is extended to 5 layers. At the same time, the number of channels, kernel size, stride and other parameters of each layer are controlled to reduce the increase in parameters. The convolution process uses padding operation to keep the output feature mapping to 1 / 8 of the original image. The extended structure can extract more spatial information from the original image. The specific details are as follows: The first convolutional layer uses a 7×7 kernel, a stride of 2, and 64 channels; the second convolutional layer uses a 3×3 kernel, a stride of 2, and 64 channels; the third convolutional layer uses a 3×3 kernel, a stride of 2, and 64 channels; the fourth convolutional layer uses a 3×3 kernel, a stride of 1, and 128 channels. The third and fourth layers are extended two-layer convolutions; the fifth convolutional layer uses a 1×1 kernel, a stride of 1, and 128 channels. This five-layer convolutional structure outputs the low-level spatial information structure, as shown in the attached diagram. Figure 3 As shown, the final output is the extracted spatial feature X.

[0019] Additionally, the input image is fed into the context path, where it is used as shown in the attached diagram. Figure 5 The Spatial Pyramid Attention (SPARM) mechanism, as shown, refines contextual semantic features. (See attached image.) Figure 4 This paper employs a spatial pyramid structure composed of multiple adaptive global average pooling methods instead of the original single global average pooling method. Using three adaptive global average pooling methods allows for more efficient utilization of semantic information. The 1×1 adaptive global average pooling is a traditional global average pooling method with structure regularization; the 2×2 adaptive global average pooling balances the relationship between structural information and structure regularization; and the 4×4 adaptive global average pooling captures more feature representations and structural information. Compared to the original single global average pooling, this pooling method can balance the regularization effect of the feature map with the feature representation and structural information of the original features, while inheriting the advantages of global average pooling. The attention module SPARM uses... Figure 4The Spatial Pyramid Structure (SPS) replaces the standalone adaptive global average pooling (SPA), discarding the two fully connected perceptual layers in the SPA structure. This SPARM attention module more comprehensively captures global contextual information, including feature regularization effects and structural information. By calculating the attention vector, it guides feature learning, thereby refining the output feature Y in the downsampling stage of the context path. Figure 5 The connection path of the SPARM attention module is shown.

[0020] (3) In order to better integrate the extracted spatial and contextual semantic information, the following method is used: Figure 7 The iterative attention feature fusion module (S-iAFF) is used to enhance the feature fusion effect. Firstly, considering that the fusion process might ignore most of the image information of small and medium-sized objects, the following module is introduced: Figure 6 The Simplified Multi-Scale Channel Attention Module (SMS-CAM) aggregates multi-scale features within the attention module to mitigate problems caused by scale variations and small to medium-sized objects.

[0021] Assuming the output of the global branch in the SMS-CAM module is G(Xin) and the output of the local branch is L(Xin), the output X of the spatial path is first obtained by global average pooling to obtain g(X), and the calculation result is as follows:

[0022]

[0023] In the formula, H×W represents the size of the feature map.

[0024] The output G(Xin) of the global context and the output L(Xin) of the local context are calculated as follows:

[0025] G(Xin)=δ(B(pwconv(g(X)))) (2)

[0026] L(Xin)=δ(B(pwconv(X))) (3)

[0027] In the formula, δ represents batch normalization (BN), B represents the linear rectified function ReLU, and pwconv represents pointwise convolution.

[0028] Given a global context and a local context, the SMS-CAM module can produce the following results:

[0029]

[0030] In the formula, σ represents the Sigmoid function, SMS represents the output through the SMS-CAM module, and ⊕ represents broadcasting addition.

[0031] A two-stage SMS-CAM attention fusion method, termed S-iAFF, is adopted. Since a single-stage SMS-CAM cannot effectively offset the influence of the initial fusion, the S-iAFF fusion method of two-stage SMS-CAM can significantly reduce the impact of the initial fusion on the final fusion, allowing the fusion process to focus more on the two-stage SMS-CAM. The calculation process of input features X and Y through S-iAFF is as follows:

[0032] XΦY=SMS(X+Y)X+(1-SMS(X+Y))Y (5)

[0033] F=SMS(XΦY)X+(1-SMS(XΦY))Y (6)

[0034] In the formula, XФY represents the output after one stage of SMS-CAM, and F represents the final output after two stages of SMS-CAM. Figure 7 The structure of the S-iAFF module is shown.

[0035] (4) Since the resolution of the rock debris images initially collected is 4000×3750, and the input image of the training process is a set of sub-images with a resolution of 512×512 after cropping, the final output after training is a 512×512 sub-effect image corresponding to the input image. In order to more intuitively show the effect of rock debris segmentation, the effect images output from the neural network are then spliced ​​together according to the image cropping order in (1) to synthesize the segmentation result of the original image with an original resolution of 4000×3750.

[0036] To verify the feasibility of this method, we conducted tests on different spatial paths, contextual attention mechanisms, fusion modules, and different networks. Method 1 represents the original BiSeNetV1, Method 2 represents a 5-layer spatial structure extension of Method 1, Method 3 uses a spatial pyramid attention module (SPARM) based on Method 2, and Method 4 is a high-precision DeepLabV3+ network trained under the same training details. The verification results are shown in Table 1.

[0037] Table 1 Comparison of training efficiency and training accuracy of different methods

[0038]

[0039] The experimental comparison in Table 1 shows that optimizing the details of different modules of the BiSeNetV1 network significantly improves the segmentation accuracy. Compared with the high-precision semantic segmentation method DeepLabV3+, the number of parameters is significantly reduced while the segmentation accuracy is not much different, and the network inference speed is greatly improved.

Claims

1. An improved real-time semantic segmentation network-based method for rock cuttings image recognition, characterized in that... Includes the following steps: (1) Each original rock debris image is cropped into multiple smaller rock debris images of the same size; (2) The cropped rock debris image is input into the spatial path convolution structure in the improved BiSeNetV1 network, and the low-level spatial features X of the extracted rock debris image are output; at the same time, the rock debris image is input into the context path structure in the network using the spatial pyramid attentional refinement mechanism (SPARM), and the high-level contextual semantic features Y of the extracted rock debris image are output. The spatial path extends the 3-layer convolutional structure to a 5-layer structure. The first convolutional layer uses a 7×7 kernel, a stride of 2, and 64 channels. The second convolutional layer uses a 3×3 kernel, a stride of 2, and 64 channels. The third convolutional layer uses a 3×3 kernel, a stride of 2, and 64 channels. The fourth convolutional layer uses a 3×3 kernel, a stride of 1, and 128 channels. The third and fourth layers are extended two-layer convolutions. The fifth convolutional layer uses a 1×1 kernel, a stride of 1, and 128 channels. Finally, the low-level spatial information X is output through the five-layer convolutional structure. The context path adopts a spatial pyramid structure composed of multiple adaptive global average pooling methods. It uses three adaptive global average pooling methods: 1×1, 2×2, and 4×4, which together constitute the SPS (spatial pyramid structure) spatial pyramid pooling structure. (3) Based on the X and Y image features obtained from (2), the extracted low-level spatial features and high-level contextual semantic features of the image are fused through the S-iAFF (Simplified-iterative attentional fusion module) attention feature fusion module, and the output information is unified to the input image size through upsampling; (4) The 512×512 effect image output from the neural network is combined by image stitching to synthesize the segmentation result of the corresponding original image with a resolution of 4000×3750.

2. The method according to claim 1, characterized in that... In step (1), the original rock debris image is cropped into multiple smaller rock debris images with the same resolution. The specific cropping method is as follows: The original 4000×3750 rock debris image was sequentially cropped into 30 512×512 rock debris images. The boundaries of each cropped image overlapped. The cropped images were then used for model training and testing to reduce the running cost of model training and testing.

3. The method according to claim 1, characterized in that... Step (3) uses the S-iAFF attention feature fusion module to fuse the extracted low-level spatial features and high-level contextual semantic features of the image. The operational details are as follows: Firstly, considering that the fusion process will ignore most of the image information of small and medium-sized objects, a simplified multi-scale channel attention module (SMS-CAM) is introduced to aggregate multi-scale features within the attention module to alleviate the problems caused by scale changes and small and medium-sized objects. Assuming the output of the global branch in the SMS-CAM module is G(Xin) and the output of the local branch is L(Xin), the output X of the spatial path is first obtained by global average pooling to obtain g(X), and the calculation result is as follows: (1) In the formula, H×W represents the size of the feature map; The output G(Xin) of the global context and the output L(Xin) of the local context are calculated as follows: (2) (3) In the formula, δ represents batch normalization (BN), B represents the linear rectified function ReLU, and pwconv represents pointwise convolution; Given the global and local context outputs, the SMS-CAM module can produce the following results: (4) In the formula, σ represents the Sigmoid function, SMS represents the output through the SMS-CAM module, and ⊕ represents broadcasting addition; Furthermore, considering the significant semantic differences between the feature information of spatial paths and context paths, a two-stage SMS-CAM attention fusion method, termed S-iAFF, is adopted. Since a single-stage SMS-CAM cannot effectively offset the influence of the initial fusion, the two-stage SMS-CAM S-iAFF fusion method can significantly reduce the impact of the initial fusion on the final fusion weights, allowing the fusion process to focus more on the two-stage SMS-CAM. The calculation process of input features X and Y through S-iAFF is as follows: (5) (6) In the formula, XФY represents the output after one stage of SMS-CAM, and F represents the final output after two stages of SMS-CAM.

4. The method according to claim 1, characterized in that... In step (4), the 512×512 effect image output from the neural network is spliced ​​together according to the image cropping order in step (1) to synthesize the segmentation result of the corresponding original image with a resolution of 4000×3750, making the final overall segmentation effect more intuitive.

Images