Dust detection method based on lightweight pixel difference network

Abstract

The invention provides a dust detection method based on a lightweight pixel difference network. According to the method, a camera installed in a mine field is used for obtaining an environment image and preprocessing the environment image; and inputting the preprocessed data into the trained pixel difference network. The image preprocessing module comprises global gray transformation function processing optimized by a gravitational search algorithm (GSA), improved multi-scale Retinex (MSR) algorithm processing and gamma correction to avoid image color cast caused by image fusion. The improved lightweight pixel difference network comprises an improved depth separable convolution backbone network module, and an edge structure diagram of each stage is subjected to depth supervision and a loss function is improved; according to the pixel difference network, the improved depth separable convolution is used for replacing the traditional convolution in the backbone network to realize lightweight operation, and the maximum pooling layer is used for performing downlink sampling and generating the edge graph for performing depth supervision and refining the feature graph, so that the dust identification precision of the model and the generalization performance of the algorithm are improved.

Description

Technical Field

[0001] The invention relates to the field of dust detection, and specifically discloses a dust detection method based on a lightweight pixel difference network. Background Art

[0002] Edge detection is an important area in image processing and computer vision, especially in feature extraction, which aims to eliminate irrelevant information and retain important structural properties of the image. Feature extraction is performed by representing points with obvious changes in brightness in digital images to complete the actual problem to be solved. Dust detection technology plays a huge role in various complex real-life scenarios today, such as mines, factories, and disaster sites. Through dust detection technology, the location of image degradation phenomena such as decreased contrast and clarity, color distortion, etc. caused by dust accumulation in photos or videos can be accurately located, so as to eliminate interference in a targeted manner. This technology aims to detect dust in the acquired image, which plays an important role in locating dust areas, retaining original image information, and facilitating further detection.

[0003] Dust detection is one of the challenging tasks in edge detection methods. Dust detection provides technical support in ensuring that important information of the image is not blocked or lost in complex scenes. The lightweight pixel difference network adopts a novel pixel difference convolution, combining the traditional edge detection operator with the popular convolution operation in modern deep convolutional neural networks, and distinguishes the detection target from the complex background through the deep convolutional neural network to achieve the purpose of detection. However, there are many challenges in practical application. Dust detection needs to ensure high performance, but deep convolutional neural network detection requires a lot of pre-training, which will consume a lot of memory and energy.

[0004] In summary, the present invention proposes a dust detection method based on a lightweight pixel differential network to address the above problems. Summary of the invention

[0005] The present invention aims to provide a dust detection method based on a lightweight pixel difference network to achieve the purpose of high-performance detection of dust in images or videos.

[0006] The present invention is achieved through the following technical solutions:

[0007] A dust detection method based on a lightweight pixel difference network comprises the following steps:

[0008] S1. For the image data of coal mines, labelme software is used to annotate the image data, generate label maps corresponding to dust, use the prepared data set as the dust data set in complex scenes, and perform data preprocessing on the screened data set;

[0009] S2. The data preprocessing method is mainly through the image enhancement technology based on grayscale transformation and improved Retinex, which further highlights the dust information in the image while expanding the data set, so that the subsequent network recognition can achieve better results;

[0010] S3, input the data in the BRIND training set into the improved pixel difference network for training, and iterate repeatedly to obtain the optimal weight;

[0011] S4. Input the real-shot images in the coal mine environment into the trained pixel difference network to obtain the final detection results.

[0012] Furthermore, the data collection process described in S1 specifically includes the following contents:

[0013] In coal mine environments with different scenes and different dust concentrations, images taken with pre-deployed cameras or on-site photography are used as the initial data set.

[0014] Furthermore, the data preprocessing process described in S2 specifically includes the following contents:

[0015] The image captured by the camera is first transformed into grayscale by using the global grayscale transformation function optimized by the gravitational search algorithm (GSA) to enhance the image illumination intensity and make it closer to a uniform illumination scene. Then the image is converted into the HSV color space, and the V channel (brightness channel) is processed by the improved multi-scale Retinex (MSR) algorithm. The range-based adaptive bilateral filter and Gabor filter are used as the surround function of the Retinex algorithm, and the characteristics of the two filters are combined to enhance the brightness and details of the image. Finally, gamma correction is used to avoid image color deviation caused by image fusion. Then it is divided into training set, validation set and test set in a ratio of 8:1:1.

[0016] Furthermore, the process described in S3 specifically includes the following contents:

[0017] In the encoder part, the backbone feature extraction network adopts a depthwise separable convolution structure. The lightweight improvement mainly aims to solve the problem of matching the computing power of in-vehicle platforms, that is, replacing the traditional convolution used in the original feature extraction network with depthwise separable convolution. In the case of the same input and output and the same number of channels, compared with depthwise separable convolution, the computational amount of traditional convolution is much larger than the former, making the network model more lightweight. To make up for the loss of dust features caused by the reduction in the number of parameters, it is cascaded in the form of a residual branch in the depthwise separable convolution layer. Among them, a 3×3 traditional convolution block (including the ReLU activation function) is introduced before each depthwise separable convolution. Although the 3×3 convolution block will increase some computational amount, it has an obvious effect gain for dust feature extraction. During the channel (Pointwise) convolution and depth (Depthwise) convolution of the depthwise separable convolution, complex features can be well retained, and the outputs are concatenated to obtain a feature map. The feature map obtained in this way will retain more detailed features, such as fine features like discrete dust in the dust edge area, enhancing the expression ability of the model. In the decoder part, max pooling layers are used for downsampling, and there are 4 residual blocks in each stage (one of the residual blocks in the first stage is replaced with an initial convolution layer). The residual path in each block sequentially includes a depth convolution layer, a ReLU layer, and a point convolution layer. The number of channels in each stage is quite small to avoid the model size from being too large.

[0018] An edge map is generated through the edge structure in each stage. Based on this, the edge loss and the GT (groundtruth) map are calculated to provide depth supervision. To refine the feature map, at the end of each stage, first, a module based on compact dilated convolution (CDCM) is constructed to enrich multi-scale edge information. This module takes an n×C channel as input and produces M (M < C) channels in the output to reduce the computational overhead. Then, a compact spatial attention module (CSAM) is constructed to eliminate background noise. After that, the 1×1 convolution layer further reduces the feature volume to a single-channel map, then interpolates it to the original size, and then uses the Sigmoid function to create an edge map. The final edge map for testing is formed by fusing 4 single-channel feature maps with a concatenated convolution layer and the Sigmoid function. And considering that the training images may have uneven resolutions, for the simplicity of the overall structure, no normalization layer is used.

[0019] Improve the loss function to further improve the convergence of the network model; the original network uses a loss function mixed with the Dice Loss loss function, while the loss function used in this network training is mainly the Annotator-robust LossFunction. By taking the average of all ground truths, a new edge probability map is generated with a value range of [0, 1]. By setting the edge probability value A, the edge probability value exceeding A is used as a positive sample, and the probability value equal to 0 is used as a negative sample. In addition, if a desired probability value is less than A, it is semantically controversial to regard it as an edge point. Directly deleting this part of the pixels can avoid its impact on the network.

[0020] Beneficial Effects

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The advantages of the present invention are as follows: 1. Replacing the vanilla convolution with the improved pixel difference convolution can effectively reduce the model parameters and improve the effect of the model on detecting dust on different platforms; 2. Introducing a 3×3 convolution block (including ReLU) before the depth-separable convolution can improve the extraction effect of the complex features of dust and pay more attention to small features, thereby enhancing the expression ability of the model; 3. At each stage, an edge graph is generated through the edge structure, and the edge loss and GT (ground truth) map are calculated on this basis to provide deep supervision, which can accurately detect the dust in the image. 4. The global grayscale transformation function processing optimized by the gravitational search algorithm (GSA), the improved multi-scale Retinex (MSR) algorithm processing and the gamma correction are used to avoid the image color deviation caused by image fusion. Compared with various filtering and denoising methods of traditional edge detection, it has better effects on subtle feature objects such as dust.

[0023] In the process of network model training, the present invention introduces Annotator-robust Loss Function to replace the traditional loss function, which can make the network converge faster and solve category optimization problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the method of the present invention;

[0025] Figure 2 This is a schematic diagram of the data preprocessing process involved in the present invention;

[0026] Figure 3 A schematic diagram of a pixel difference convolution structure according to the present invention;

[0027] Figure 4It is a schematic diagram of the structure of an improved pixel difference network model involved in the present invention;

[0028] Figure 5 A schematic diagram of the structure of a compact extended convolution module (CDCM) according to the present invention;

[0029] Figure 6 A schematic diagram of the structure of a compact spatial attention module (CSAM) according to the present invention;

[0030] Figure 7 The photos taken on site of the coal mine involved in the present invention; Specific implementation plan

[0031] The present invention provides a dust detection method based on a lightweight pixel difference network. In order to make the technical solution and effect of the present invention clearer and more specific, the technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0032] Example

[0033] As shown in Figure (1), a dust detection method based on a lightweight pixel difference network includes:

[0034] S1. For the image data of coal mines, labelme software is used to annotate the image data, generate label maps corresponding to dust, use the prepared data set as the dust data set in complex scenes, and perform data preprocessing on the screened data set;

[0035] S2. The data preprocessing method is mainly through the image enhancement technology based on grayscale transformation and improved Retinex, which further highlights the dust information in the image while expanding the data set, so that the subsequent network recognition can achieve better results;

[0036] S3, input the data in the BRIND training set into the improved pixel difference network for training, and iterate repeatedly to obtain the optimal weight;

[0037] S4. Input the real-shot images in the coal mine environment into the trained pixel difference network to obtain the final detection results.

[0038] The specific method in this embodiment is described in detail below:

[0039] 1. Selecting Datasets

[0040] The dust image data comes from the real coal mine environment. It is collected in different work areas of the coal mine by pre-arranged cameras or mobile shooting devices to obtain the final data set. The collected and screened images are manually annotated with semantic segmentation areas in the image pixel area using labelme software. In order to facilitate dust identification, dust is divided into three categories according to different recognition levels and dust concentrations: low concentration, medium concentration, and high concentration. A json file with semantic segmentation information is obtained, and then the json file is converted into a grayscale image with a bit depth of 1.

[0041] 2. Data Preprocessing

[0042] As shown in Figure (2), the collected image is processed using a global grayscale transformation function. It transforms each pixel based on the similarity of density distribution between adjacent pixels. It can effectively enhance the grayscale image, well preserve the details of the original image, and also prevent the image from being blurred and degraded in the subsequent image enhancement process.

[0043] The global grayscale transformation function marks the grayscale value with the largest number of pixels as Max. When the left pixel approaches 0 and the right pixel approaches 255, the contrast increases. In addition, the contrast and brightness increase positively according to the similarity of the intensity distribution between adjacent pixels. Its transformation function is as follows:

[0044]

[0045] Where S(x,y) is the similarity between the pixel at (x,y) and the surrounding pixels; G max is the gray value with the largest number of pixels in the image; ψ(x,y) represents the standard deviation value in the 3×3 pixel area. Its formula is shown in (2):

[0046]

[0047] Where: m(x,y) is the average value of pixels in a 3×3 pixel area.

[0048] In order to determine the pixel similarity of a 3×3 pixel region, a relationship matrix is ​​established by the distance between pixels in the 3×3 pixel region. Assume that there are two pixels k and l in the 3×3 region, and the distance formula between them is as follows:

[0049] d kl =|P k -P l | (3)

[0050] Then the exponential function in formula (4) is used to calculate the similarity between two pixels, and its value distribution ranges from 0 to 1. If its value is close to 0, the two pixels are not similar, and if it is close to 1, they are considered similar.

[0051]

[0052] Where: D is the normalization coefficient, and the results can be graded by assigning values. The similarities between the 9 pixels in the 3×3 area are used to create a 9×9 similarity relationship matrix. Finally, the pixel area is translated until the similarity of the last pixel in the lower right corner of the image is calculated.

[0053] 3. Improve the network model

[0054] The pixel difference network (PiDiNet) has a significant effect on subtle feature targets, mainly due to the combined effect of its unique pixel difference convolution (PDC) design, multi-level feature fusion, efficient inference speed, and robust loss function. These advantages make PiDiNet perform well in edge detection tasks, especially when dealing with complex and subtle image structures, such as dust.

[0055] As shown in Figure (4), the backbone feature extraction network uses improved pixel difference convolution (PDC), replacing the traditional convolution used in the original feature extraction network with depthwise separable convolution. When the input and output are the same and the number of channels is the same, the traditional convolution is much more computationally intensive than the depthwise separable convolution, making the network model more lightweight. The lightweight improvement is mainly to solve the problem of matching the computing power of the vehicle platform. The difference between pixel difference convolution and original convolution is that when performing convolution operations, the original pixels in the local feature map patch covered by the convolution kernel are replaced by pixel differences. The functions of the original convolution and PDC can be expressed by (5)(6):

[0056]

[0057]

[0058] In the formula, x i and x i ′ is the input pixel, w i is the weight in the k×k convolution kernel. P is the set of pixel pairs selected from the current local patch.

[0059] The traditional convolution of its hidden layer is replaced by depthwise separable convolution. The characteristic of depthwise separable convolution is that channel (Pointwise) convolution is performed first and then depth (Depthwise) convolution, so that each channel pays more attention to the spatial information characteristics; at the same time, no activation function is required between channel (Pointwise) convolution and depth (Depthwise) convolution. The ReLU activation function is used before group convolution to form a convolution block. The output of every two such convolution blocks is subjected to maximum pooling to extract dust features and reduce the number of parameters and feature maps.

[0060] For the computational complexity of traditional convolution and depthwise separable convolution, assuming that the input feature map size is H×W, there are C channels, the convolution kernel size is K×K, and the number of output channels is the same as the number of input channels, the computational complexity of depthwise separable convolution is:

[0061] H×W×C×K×K+H×W×C×C(7) The computational complexity of traditional convolution is:

[0062] H×W×C×C×K×K(8)

[0063] Compared with depthwise separable convolution, traditional convolution has 8 times the computational complexity of the former, making the network model more lightweight.

[0064] In terms of encoder, Local Binary Pattern (LBP) and its robust variant Extended LBP (ELBP) are used to encode pixel relationships from different directions. ELBP introduces additional parameters based on LBP, including radius and number of neighborhood points, to increase the ability to describe texture features, which can enrich the detected dust details. And the use of circular neighborhoods instead of square neighborhoods is more in line with the characteristics of dust that needs to be detected.

[0065] By integrating the ELBP with the CNN convolution, we obtain three PDC instances as shown in Figure (3), which we name as central PDC (CPDC), angular PDC (APDC), and radial PDC (RPDC). The pixel pairs in the local patch are easy to understand. For example, for APDC with a kernel size of 3×3, 8 pairs are created in the angular direction of the 3×3 local patch (so m=8), and then the pixel differences obtained from these pairs are convolved with the kernel by element-wise multiplication with the kernel weights and then summed to generate the values ​​in the output feature map. Although this will increase the amount of calculation, it has a significant effect gain for dust feature extraction.

[0066] To learn rich hierarchical edge representations, we also use edge structures to generate an edge map from each stage, on which the edge loss is calculated against the ground truth map to provide depth supervision. To refine the feature maps, starting from the end of each stage, a Compact Dilated Convolution Module (CDCM) is first constructed to enrich multi-scale edge information. This module takes an n×C channel input and produces M (M < C) channels in the output to reduce the computational overhead. Then, a Compact Spatial Attention Module (CSAM) is constructed to eliminate background noise and suppress irrelevant noise. After that, a 1×1 convolutional layer further reduces the feature volume to a single-channel map, which is then interpolated to the original size, and then a Sigmoid function is used to create the edge map. The final edge map for testing is formed by fusing 4 single-channel feature maps with concatenation, convolutional layers, and the Sigmoid function.

[0067] IV. Training the Model

[0068] Traditional loss functions, such as cross-entropy, mean square, etc., perform poorly in the case of data class imbalance. The smaller classes will be overwhelmed by the larger classes, resulting in a high prediction accuracy during training but a poor application efficiency. To match the edge maps generated by the adopted edge structure, the Annotator-robust Loss Function is used during network training. By taking the mean of all ground truths, a new edge probability map is generated, and the most ideal training effect is obtained by setting the edge probability value. For the i-th pixel in the j-th edge map, the pixel value is The loss is calculated as:

[0069]

[0070] where y i is the true edge probability, η is a predefined threshold. This means that when calculating the loss, if a pixel is marked as positive by fewer than η annotators, the pixel is discarded and not considered as a sample to avoid confusion. β is the percentage of negative pixel samples, and α = λ·(1 - β). The total loss is

[0071] To prevent the feature extraction effect from being poor due to the overly random weights of the backbone network and ensure that the model can converge faster. First, pre-training is performed on BRIND. The training set, validation set, and test set of BRIND consist of 200, 100, and 200 images respectively. Each image has 4 to 9 annotators. The training images in the dataset are augmented by flipping (2 times), scaling (3 times), and rotating (16 times), resulting in the training set being 96 times larger than the non-augmented version. After obtaining the appropriate weights, the dust images in the coal mine are then detected. This can reduce the final number of training times for dust detection and improve the detection efficiency.

[0072] The above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A dust detection method based on lightweight pixel difference network, It is characterized in that The described method is implemented by the following steps: S1. For the image data of coal mines, labelme software is used to annotate the image data, generate label maps corresponding to dust, use the prepared data set as the dust data set in complex scenes, and perform data preprocessing on the screened data set; S2. The data preprocessing method is mainly through the image enhancement technology based on grayscale transformation and improved Retinex, which further highlights the dust information in the image while expanding the data set, so that the subsequent network recognition can achieve better results; S3, sending the preprocessed data to the improved pixel difference network for training, and repeatedly iterating to obtain the optimal weight; S4. Input the real-shot images in the coal mine environment into the trained pixel difference network to obtain the final detection results.

2. The dust detection method based on a lightweight pixel difference network as claimed in claim 1, It is characterized in that The step S2 comprises the following sub-steps: The image captured by the camera is first transformed into grayscale by using the global grayscale transformation function optimized by the gravitational search algorithm (GSA) to enhance the image illumination intensity and make it closer to a uniform illumination scene. Then the image is converted into the HSV color space, and the V channel (brightness channel) is processed by the improved multi-scale Retinex (MSR) algorithm. The range-based adaptive bilateral filter and Gabor filter are used as the surround function of the Retinex algorithm, and the characteristics of the two filters are combined to enhance the brightness and details of the image. Finally, gamma correction is used to avoid image color deviation caused by image fusion. Then it is divided into training set, validation set and test set in a ratio of 8:1:

1.

3. The dust detection method based on a lightweight pixel difference network as claimed in claim 1, It is characterized in that The step S3 includes the following contents: In the encoder part, the backbone feature extraction network adopts a depthwise separable convolution structure. The lightweight improvement mainly aims to solve the problem of computing power on different platforms, that is, replacing the traditional convolution in its hidden layer with depthwise separable convolution. When the input and output are the same and the number of channels is also the same, compared with depthwise separable convolution, the computational amount of traditional convolution is much larger than the former, making the network model more lightweight. To make up for the loss of dust features caused by the reduction of the number of parameters, it is cascaded in the form of a residual branch in the depthwise separable convolution layer. Among them, a 3×3 traditional convolution block (including the ReLU activation function) is introduced before each depthwise separable convolution. Although using a 3×3 convolution kernel will increase a certain amount of computational effort, it can significantly improve the effect in dust feature extraction. During the pointwise convolution and depthwise convolution of the depthwise separable convolution, complex features can be well retained, and the outputs are concatenated to obtain a feature map. The feature map obtained in this way will retain more detailed features, such as fine features like discrete dust in the dust edge area, enhancing the expression ability of the model. In the decoder part, max pooling layers are used for downsampling, and there are 4 residual blocks in each stage (one of the residual blocks in the first stage is replaced by an initial convolution layer). The residual path in each block sequentially includes a depth convolution layer, a ReLU layer, and a point convolution layer. The number of channels in each stage is quite small to avoid the model size from being too large. An edge map is generated through the edge structure in each stage. Based on this, the edge loss and the GT (ground truth) map are calculated to provide depth supervision. To refine the feature map, at the end of each stage, first, a module based on compact dilated convolution (CDCM) is constructed to enrich multi-scale edge information. This module takes channels as input and generates M (M < C) channels in the output to reduce the computational overhead. Then, a compact spatial attention module (CSAM) is constructed to eliminate background noise. After that, the convolution layer further reduces the feature volume to a single-channel map, then interpolates it to the original size, and then uses the Sigmoid function to create an edge map. The final edge map for testing is formed by fusing 4 single-channel feature maps with a concatenated convolution layer and the Sigmoid function. And considering that the resolution of training images may be uneven, for the simplicity of the overall structure, no normalization layer is used. Improve the loss function to further improve the convergence of the network model; the original network uses a loss function mixed with the Dice Loss loss function, while the loss function used in this network training is mainly the Annotator-robust LossFunction. By taking the average of all ground truths, a new edge probability map is generated with a value range of [0, 1]. By setting the edge probability value A, the edge probability value exceeding A is used as a positive sample, and the probability value equal to 0 is used as a negative sample. In addition, if a desired probability value is less than A, it is semantically controversial to regard it as an edge point. Directly deleting this part of the pixels can avoid its impact on the network.

Images