An image super-resolution reconstruction method for a wearable helmet

Abstract

This invention discloses an image super-resolution reconstruction method for wearable helmets. The method includes the following steps: constructing a degradation model to obtain high- and low-resolution image pairs; constructing a deep separable convolutional neural network (DCNN) model to generate super-resolution images; training the DCNN model by calculating loss between the super-resolution images and the input high-resolution images, and optimizing the DCNN model; testing the reconstructed super-resolution images; preprocessing the tested images; and validating the DCNN model. This invention effectively addresses image acquisition in low-light environments by using a degradation model to reconstruct high-resolution images; and by constructing a DCNN model to retain more texture details and reconstruct fine images.

Description

Technical Field

[0001] This invention belongs to the field of image enhancement technology, and specifically relates to an image super-resolution reconstruction method for wearable helmets. Background Technology

[0002] With social development and technological progress, industrial metallurgical operations are becoming increasingly intelligent, with wearable smart helmets already widely used in industry. For example, in mines, wearable helmets, while ensuring safety, incorporate various intelligent auxiliary detection methods such as coal mine inspection, slurry bubble characteristic detection, and foreign object detection. These can replace manual high-intensity, high-risk, and repetitive labor, reducing labor intensity, lowering accident risks, and resulting in significant ecological, social, and economic benefits. However, due to the low-configuration cameras in wearable helmets and the low-light conditions in coal mines, the images captured by the cameras are often blurry and low-resolution. This presents significant challenges for smart helmet inspections, sometimes even preventing the next step from proceeding.

[0003] Currently, the most studied and effective image super-resolution reconstruction method is the deep learning-based reconstruction method. Although the current deep learning-based reconstruction technology has achieved excellent results in subjective evaluation metrics, the reconstructed images still have problems such as loss of high-frequency details, image smoothing, and artifacts. Therefore, super-resolution image reconstruction technology still needs further improvement.

[0004] The image super-resolution reconstruction method based on multi-scale residual feature fusion, disclosed in China (application number: 202111250771.2; publication number: CN 114187174A), includes the following steps: S1: Acquire images of different resolutions and preprocess them to obtain high- and low-resolution image pairs; S2: Construct a multi-scale feature extraction module based on depthwise separable convolution to extract features from the preprocessed high- and low-resolution image pairs and output feature maps; S3: Construct a residual feature fusion module to perform residual feature fusion processing on the output feature maps; S4: Construct an enhanced attention module to process the feature maps after residual feature fusion; S5: Use an adaptive upsampling module to upsample the feature maps output in step S4 to generate a super-resolution image; S6: Construct a Charbonnier-based... The loss function module of Loss is used to process the super-resolution image; S7: Construct a super-resolution image reconstruction model based on multi-scale residual feature fusion, and input the super-resolution image processed in step S6 into the super-resolution reconstruction model for training; S8: Input the image to be processed into the super-resolution image reconstruction model based on multi-scale residual feature fusion for processing, and obtain the super-resolution reconstructed image information of the image to be processed.

[0005] The aforementioned patent's image super-resolution reconstruction method based on multi-scale residual feature fusion still has the following shortcomings: insufficient utilization of scale information, weak channel interaction capability of depth-separable convolution, which makes the algorithm for generating super-resolution images complex and results in poor quality of generated super-resolution images. Summary of the Invention

[0006] To address the aforementioned shortcomings, this invention provides an image super-resolution reconstruction method for wearable helmets.

[0007] The objective of this invention is achieved as follows: a method for image super-resolution reconstruction for wearable helmets, characterized in that the method includes the following steps:

[0008] Step 1: Construct a degradation model to obtain high- and low-resolution image pairs;

[0009] Step 2: Construct a deep separable convolutional neural network model to generate super-resolution images;

[0010] Step 3: Train the deep separable convolutional neural network model, calculate the loss between the super-resolution image and the input high-resolution image, and optimize the deep separable convolutional neural network model;

[0011] Step 4: Test the depthwise separable convolutional neural network model on the reconstructed super-resolution image;

[0012] Step 5: Preprocess the image and input it into a depthwise separable convolutional neural network model for verification;

[0013] Step 6: Validate the depthwise separable convolutional neural network model, evaluate the depthwise separable convolutional neural network model, and assess the quality standards of the reconstructed super-resolution image.

[0014] Preferably, the construction of the degradation model includes the following steps:

[0015] Step 1-1: Construct a dataset consisting of real images from underground mines;

[0016] Steps 1-2: Convert the real image from the mine to the HSV color space, and randomly reduce the values ​​of the saturation (S) and brightness (V) channels to simulate a low-light image;

[0017] Steps 1-3: Degradation operation is performed on the low-light images in the dataset. The high-resolution images in the dataset are downsampled by a scale factor of 2x, 3x, and 4x using the bicubic linear interpolation method to obtain the corresponding low-resolution images.

[0018] The degradation operation is represented by the following formula:

[0019] x=D(y=(y HSV ×s)↓ r

[0020] Where x represents the degraded image, y represents the original image, D represents the degradation function, s is the reduction factor, and r represents the scaling factor.

[0021] Preferably, the construction of the deep separable convolutional neural network model, which generates super-resolution images, includes the following steps:

[0022] Step 2-1: Constructing a shallow feature extraction module for a depthwise separable convolutional neural network model, performing initial feature extraction on the input image to obtain a shallow feature extraction map;

[0023] Step 2-2: Construct a feature fusion residual group module for a deep separable convolutional neural network model to extract deep features from the extracted shallow feature maps, thus obtaining deep feature maps;

[0024] Steps 2-3: Construct an image reconstruction module of a depthwise separable convolutional neural network model, upsample the deep feature maps, and then generate a super-resolution image through a standard 3×3×3 convolution.

[0025] Preferably, in step 3, the training of the depth-separable convolutional neural network model is carried out by constructing a training set, which uses real images from underground mines; wherein, the high-resolution images HR in the dataset are downsampled by a scale factor of 2x, 3x, and 4x using a bicubic linear interpolation method to obtain the corresponding low-resolution images LR; the reconstructed super-resolution images are then compared with the input high-resolution images to calculate the loss.

[0026] Preferably, in step 4, the test depth-separable convolutional neural network model is performed by constructing a test set, which includes a low-resolution image LR with bicubic linear interpolation downsampling scale factors of x2, x3, and x4, and the corresponding high-resolution image HR.

[0027] Preferably, in step 5, the images after testing are preprocessed, the patch size is set, and the high- and low-resolution image pairs in the training set are cropped at random distances starting from the image center position. The training set is then saved in .npy format as input to the depthwise separable convolutional neural network model.

[0028] Preferably, in step 6, the depthwise separable convolutional neural network model is verified by constructing a verification set;

[0029] Criteria for evaluating the quality of super-resolution image reconstruction in the Y channel transformed to YCbCr space using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity-SSIM:

[0030]

[0031] Higher PSNR and SSIM scores indicate that the reconstructed image is closer to the original image.

[0032] Preferably, the feature fusion residual group module for constructing the deep separable convolutional neural network model in step 2-2, which performs deep feature extraction on the extracted shallow feature map to obtain the deep feature map, includes the following steps:

[0033] Step 2-2-1: Construct the deep separable convolutional channel attention residual module of the deep separable convolutional neural network. Input the shallow feature map and use a single deep convolution with a kernel size of 96×7×7, a stride of 1, and padding of 3 to obtain the spatial feature relationship of the image through a large receptive field.

[0034] Step 2-2-2: Construct the channel attention module of the deep separable convolutional neural network, perform average pooling on the spatial dimension, learn the channel attention through two fully connected layers, and use Sigmoid for layer normalization to obtain the channel attention feature vector. Multiply the channel attention feature vector with the feature vector after depth group convolution to obtain the weighted depth convolution feature, and then use layer normalization again.

[0035] The feature map with channel attention weighting is subjected to a point convolution with a kernel of 384×1×1 to increase its dimensionality. After applying the GELU activation function, a point convolution with a kernel of 96×1×1 is applied to reduce its dimensionality, thereby improving the channel interaction capability of the depth convolution. The input feature map and the residual extracted feature map are then added together via short connections to obtain the output feature map.

[0036] Step 2-2-3: Construct a feature fusion module for a deep separable convolutional neural network. The feature fusion module consists of six deep separable convolutional channel attention residual modules connected in series. The feature maps output by the six deep separable convolutional channel attention residual modules are fused along the channel dimension. Then, a 96×3×3 convolution kernel is used for feature recombination. The long connections of the input feature map and the residual extracted feature map are added together to form the feature fusion module. The feature fusion module utilizes the complementarity between features, fuses the advantages between features, eliminates redundant information caused by the correlation between different features, and improves the performance of the model.

[0037] Step 2-2-4: Construct a depthwise separable convolutional channel attention fusion residual group module. This module consists of four feature fusion modules connected in series. The input feature map and the residual extracted feature map are added together via long connections to form the depthwise separable convolutional channel attention fusion residual group module. This module utilizes residual groups as the foundation for deeper networks. Information can flow within each residual group, obtaining more useful information and focusing on richer features. Through residual learning, the model achieves better performance.

[0038] Preferably, the reconstructed super-resolution image and the high-resolution image are subjected to loss calculation, and the loss calculation adopts the following formula:

[0039]

[0040] Where L1 is the average loss, x HR For the high-resolution original image, y SR To reconstruct images for super-resolution.

[0041] The beneficial effects of this invention are: 1. By using a degradation model to map the features of low-light images to normal-light images, it is possible to effectively handle image acquisition in low-light environments and reconstruct high-resolution images.

[0042] 2. By constructing a deep separable convolutional neural network model, it is helpful to capture multi-scale contextual information and reconstruct detailed images; the feature fusion module can deeply fuse the features learned by each convolution, learn richer contextual information, and retain more texture details.

[0043] 3. By constructing a channel attention module of a deep separable convolutional neural network and using Sigmoid for layer normalization to obtain the channel attention feature vector, the training time is reduced and the training speed of the network is accelerated. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention.

[0045] Figure 2 This is a diagram of the depthwise separable convolutional channel attention residual module.

[0046] Figure 3 This is a diagram of the feature fusion module.

[0047] Figure 4 Module diagram for depthwise separable convolutional channel attention fusion residual group Detailed Implementation

[0048] The invention will be further summarized below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, an image super-resolution reconstruction method for wearable helmets includes:

[0050] Step 1: Construct a degradation model to obtain high- and low-resolution image pairs;

[0051] Degradation models include:

[0052] Step 1-1: Construct a dataset consisting of real images from underground mines;

[0053] Steps 1-2: Convert the real image from the mine to the HSV color space, and randomly reduce the values ​​of the saturation (S) and brightness (V) channels to simulate a low-light image;

[0054] Steps 1-3: Degradation operation is performed on the low-light images in the dataset. The high-resolution images in the dataset are downsampled by a scale factor of 2x, 3x, and 4x using the bicubic linear interpolation method to obtain the corresponding low-resolution images.

[0055] The degradation operation is represented by the following formula:

[0056] x=D(y=(y HSV ×s)↓ r

[0057] Where x represents the degraded image, y represents the original image, D represents the degradation function, s is the reduction factor, and r represents the scaling factor.

[0058] Step 2: Construct a deep separable convolutional neural network model to generate super-resolution images;

[0059] Step 2-1: Constructing a shallow feature extraction module for a depthwise separable convolutional neural network model, performing initial feature extraction on the input image to obtain a shallow feature extraction map;

[0060] Step 2-2: Construct a feature fusion residual group module for a deep separable convolutional neural network model to extract deep features from the extracted shallow feature maps, thus obtaining deep feature maps;

[0061] like Figure 2 As shown, the feature fusion residual group module for constructing a deep separable convolutional neural network model performs deep feature extraction on the extracted shallow feature maps to obtain deep feature maps, including the following steps:

[0062] Step 2-2-1: Construct the deep separable convolutional channel attention residual module of the deep separable convolutional neural network. Input the shallow feature map and use a single deep convolution with a kernel size of 96×7×7, a stride of 1, and padding of 3 to obtain the spatial feature relationship of the image through a large receptive field.

[0063] Step 2-2-2: Construct the channel attention module of a deep separable convolutional neural network. Perform average pooling on the spatial dimension, learn channel attention through two fully connected layers, and use Sigmoid for layer normalization to obtain the channel attention feature vector. Multiply the channel attention feature vector by the features after deep group convolution to obtain the weighted deep convolutional features. Calculate the mean and variance of these features, and then normalize this layer's features to reduce training time and accelerate network training. The above can be expressed by the following formula:

[0064]

[0065] A1=[A 1,1 A 1,2 ,...,A 1,n ]

[0066] B1 = CA(F0)

[0067] C1 = A1·B1

[0068]

[0069]

[0070]

[0071] Where A is the feature vector after depthwise convolution, n represents the nth neuron, CA(·) is the channel attention module function, B is the feature vector of channel attention, C is the depthwise convolution feature after channel attention weighting, and γ and β are the scaling and translation parameter vectors with the same dimension as C.

[0072] The channel attention-weighted feature map is subjected to a 384×1×1 point convolution to increase its dimensionality. The GELU activation function is used to introduce randomness into the σ activation function, making the training process more robust. Then, a 96×1×1 point convolution is used to reduce its dimensionality, improving the channel interaction capability of the depthwise convolution. The input feature map and the residual extracted feature map are then added together via short connections to obtain the output feature map.

[0073] The above can be expressed by the following formula:

[0074]

[0075]

[0076] E1 = GELU(D1)

[0077]

[0078] Where D and F are point convolution operations, and E is the GELU activation function.

[0079] like Figure 3 As shown, step 2-2-3: Construct a feature fusion module for a deep separable convolutional neural network. The feature fusion module consists of six deep separable convolutional channel attention residual modules connected in series. The feature maps output by the six deep separable convolutional channel attention residual modules are fused along the channel dimension. Then, a 96×3×3 convolution kernel is used for feature recombination. The long connections of the input feature map and the residual extracted feature map are added together to form the feature fusion module. The purpose of constructing the feature fusion module is to utilize the complementarity between features, fuse the advantages between features, eliminate redundant information caused by the correlation between different features, and improve the performance of the model.

[0080] like Figure 4 As shown, step 2-2-4: Construct a depthwise separable convolutional channel attention fusion residual group module. This module consists of four feature fusion modules connected in series. The input feature map and the residual extracted feature map are added together via long connections to form the depthwise separable convolutional channel attention fusion residual group module. The purpose of constructing this module is to use residual groups as the basic modules for deeper networks. Information can flow between each residual group, obtaining more useful information and focusing on richer features. Through residual learning, the model can achieve better performance.

[0081] Steps 2-3: Construct an image reconstruction module for a deep separable convolutional neural network model, upsample the deep feature maps, and then generate super-resolution images using a 3×3×3 standard set.

[0082] Step 3: Train the deep separable convolutional neural network model, calculate the loss between the super-resolution image and the input high-resolution image, and optimize the deep separable convolutional neural network model;

[0083] A training set was constructed using real images from underground mines, with images numbered 1-810 selected, and images numbered 1-800 used as the training set. The high-resolution images (HR) in the dataset were downsampled by a scale factor of 2x, 3x, and 4x using bicubic interpolation to obtain the corresponding low-resolution images (LR). The reconstructed super-resolution images were then compared with the input high-resolution images to calculate the loss.

[0084] The Adam optimizer was used with a learning rate of 0.0002 and a batch size of 8. The learning rate was halved after every 100 training epochs, for a total of 500 training epochs.

[0085] The reconstructed super-resolution image and the input high-resolution image are compared to calculate the loss, and the network parameters are updated using the backpropagation algorithm. The learning ability of the model is improved by minimizing the loss function. The specific formula for the average loss function L1 is as follows:

[0086]

[0087] Where L1 is the average loss, x HR For the high-resolution original image, y SR To reconstruct images for super-resolution.

[0088] Step 4: Test the depthwise separable convolutional neural network model on the reconstructed super-resolution image; based on the performance of the trained super-resolution reconstruction network model on the validation set, select the network model parameters with the best validation set evaluation metrics. Test on the test set to evaluate the existing network model.

[0089] Step 5: Preprocess the test images and input them into the deep separable convolutional neural network model for verification; preprocess the test images, set the patch size, crop the high- and low-resolution image pairs in the training set at random distances starting from the image center position, and save the training set as .npy format as input to the deep separable convolutional neural network model.

[0090] Step 6: Validate the depthwise separable convolutional neural network model, evaluate the depthwise separable convolutional neural network model, and assess the quality standards of the reconstructed super-resolution image.

[0091] A validation set was constructed, with images 801-810 serving as the validation set. Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were used to evaluate the quality of super-resolution image reconstruction on the Y channel of the transformed YCbCr space.

[0092]

[0093] Higher PSNR and SSIM scores indicate that the reconstructed image is closer to the original image.

[0094] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for image super-resolution reconstruction for wearable helmets, characterized in that: The method includes the following steps: Step 1: Construct a degradation model to obtain high- and low-resolution image pairs; Step 2: Construct a deep separable convolutional neural network model. The deep separable convolutional neural network generates super-resolution images, including the following steps: Step 2-1: Constructing a shallow feature extraction module for a deep separable convolutional neural network model, performing initial feature extraction on the input image to obtain a shallow feature extraction map; Step 2-2: Construct a feature fusion residual group module for a deep separable convolutional neural network model to extract deep features from the extracted shallow feature maps, obtaining deep feature maps. This includes the following steps: Step 2-2-1: Construct the deep separable convolutional channel attention residual module of the deep separable convolutional neural network. Input the shallow feature map and use a single deep convolution with a kernel size of 96×7×7, a stride of 1, and padding of 3 to obtain the spatial feature relationship of the image through a large receptive field. Step 2-2-2: Construct the channel attention module of the deep separable convolutional neural network, perform average pooling on the spatial dimension, learn the channel attention through two fully connected layers, and use Sigmoid for layer normalization to obtain the channel attention feature vector. Multiply the channel attention feature vector with the feature vector after depth group convolution to obtain the weighted depth convolution feature, and then use layer normalization again. The feature map with channel attention weighting is subjected to a point convolution with a kernel of 384×1×1 to increase its dimensionality. After applying the GELU activation function, a point convolution with a kernel of 96×1×1 is applied to reduce its dimensionality, thereby improving the channel interaction capability of the depth convolution. The input feature map and the residual extracted feature map are then added together via short connections to obtain the output feature map. Step 2-2-3: Construct a feature fusion module for a deep separable convolutional neural network. The feature fusion module consists of six deep separable convolutional channel attention residual modules connected in series. The feature maps output by the six deep separable convolutional channel attention residual modules are fused along the channel dimension. Then, a 96×3×3 convolution kernel is used for feature recombination. The long connections of the input feature map and the residual extracted feature map are added together to form the feature fusion module. The feature fusion module utilizes the complementarity between features, fuses the advantages between features, and eliminates redundant information caused by the correlation between different features. Step 2-2-4: Construct a depthwise separable convolutional channel attention fusion residual group module. This module consists of four feature fusion modules connected in series. The input feature map and the residual extracted feature map are added together via long connections to form the depthwise separable convolutional channel attention fusion residual group module. This module utilizes residual groups as the foundation for deeper networks. Information can flow within each residual group, obtaining more useful information and focusing on richer features. Through residual learning, the model achieves better performance. Steps 2-3: Construct an image reconstruction module for a deep separable convolutional neural network model, upsample the deep feature maps, and then generate a super-resolution image through a standard 3×3×3 convolution; Step 3: Train the deep separable convolutional neural network model, calculate the loss between the super-resolution image and the input high-resolution image, and optimize the deep separable convolutional neural network model; Step 4: Test the depthwise separable convolutional neural network model on the reconstructed super-resolution image; Step 5: Preprocess the image and input it into a depthwise separable convolutional neural network model for verification; Step 6: Validate the depthwise separable convolutional neural network model, evaluate the depthwise separable convolutional neural network model, and assess the quality standards of the reconstructed super-resolution image.

2. The image super-resolution reconstruction method for wearable helmets according to claim 1, characterized in that: The construction of the degradation model includes the following steps: Step 1-1: Construct a dataset consisting of real images from underground mines; Steps 1-2: Convert the real image from the mine to the HSV color space, and randomly reduce the values ​​of the saturation (S) and brightness (V) channels to simulate a low-light image; Steps 1-3: Degradation operation is performed on the low-light images in the dataset. The high-resolution images in the dataset are downsampled by a scale factor of 2x, 3x, and 4x using the bicubic linear interpolation method to obtain the corresponding low-resolution images. The degradation operation is represented by the following formula: ； Where x represents the degraded image, y represents the original image, D represents the degradation function, s is the reduction factor, and r represents the scaling factor.

3. The image super-resolution reconstruction method for wearable helmets according to claim 1, characterized in that: In step 3, the deep separable convolutional neural network model is trained by constructing a training set, which uses real images from underground mines. The high-resolution images HR in the dataset are downsampled by a scale factor of 2x, 3x, and 4x using bicubic linear interpolation to obtain the corresponding low-resolution images LR. The reconstructed super-resolution images are then compared with the input high-resolution images to calculate the loss.

4. The image super-resolution reconstruction method for wearable helmets according to claim 1, characterized in that: In step 4, the depth-separable convolutional neural network model is tested by constructing a test set, which includes low-resolution images LR with scale factors of x2, x3, and x4 using bicubic linear interpolation downsampling, and corresponding high-resolution images HR.

5. The image super-resolution reconstruction method for wearable helmets according to claim 1, characterized in that: In step 5, the images after testing are preprocessed, the patch size is set, and the high- and low-resolution image pairs in the training set are cropped at random distances starting from the image center position. The training set is then saved in .npy format as input to the depthwise separable convolutional neural network model.

6. The image super-resolution reconstruction method for wearable helmets according to claim 1, characterized in that: In step 6, the depthwise separable convolutional neural network model is verified by constructing a verification set. Criteria for evaluating the quality of super-resolution image reconstruction in the Y channel transformed to YCbCr space using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity-SSIM: ； Higher PSNR and SSIM scores indicate that the reconstructed image is closer to the original image.

7. The image super-resolution reconstruction method for wearable helmets according to claim 3, characterized in that: The reconstructed super-resolution image and the input high-resolution image are then compared to perform loss calculations. The loss calculation uses the following formula: ； Where L1 is the average loss, For high-resolution original images, To reconstruct images for super-resolution.

Images