Mine driving working face low-quality image super-resolution reconstruction method and system

Abstract

The invention discloses a mine driving face low-quality image super-resolution reconstruction method and system, and the method comprises the steps: adding an MAB and an EFN module in an FMB of a MetaFormer style structure, fusing an MHFE module and an MLSKA module in the MAB, carrying out the up and down sampling of output features through employing the MHFE module, so as to simulate a multi-scale high-pass filter, carrying out the learning of the non-local structure information of the output features in S1 through employing the MLSKA module, and carrying out the reconstruction of a low-quality image of a mine driving face. And long-distance dependence is captured. The system comprises a shallow feature extraction module, a deep feature extraction module and a reconstruction module. According to the method, calculation can be simplified, efficient combination of global information and local information can be realized, and meanwhile, the recovery capability of high-frequency details is improved.

Description

Technical Field

[0001] The present invention relates to a method and system for super-resolution reconstruction of low-quality images in a mine tunneling working face, belonging to the technical field of image super-resolution reconstruction. Background Art

[0002] In the complex environment of a mine, especially on a coal mine tunneling working face, the acquisition and processing of images face many problems such as complex lighting conditions, uneven light intensity, image blurring caused by a large amount of dust generated during tunneling operations, image noise caused by electronic device interference, and limited computing power of hardware devices. In mine tunneling operations, the acquired images may be blurred and severely interfered by noise due to factors such as low illuminance, strong dynamic changes, dust, and smoke, making it difficult to provide clear detailed information. This poses a great challenge to the reliability and accuracy of mine automation monitoring systems, personnel positioning, equipment fault detection, and environmental risk warning systems. Therefore, studying the super-resolution reconstruction technology of low-quality images is of great significance and necessity for improving the operation efficiency and safety of mine tunneling working faces.

[0003] Image super-resolution reconstruction is a technology aimed at reconstructing a low-resolution degraded image into a high-resolution clear image. Currently, the main methods for image super-resolution reconstruction are interpolation-based methods, reconstruction-based methods, and learning-based methods. Although significant progress has been made in super-resolution models based on deep learning, they still face challenges such as high computational complexity, limited deployment, difficulty in balancing global and local information, and difficulty in recovering high-frequency information in practical applications. Summary of the Invention

[0004] The object of the present invention is to provide a method and system for super-resolution reconstruction of low-quality images in a mine tunneling working face, which can simplify the calculation, efficiently combine global and local information, and improve the ability to recover high-frequency details.

[0005] To achieve the above object, the present invention provides a method for super-resolution reconstruction of low-quality images in a mine tunneling working face, including the following steps:

[0006] S1. After the input low-resolution image is first subjected to shallow feature extraction and feature space transformation, deep feature extraction is then performed. The deep feature extraction module is composed of stacked BRB modules, and the BRB module is composed of stacked FMB modules.

[0007] S2. Add a multi-scale attention module MAB and an efficient feed-forward network EFN module to the feature fusion module FMB of the MetaFormer style structure; after processing the features in the FMB, send them to the MAB for learning the global structure and local details, perform layer normalization on the shallow features again after residual mapping, and then send them to the EFN module for optimizing the non-linear expression ability. After residual mapping, output the features.

[0008] S3. In the MAB, fuse the multi-scale high-frequency gain MHFE module and the multi-scale large kernel separable attention MLSKA module, and perform channel fusion on the features processed and output by the MHFE module and the MLSKA module; the EFN module refines the important features.

[0009] S4. Use the MHFE module to perform upsampling and downsampling on the output features in S1 to simulate a multi-scale high-pass filter, and further enhance the different-scale high-frequency information extracted.

[0010] S5. Use the MLSKA module to learn the non-local structure information of the output features in S1 and capture long-range dependencies.

[0011] S6. The reconstruction module receives the features input by the deep feature extraction module and converts them into the final reconstructed image for output.

[0012] Furthermore, the low-resolution image input in S1 first passes through the shallow feature extraction module and then through the deep feature extraction module. The feature extraction process is described as follows:

[0013] S1.1. Extract shallow features from the input low-resolution image:

[0014] F s = Conv 3×3 (I LR );

[0015] Among them, Conv 3×3 represents a standard 3×3 convolution operation, I LR is the input low-resolution image, and F s represents the extracted shallow feature map.

[0016] S1.2. Extract deep features from the features output by the shallow feature extraction module:

[0017] The deep feature extraction module adopts a cascaded modular design, and the output of each module is passed to the next module for step-by-step optimization. The deep feature extraction process is described as follows:

[0018] F i+1 = BRB i+1(F i ), where \(i\in[1,N]\);

[0019] Among them, \(F\) i+1 is the output of the \((i + 1)\)-th basic residual block (BRB), and \(N\) is the number of deep modules. By fusing the output features of all BRB modules, the final deep feature representation is obtained:

[0020]

[0021] Among them, \(F\) k is the output feature of the \(k\)-th BRB module, Conc represents feature concatenation, represents element-wise addition.

[0022] Each BRB module further extracts the correlation between features through the combination of residual connection and convolution operation. The formula is described as follows:

[0023]

[0024] Among them, \(F\) i is the output of the \(i\)-th basic residual block (BRB) as the input of the \((i + 1)\)-th BRB, and \(F\) i+1 is the output of the \((i + 1)\)-th FMB, and \(\varphi\) is the GELU activation function.

[0025] Furthermore, in the above \(S2\), the MAB module and the EFN module are embedded in the MetaFormer style structure to extract deep features, and the extraction process is shown as follows:

[0026]

[0027] Among them, \(x\) is the input feature of the FMB, \(x\) 1 is a temporary variable, and \(x\) 2 is the output variable of the FMB; LN is the LayerNormer layer normalization module, and MAB and EFN represent the MAB module and the EFN module respectively.

[0028] Furthermore, the formula for channel fusion of the features processed and output by the MHFE module and the MLSKA module in the above \(S3\) is:

[0029] Y, Z = Split(Conv 1×1 (X));

[0030] Among them, Y and Z are the sub-features obtained by channel splitting the feature X input to the MAB module;

[0031] S = MHFE(Y); V = MLSKA(Z);

[0032] Among them, MHFE and MLSKA represent the multi-scale high-frequency gain module and the multi-scale large kernel separable attention module respectively;

[0033]

[0034] Among them, M is the output of the MAB module;

[0035] The refined description of the important features by the EFN module is:

[0036] E 1 ,E 2 = Split(φ(Conv 1×1 (E)));

[0037] E' = Conv 1×1 (Conc(Conv 3×3 (E 1 ),E 2 ));

[0038] Among them, E 1 ,E 2 are the sub-features after channel splitting of E respectively, and E' is the output structure of the EFN module.

[0039] Furthermore, in S4, the MHFE module uses upsampling and downsampling by 2 times, 4 times, and 8 times respectively to simulate a high-end filter to extract multi-scale high-frequency detail information, and then uses a 3×3 convolutional kernel to further extract features and gain high-frequency details of this high-frequency information. The process is as follows:

[0040] S4.1: Use a 1×1 point convolution Conv 1×1 to reduce the dimension of the feature map and output Y 0 :

[0041] Y 0 = φ(Conv 1×1 (Y)); where Y is the input feature of the MHFE module;

[0042] Y 1 = U 8 (D 8 (Y 0 ));

[0043] Y 2 = U 4 (D 4 (Y 0 ));

[0044] Y 3 = U 2 (D 2 (Y 0 ));

[0045] Among them, D i represents i-fold downsampling, and U i represents i-fold upsampling;

[0046] While performing upsampling and downsampling on the input features, two 1×1 pointwise convolutions Conv 1×1 are used to learn the feature in the channel dimension:

[0047] Y 4 = Conv 1×1 (Conv 1×1 (Y 0 ));

[0048] S4.2. Subtract the extracted low-frequency information Y i from Y 4 to extract the high-frequency information Y', i realize the elimination of redundant features, and then use Conv 3×3 to perform gain on high-frequency features and realize differential learning of features:

[0049] Y' i = φ(Conv 3×3 (Y i - Y 4 ), i ∈ {1, 2, 3};

[0050]

[0051]

[0052] Among them, σ is the Sigmoid function, is element-wise multiplication, and S is the output of the MHFE module.

[0053] Furthermore, the specific process of S5 is as follows:

[0054] S5.1. Divide the feature map Z of the input MLSKA into three sub-feature maps Z 1 , Z 2 , Z 3 , and apply multi-scale large kernel separable convolution to each sub-feature map:

[0055] Z 1 , Z 2 , Z 3 = Split(Conv 1×1 (Z));

[0056] Among them, Split represents channel splitting;

[0057] LSKA i,j,1 (F) = Conv1×1 (DWDConv j×1 (DWDConv 1×j (DWConv i×1 (DWConv 1×i (F)))));

[0058] Among them, LSKA i,j,1 (F) is a large kernel separable convolution function, which is composed of the following: depthwise convolution DWConv of size 1×i 1×i , depthwise convolution DWConv of size i×1 i×1 , dilated depthwise convolution DWConv of size 1×j 1×j , dilated depthwise convolution DWConv with dilation rate j of size j×1 j×1 , pointwise convolution Conv of size 1×1 1×1 , and F is an intermediate feature variable;

[0059] S5.2. Use large kernel separable attention for self-adjustment of features:

[0060]

[0061] Its large kernel separable convolutions are respectively used to simulate large kernel convolutions with receptive field sizes of 7, 23, and 35, and the dilation rates of the corresponding dilated convolutions are 2, 3, and 4 respectively;

[0062] S5.3. Fuse all sub-feature maps:

[0063] Z' = Conc(Z' 1 , Z' 2 , Z' 3 );

[0064]

[0065] Among them, Conc is channel concatenation, and V is the output of MLSKA.

[0066] Furthermore, the reconstruction module in S6 is responsible for converting the deep feature F d into the final super-resolution image I SR , and the specific process is as follows:

[0067] I SR = Conv 3×3 (Shuffle(φ(Conv 3×3 (F d ))))

[0068] Among them, Shuffle represents the pixel rearrangement operation, F d represents the output of the deep feature extraction layer, and I SRRepresents the result of image super-resolution reconstruction.

[0069] The present invention also provides a low-quality image super-resolution reconstruction system for a mine tunneling face, which is characterized by comprising a shallow feature extraction module, a deep feature extraction module, and a reconstruction module;

[0070] The described shallow feature extraction module extracts the basic texture information of the input low-resolution image through single-layer convolution operation, providing an initial feature representation for subsequent deep feature extraction;

[0071] The deep feature extraction module includes a multi-scale attention module, an efficient feed-forward network module, a basic residual block, a multi-scale large kernel separable attention module, and a multi-scale high-frequency gain module; the deep feature extraction module adopts a cascaded modular design, and the output of each module is passed to the next module for gradual optimization;

[0072] The reconstruction module is used to convert the deep features into the final super-resolution image.

[0073] In the present invention, by adding a multi-scale attention module MAB and an efficient feed-forward network EFN module to the feature fusion module FMB, and fusing a multi-scale high-frequency gain MHFE module and a multi-scale large kernel separable attention MLSKA module in the MAB; the MHFE module focuses on the mining and gain of multi-scale high-frequency information. By introducing a differential feature learning mechanism, it fully models different-scale features and suppresses redundancy, making up for the deficiency of the long-distance dependence modeling in local detail learning ability. Through efficient high-frequency information extraction operations, the MHFE module significantly enhances the recovery ability of key details such as edges and textures, providing finer feature support for super-resolution reconstruction; the MLSKA module combines the large kernel attention mechanism with the multi-scale mechanism, taking advantage of the large receptive field to capture global structure information at different scales and reducing the computational complexity of self-attention in the Transformer model; using a lightweight EFN module to replace the traditional multi-layer perceptron MLP reduces the number of model parameters and computational complexity. The present invention realizes the efficient modeling and fusion of global and local features, and at the same time greatly improves the extraction and recovery ability of high-frequency information, achieving high-performance super-resolution reconstruction. Brief Description of the Drawings

[0074] Figure 1 is a schematic diagram of the working process of the method of the present invention;

[0075] Figure 2 is a schematic diagram of the processes of shallow feature extraction, deep feature extraction, and reconstruction of the present invention;

[0076] Figure 3 is a comparison diagram of image processing in an embodiment of the present invention. Detailed Embodiments

[0077] The present invention will be further described below with reference to the accompanying drawings.

[0078] As Figure 1 shown, a method for super-resolution reconstruction of low-quality images in a mine tunneling working face includes the following steps:

[0079] S1. After the input low-resolution image is first subjected to shallow feature extraction and feature space transformation, deep feature extraction is then performed. The deep feature extraction module is mainly composed of stacked BRB modules, and the BRB module is composed of stacked FMB modules.

[0080] S2. Add a multi-scale attention module MAB and an efficient feed-forward network EFN module to the feature fusion module FMB of the MetaFormer style structure; after processing the features in the FMB and sending them to the MAB for learning of global structure and local details, after residual mapping, the shallow features are subjected to another layer normalization process and then sent to the EFN module for optimization of non-linear expression ability, and then the features are output after residual mapping;

[0081] S3. In the MAB, fuse the multi-scale high-frequency gain MHFE module and the multi-scale large kernel separable attention MLSKA module, and perform channel fusion on the features processed and output by the MHFE module and the MLSKA module; the EFN module refines important features;

[0082] S4. Use the MHFE module to perform upsampling and downsampling on the output features in S1 to simulate a multi-scale high-pass filter, and further gain the extracted high-frequency information at different scales;

[0083] S5. Use the MLSKA module to learn the non-local structure information of the output features in S1 and capture long-range dependencies.

[0084] S6. The reconstruction module receives the features input by the deep feature extraction module and converts them into the final reconstructed image for output.

[0085] Embodiment: The network structure of the present invention includes a shallow feature extraction module, a deep feature extraction module, and a reconstruction module, as Figure 2 shown. Input I LR low-resolution image, use a 3×3 convolution as the shallow feature extraction layer. The extracted shallow features will first be input into the deep feature extraction module to further extract deep features, and secondly, they will also be mapped as residual features to the end of the deep feature extraction module to retain low-level low-frequency information;

[0086] In the depth feature extraction layer, several basic residual block (BRB) modules are stacked, and the outputs of all BRB modules are concatenated in the channel dimension and then fused and learned in the channel dimension. Finally, the global residual mapping of the element-added shallow features is performed, which effectively avoids the training problem of gradient disappearance;

[0087] In the BRB module, several feature multi-branch (FMB) modules are stacked for deep feature extraction, and the output results of all FMB modules are concatenated and fused in the channel dimension, and finally the original input features are received as the local residual mapping;

[0088] In the FMB module, the LayerNorm layer is used for layer normalization, and then it is input into the multi-head attention block (MAB) module for global and local information extraction. Then the LayerNorm layer is used for layer normalization again, and then it is input into the efficient feature network (EFN) feed-forward network module for channel dimension interaction;

[0089] In the MAB module, there are two parallel branches. One branch uses the multi-head feature extraction (MHFE) module for high-frequency feature extraction, and one parallel branch uses the multi-level spatial kernel attention (MLSKA) module for non-local structure information modeling. Finally, the feature fusion is simply performed through channel dimension concatenation and point convolution, so that the module takes into account the advantages of large receptive field and local detail learning. In the MLSKA module, first, point convolution is used to expand the feature in the channel dimension, and then it is divided into three sub-feature maps by channel splitting. Different-scale large kernel separable attention processing is performed on each sub-feature, and the gated aggregation mechanism is used to suppress the block effect. Finally, the three processed features are fused to output a multi-scale large kernel separable attention map, and the input feature is self-modulated and then output. In the MHFE module, first, point convolution is used to compress the feature in the channel dimension, and then upsampling by ×2, ×4, and ×8 times is used to extract multi-scale high-frequency features. The extracted high-frequency features are separately gain-adjusted, and then the features of the three branches are fused and output through the sigmoid function to obtain the high-frequency feature gain weight to realize the high-frequency information self-modulation function of the input feature. Finally, a 3×3 convolution is used to restore the dimension and output the result; In the image reconstruction layer, two 3×3 convolutions and pixel rearrangement operations are included to realize the upsampling of the image.

[0090] The proposed MLSKA and MHFE modules in the present invention achieve efficient modeling and fusion of global and local features, and at the same time greatly improve the extraction and recovery ability of high-frequency information. As Figure 3 (a) shows, the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) of the low-resolution image of the tunneling working face are 35.91 and 0.9810 respectively, Figure 3 (b) shows, the PSNR and SSIM of the reconstructed image after super-resolution reconstruction using the present invention are 36.63 and 0.9842 respectively, and it can be clearly seen that the indexes are significantly improved and the image details are clearer.

Claims

1. A method for super-resolution reconstruction of low-quality images of a mine excavation working face, characterized in that: The steps include: S1. The input low-resolution image is first subjected to shallow feature extraction and feature space transformation, and then deep feature extraction is performed; wherein the deep feature extraction module is composed of a stack of BRB modules, and the BRB module is composed of a stack of FMB modules; S2. Add a multi-scale attention module MAB and an efficient feedforward network EFN module to the feature fusion module FMB of the MetaFormer style structure; after processing the features in FMB, send them to MAB for learning the global structure and local details, and then perform a layer-by-layer normalization on the shallow features after residual mapping and send them to the EFN module for optimizing the nonlinear expression ability, and then output the features after residual mapping; S3. In MAB, the multi-scale high-frequency gain MHFE module and the multi-scale large-core separable attention MLSKA module are integrated to perform channel fusion on the features processed and output by the MHFE module and the MLSKA module; the EFN module refines the important features; S4, using the MHFE module to up- and down-sample the output features in S1 to simulate a multi-scale high-pass filter, and further gain the extracted high-frequency information of different scales; S5, use the MLSKA module to learn the non-local structural information of the output features in S1 and capture long-range dependencies; S6. The reconstruction module receives the features input by the deep feature extraction module and converts them into the final reconstructed image for output.

2. The method for super-resolution reconstruction of low-quality images of a mine excavation working face according to claim 1, characterized in that: The low-resolution image input in S1 first passes through the shallow feature extraction module and then passes through the deep feature extraction module. The feature extraction process is described as follows: S1.

1. Extract shallow features from the input low-resolution image: F s =Conv 3×3 (I LR ); Among them, Conv 3×3 represents a 3×3 standard convolution operation, I LR is the input low-resolution image, F s Represents the extracted shallow feature map; S1.

2. Perform deep feature extraction on the features output by the shallow feature extraction module: The deep feature extraction module adopts a cascaded modular design. The output of each module is passed to the next module for step-by-step optimization. The process of deep feature extraction is described as follows: F i+1 =BRB i+1 (F i ),i∈[1,N]; Among them, F i+1 is the output of the i+1th basic residual block (BRB), N is the number of deep modules, and the final deep feature representation is obtained by fusing the output features of all BRB modules: F d =Conv 3×3 (F s ⊕Conv 3×3 (φ(Conv 1×1 (Conc(F k )))))),k∈[1,N]; Among them, F k is the output feature of the kth BRB module, Conc represents feature concatenation, and ⊕ represents element-by-element addition; Each BRB module further extracts the correlation between features through a combination of residual connection and convolution operation. The formula is described as follows: F i+1 =F i ⊕Conv 3×3 (φ(Conv 1×1 (Conc(F i )))),i∈[1,N] Among them, F i The output of the i-th basic residual block (BRB) is used as the input of the i+1-th BRB, F i+1 is the output of the i+1th FMB, and φ is the GELU activation function.

3. The method for super-resolution reconstruction of low-quality images of a mine excavation working face according to claim 1, characterized in that: In S2, the MAB module and the EFN module are embedded in the MetaFormer style structure to extract deep features. The extraction process is shown as follows: x1=x⊕MAB(LN(x)); x2=x1⊕EFN(LN(x1)); Among them, x is the input feature of FMB, x1 is a temporary variable, and x2 is the output variable of FMB; LN is the LayerNormer layer normalization module, MAB and EFN represent the MAB module and EFN module respectively.

4. The method for super-resolution reconstruction of low-quality images of a mine excavation working face according to claim 1, characterized in that: The formula for channel fusion of the features processed and output by the MHFE module and the MLSKA module in S3 is: Y,Z=Split(Conv 1×1 (X)); Among them, Y and Z are the sub-features obtained by channel segmentation of the feature X input to the MAB module; S = MHFE (Y); V = MLSKA (Z); Among them, MHFE and MLSKA represent the multi-scale high-frequency gain module and the multi-scale large-kernel separable attention module respectively; M=Conv 1×1 (Conc(S,V))⊕X; Where M is the output of the MAB module; The EFN module describes the important features in detail as follows: E1,E2=Split(φ(Conv 1×1 (AND))); E'=Conv 1×1 (Conc(Conv 3×3 (E1),E2)); Among them, E1 and E2 are the sub-features after channel segmentation of E, and E' is the output structure of the EFN module.

5. The method for super-resolution reconstruction of low-quality images of a mine excavation working face according to claim 1, characterized in that: In the S4, the MHFE module uses 2x, 4x, and 8x up and down sampling to simulate high-matching filters to extract multi-scale high-frequency detail information, and then uses a 3×3 convolution kernel to further extract features of the high-frequency information and gain high-frequency details. The process is as follows: S4.1, using 1×1 point convolution Conv 1×1 Reduce the dimension of the feature map and output Y0: Y0=φ(Conv 1×1 (Y)); where Y is the input feature of the MHFE module; Y1 = U8 (D8 (Y0)); Y2=U4(D4(Y0)); Y3=U2(D2(Y0)); Among them, D i represents i-fold downsampling, U i Indicates i-fold upsampling; While upsampling and downsampling the input features, two 1×1 point convolutions Conv 1×1 Learning the channel dimension of features: Y4=Conv 1×1 (Conv 1×1 (Y0)); S4.

2. Extracting low-frequency information Y i Subtract from Y4 to extract high-frequency information Y' i , to eliminate redundant features, and then use Conv 3×3 Gain high-frequency features and achieve differentiated learning of features: AND' i =φ(Conv 3×3 (AND i -Y4)),i∈{1,2,3}; Y'=σ(Y'1⊕Y'2⊕Y'3⊕Y4); Among them, σ is the Sigmoid function, is the element-wise multiplication and S is the output of the MHFE module.

6. The method for super-resolution reconstruction of low-quality images of a mine excavation working face according to claim 1, characterized in that: The specific process of S5 is as follows: S5.

1. Divide the input MLSKA feature map Z into three sub-feature maps Z1, Z2, and Z3, and apply multi-scale large-kernel separable convolution to each sub-feature map: Z1,Z2,Z3=Split(Conv 1×1 (Z)); Among them, Split means channel splitting; LSKA i,j,1 (F)=Conv 1×1 (DWDConv j×1 (DWDConv 1×j (DWConv i×1 (DWConv 1×i (F))))); Among them, LSKA i,j,1 (F) is a large kernel separable convolution function, which is composed of the following: 1×i size deep convolution DWConv 1×i , i×1 size depth convolution DWConv i×1 , 1×j size deep dilated convolution DWConv 1×j , j×1 size dilation rate j deep dilated convolution DWConv j×1 , 1×1 point convolution Conv 1×1 , F is the intermediate characteristic variable; S5.

2. Self-adjustment of features using large kernel separable attention: Its large-kernel separable convolution is used to simulate large-kernel convolutions with receptive field sizes of 7, 23, and 35, respectively, and the corresponding dilation rates of the dilated convolution are 2, 3, and 4, respectively; S5.

3. Fusion of all sub-feature maps: Z' = Conc(Z'1, Z'2, Z'3); Among them, Conc is the channel concatenation and V is the output of MLSKA.

7. The method for super-resolution reconstruction of low-quality images of a mine excavation working face according to claim 1, characterized in that: The reconstruction module of S6 is responsible for transforming the deep features F d Converted to the final super-resolution image I SR , the specific process is: I SR =Conv 3×3 (Shuffle(φ(Conv 3×3 (F d )))); Among them, Shuffle represents the pixel rearrangement operation, F d represents the output of the deep feature extraction layer, I SR Represents the image super-resolution reconstruction result.

8. A super-resolution reconstruction system for low-quality images of a mine excavation working face, characterized in that: It includes a shallow feature extraction module, a deep feature extraction module and a reconstruction module; The shallow feature extraction module extracts basic texture information of the input low-resolution image through a single-layer convolution operation, providing initial feature representation for subsequent deep feature extraction; The deep feature extraction module includes a multi-scale attention module, an efficient feedforward network module, a basic residual block, a multi-scale large core separable attention module and a multi-scale high-frequency gain module; the deep feature extraction module adopts a cascaded modular design, and the output of each module is passed to the next module for step-by-step optimization; The reconstruction module is used to convert the deep features into the final super-resolution image.

Images