Deep learning-based mine image super-resolution reconstruction method, medium and equipment

Abstract

The invention relates to a mine image super-resolution reconstruction method based on deep learning, a medium and equipment. The method comprises the following steps: starting a mine image super-resolution reconstruction task; acquiring image data in a mine environment; the method comprises the following steps: preprocessing images, denoising the images and enhancing image data by using an image denoising technology so as to increase the number of the images by multiple times; constructing a super-resolution deep learning model; training a super-resolution deep learning model; and inputting a to-be-reconstructed mine image, and performing super-resolution reconstruction on the to-be-reconstructed mine image by using the super-resolution deep learning model. According to the method, the super-resolution deep learning model is constructed, so that the edge details of the mine image are effectively improved, the fusion degree of the feature information is enhanced, and the reconstructed mine image is clearer and rich in details.

Description

Technical Field

[0001] The present invention relates to the field of coal mine intelligence, and in particular to a mine image super-resolution reconstruction method based on deep learning. Background Art

[0002] Intelligent coal mines are the core technical support for the coal industry to achieve high-quality development. High-resolution underground images can provide key data support for intelligent analysis in the coal mine field, and their application scope includes video surveillance, personnel behavior recognition, and coal gangue transportation. However, due to the limitations of the imaging system, the harsh underground environment, and imaging technology, underground images of coal mines often show degradation phenomena such as gray blur and loss of edge information, and the spatial resolution is low, which cannot meet the needs of intelligent applications for high-quality images. Therefore, it is particularly necessary to study super-resolution reconstruction technology for mine images in order to improve the clarity and detail retention of mine images and further promote the rapid development of intelligent coal mines. Summary of the invention

[0003] In view of this, the present invention provides a mine image super-resolution reconstruction method based on deep learning, thereby solving or at least alleviating one or more of the above-mentioned problems and other problems existing in the prior art.

[0004] In order to achieve the aforementioned objective, a first aspect of the present invention provides a mine image super-resolution reconstruction method based on deep learning, wherein the method comprises the following steps:

[0005] S1: Start the mine image super-resolution reconstruction task;

[0006] S2: Acquire image data in a mine environment;

[0007] S3: preprocessing the image, using image denoising technology to denoise the image and enhance the image data so as to increase the number of the image by multiple times;

[0008] S4: Build a super-resolution deep learning model;

[0009] S5: Using the enhanced image data to train a super-resolution deep learning model;

[0010] S6: Input a mine image to be reconstructed, and use the super-resolution deep learning model to perform super-resolution reconstruction on the mine image.

[0011] In the method as described above, optionally, in step S2, the image is acquired from a mine dataset.

[0012] In the method described above, optionally, in step S3, the image denoising technique is Gaussian filtering, and the method of enhancing the image data includes transforming, cropping, randomly rotating 90°, 180° and 270°, and horizontally flipping the image.

[0013] In the method as described above, optionally, in step S4, the super-resolution deep learning model includes

[0014] A shallow feature extraction module, wherein the shallow feature extraction module uses a convolution layer to extract shallow features of the image, wherein the shallow features include contours and edges, colors, textures, and shapes of the image;

[0015] A high-level feature extraction module, the high-level feature extraction module includes dual-space feature extraction and dual-channel feature extraction, the high-level feature extraction module is used to extract high-level features of the image data, the high-level features include high-level semantic information of the image, contextual relationship in the image, spatial layout, event detection, attribute recognition;

[0016] A reconstruction module, wherein the reconstruction module uses an upsampling method to upsample the deep features finally extracted by the shallow feature extraction module and the high-level feature extraction module, and uses a convolutional layer to aggregate features before and after the upsampling operation. The reconstruction module converts the low-resolution image data into the high-resolution image data.

[0017] In the method described above, optionally, the super-resolution deep learning model includes 1 shallow feature extraction module, 4 high-level feature extraction modules and 1 reconstruction module, and each of the high-level feature extraction modules includes 3 dual-space feature extractions and 3 dual-channel feature extractions.

[0018] In the method as described above, optionally, the dual-space feature extraction includes a spatial window attention, a local convolution module, an adaptive interaction module and a dual-group feedforward network, the dual-space feature extraction normalizes the image data, and then after the spatial window attention and the local convolution module process the image data in parallel, the image data is normalized again, and then the image data is processed using the dual-group feedforward network;

[0019] The dual-channel feature extraction includes channel attention, the local convolution module, the adaptive interaction module and the dual-group feedforward network. The dual-channel feature extraction normalizes the image data, and then the image data is processed in parallel by the channel window attention and the local convolution module, and then the image data is normalized again, and then the image data is processed using the dual-group feedforward network.

[0020] In the method as described above, optionally, in the spatial window attention, the input dimension is R H×W×3 A matrix X of , wherein H represents the height of the image, W represents the width of the image, and 3 represents that the image has three color channels, linearly projecting the matrix X to obtain a linear projection matrix, and multiplying the matrix X with the corresponding linear projection matrix to obtain a query Q matrix, a key K matrix, and a value V matrix,

[0021] Q=XW Q , K = XW K , V = XW V

[0022] Among them, W Q , W K , W V is the linear projection matrix omitting the bias,

[0023] Then the query Q matrix, key K matrix and value V matrix are divided into non-overlapping windows, and each window is flattened. Each window contains N w pixels, and after flattening, the reshaped projection matrices are represented as Q s ,K s ,V s , and Q s ,K s ,V s The dimension is in Indicates the number of windows, N w represents the number of pixels in each window, C represents the number of channels, and then Q s ,K s ,V s Divided into h heads, Q s =[Q s 1 ,…,Q s h ],K s =[K s 1 ,…,K s h ], and V s =[V s 1 ,…,V s h ], the dimension of each of the heads is The output Y of the i-th head s i Defined as:

[0024]

[0025] Where D represents the relative position code,

[0026] The output Y for all the heads s i Reshape and concatenate to get a dimension of R H×W×C The matrix Y s , the spatial window attention output,

[0027] Y s =concat(Y s 1 ,…,Y s h ),

[0028] SW-SA(X)=Y s W p

[0029] Among them, W p The dimension is R C×C The linear projection of all features is fused.

[0030] In the method described above, optionally, the local convolution module includes extracting local features of the image data through a convolution operation, using depthwise separable convolution to more effectively and deeply extract the local features, then using a compression and excitation attention mechanism to optimize the local features, and again using the convolution operation to integrate and extract the local features.

[0031] In the method as described above, optionally, the adaptive interaction module includes spatial interaction and channel interaction. In the adaptive interaction module, feature F1 and feature F2 are input, and the dimension of feature F1 and feature F2 is R H×W×C , the feature F1 is the output of the spatial window attention, the feature F2 is the output of the local convolution module, and the channel attention map of the spatial interaction is calculated as follows:

[0032] S-Map(F1)=f(W2σ(W1F1))

[0033] C-Map(F1)=f(W4σ(W3H GP (F1)))

[0034] Among them, H GP represents global average pooling, f(.) represents the sigmoid function, and σ(.) represents the GELU function. W(.) represents the point-by-point convolution weight used to reduce or enlarge the channel dimension. The scaling ratios of W1 and W2 are r1 and C / r1 respectively. The scaling ratio of W3 is r2, and the expansion ratio of W4 is r2.

[0035] The calculation formula of the spatial interaction is:

[0036] SI(F1,F2)=A⊙S-Map(F1)

[0037] The calculation formula for the channel interaction is:

[0038] CI(F1,F2)=A⊙C-Map(F1)

[0039] where ⊙ represents element-wise multiplication.

[0040] In the method described above, optionally, the local convolution module includes extracting local features of the image data through a convolution operation, using depthwise separable convolution to more effectively and deeply extract the local features, then using a compression and excitation attention mechanism to optimize the local features, and again using the convolution operation to integrate and extract the local features.

[0041] In the method as described above, optionally, the formula of the dual-group feedforward network is:

[0042] X exp1 =Exp1(X input )

[0043] X exp2 =DWConv(X exp1 )

[0044] X1,X2=Split(X exp2 )

[0045] X s-gate =X1.Gelu(X1)

[0046] X c-gate =X2.Gelu(X1)

[0047] X output =Sqz(Cat(X s-gate ,X c-gate )

[0048] Among them, X input and X output Represent the input and output features respectively, X (.) represents the intermediate features, Exp1 and Sqz are channel expansion and squeezing operations respectively, and DWConv is the spatial information extraction achieved through 3×3 depthwise separable convolution.

[0049] In the method as described above, optionally, the upsampling method is a pixel rearrangement upsampling method.

[0050] In the method as described above, optionally, in step S5, when the super-resolution deep learning model is trained, the image input into the super-resolution deep learning model is divided into image blocks of 64 pixels × 64 pixels, the number of iterations of the super-resolution deep learning model is 200,000, and the number of data samples simultaneously input into the super-resolution deep learning model in each iteration is 16.

[0051] In the method as described above, optionally, the super-resolution deep learning model is trained by an optimizer, and the parameters β1 and β2 of the optimizer are set to 0.9 and 0.99, wherein β1 is used to estimate the homogeneity of the gradient and β2 is used to estimate the variance of the gradient, and the super-resolution deep learning model is optimized by minimizing the mean absolute error loss function, and the initial learning rate of the super-resolution deep learning model is set to 5×10 -3 , and dynamically adjust the learning rate, which is halved based on the previous iteration number [50000, 100000, 150000, 175000] as a mark.

[0052] In order to achieve the aforementioned objectives, the second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions or a computer program, and when the computer-executable instructions or the computer program are executed by a processor, the method as described in any one of the aforementioned first aspects is implemented.

[0053] In order to achieve the aforementioned object, a third aspect of the present invention provides a data processing device, wherein the data processing device comprises:

[0054] A memory, wherein the memory is used to store computer executable instructions or computer programs;

[0055] A processor, wherein the processor is used to implement the method as described in any one of the first aspects above when executing the computer executable instructions or computer programs stored in the memory.

[0056] Through the above technical solution, a certain number of images are selected from the mine data set, and then image denoising and data enhancement are performed through preprocessing; a super-resolution deep learning model is constructed; a super-resolution deep learning model is trained; the mine image to be reconstructed is input, and the super-resolution deep learning model is used to perform super-resolution reconstruction on the mine image to be reconstructed. By constructing a super-resolution deep learning model, the present invention effectively improves the edge details of the mine image and enhances the degree of fusion of feature information, making the reconstructed mine image clearer and richer in details. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The disclosure of the present invention will become more apparent with reference to the accompanying drawings. It should be understood that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings:

[0058] Figure 1 A flowchart of the steps of a mine image super-resolution reconstruction method based on deep learning of the present invention;

[0059] Figure 2 A block diagram of a super-resolution reconstruction network model constructed by the present invention;

[0060] Figure 3 for Figure 2 Block diagram of dual-space feature extraction in the super-resolution reconstruction network model;

[0061] Figure 4 for Figure 2 Block diagram of dual-channel feature extraction in the super-resolution reconstruction network model;

[0062] Figure 5 for Figure 3 and Figure 4 Block diagram of the local convolution module in the structure;

[0063] Figure 6 yes Figure 3 and Figure 4 Block diagram of the adaptive interaction module in the structure;

[0064] Figure 7 yes Figure 3 and Figure 4 Structural diagram of the dual-group feedforward network in the structure;

[0065] Figure 8 It is an output test image of other reconstruction methods and the method of the present invention. DETAILED DESCRIPTION

[0066] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0067] Figure 1 This is a flowchart of the steps of a mine image super-resolution reconstruction method based on deep learning of the present invention.

[0068] In such Figure 1 In the illustrated embodiment, the mine image super-resolution reconstruction method based on deep learning includes the following steps:

[0069] In step S1, the mine image super-resolution reconstruction task is started.

[0070] In step S2, image data in a mine environment is acquired.

[0071] It can be understood that there are unique environmental conditions in the mine, including complex tunnel layouts, various types of equipment, etc. The image data in the mine environment can intuitively reflect the actual scene appearance of the mine. A model is established, and the input image data is processed according to the mine image super-resolution reconstruction method based on deep learning of the present invention. In an optional embodiment, the experimental process of the model can use a mine data set as the basic material. The mine data set is specially constructed for the mine environment and includes images collected from underground coal mines.

[0072] In step S3, the image is preprocessed by using image denoising technology to denoise the image and enhance the image data so as to increase the image data by multiple times.

[0073] Image denoising technology is used to remove noise from images. Image denoising technology such as Gaussian filtering can effectively reduce image noise and improve image clarity. It is understood that some noise will be generated in the process of image acquisition and transmission, such as randomly appearing bright spots, dark spots and other interfering pixels, and there may be some abnormal values ​​that obviously deviate from the normal pixel value range. Gaussian filtering is an image processing technology. Gaussian filtering calculates the weighted average of each pixel in the image and its surrounding pixels based on the Gaussian function. The closer the pixel is to the center pixel, the higher the weight it has in the weighted average, and the farther it is, the lower the weight. Gaussian filtering can smooth the image and remove noise, and retain the better edge features of the image.

[0074] Methods for enhancing the image data include transforming, cropping, randomly rotating 90°, 180° and 270°, and horizontally flipping the image.

[0075] In different embodiments, the image transformation may include various forms, such as changing the brightness and contrast of the image, so that the image presents different visual effects. In an optional embodiment, by randomly rotating the image by 90°, 180° and 270○ and flipping it horizontally, images of different angles and orientations are added, the diversity of data is increased, and the model can be exposed to more images of the same content in different forms when learning, thereby improving the generalization ability of the model. It should be understood that the generalization ability of the model refers to the ability of the model to make accurate judgments and output reasonable results when facing new data. After image preprocessing, the number of pictures in the mine data set can be increased to multiple times the original number. The mine data set has more image data, which helps to further optimize the effect of model training, so that the model can learn more comprehensive and accurate underground image features.

[0076] In step S4, a super-resolution deep learning model is constructed.

[0077] The super-resolution deep learning model includes a shallow feature extraction module, a high-level feature extraction module and a reconstruction module. The shallow feature extraction module extracts shallow features of the image by using a convolution layer. Shallow features include information such as the image's contour and edge, color, texture and shape. The high-level feature extraction module is used to extract high-level features of image data. High-level features include high-level semantic information of the image, contextual relationships in the image, spatial layout, event detection, and attribute recognition. High-level semantic information includes objects and scenes in the image. High-level features help computers understand and analyze image content more deeply. In an optional embodiment, the constructed super-resolution deep learning model includes a shallow feature extraction module, four high-level feature extraction modules and a reconstruction module.

[0078] In step S5, the enhanced data is used to train a model of a super-resolution reconstruction network so that it has the ability to reconstruct a high-resolution image from a low-resolution image.

[0079] Each time, data of a certain batch size and a certain image block size are input into the model to train the model. It should be understood that smaller image blocks can focus on local details, but may lose overall relevance; larger image blocks take into account more overall information, but will increase the amount of calculation and may not capture enough local subtle features. Batch size refers to the number of data samples fed into the model for training in one iteration. The choice of batch size will affect the efficiency and stability of model training. A smaller batch size allows the model to be updated more frequently, and a larger batch size makes the process of training the model more stable. In an optional embodiment, each image input into the super-resolution deep learning model is divided into 64 pixels × 64 pixels image blocks, and the batch size is 16.

[0080] The number of iterations refers to the number of times the model completely traverses the entire training data set and updates the parameters. The number of iterations determines the duration of model training and the degree of adequacy of model learning. Through multiple iterations, the model can continuously adjust its own parameters so that it can better fit the rules in the training data. In an optional embodiment, the number of training iterations is set to 200,000.

[0081] In an optional embodiment, the Adam optimizer is selected to update and optimize the parameters of the model, and its parameters β1 are set to 0.9 and β2 are set to 0.99, where β1 is used to estimate the homogeneity of the gradient, and β2 is used to estimate the variance of the gradient, and the model is optimized by minimizing the mean absolute error loss function. It can be understood that the mean absolute error function is the average value of the absolute value of the difference between the predicted value and the true value. During the training process of the model, the model will output the predicted value of the image feature, and then compare the predicted value with the corresponding actual true value of the image feature, calculate the value of the mean absolute error loss function, and continuously adjust the parameters of the model so that the value of the mean absolute error loss function becomes smaller and smaller, which means that the model's predicted value of the image feature is getting closer and closer to the corresponding actual true value of the image feature, thereby improving the accuracy and performance of the model.

[0082] The learning rate determines the step size of each parameter update in model training. If the learning rate is too large, the model may not converge or even diverge, that is, it cannot find the optimal parameter value and the training effect is very poor; if the learning rate is too small, the model training will be very slow and it will take a long time to achieve good performance. In an optional embodiment, the initial learning rate is set to 5×10 -3 . And the present invention dynamically adjusts the learning rate so as to reasonably adjust the speed of parameter update according to the convergence of the model at different stages of training. In an optional embodiment, the number of iterations [50000, 100000, 150000, 175000] is marked and halved on the basis of the previous one. That is, when the model is trained to the 50000th iteration, the learning rate becomes half of the original; when it reaches the 100000th iteration, the learning rate is halved again, and so on, and the learning rate is halved at the subsequent marked iterations.

[0083] In an optional embodiment, the model of the present invention is implemented under the PyTorch framework based on the Geforce RTX 3090 graphics processor. Geforce RTX 3090 is a powerful graphics processor with strong computing power, which can accelerate a large number of matrix operations and other computing tasks during model training and improve the efficiency of training. PyTorch is a widely used open source framework for deep learning. It provides a rich function library, convenient model building and training interfaces, etc., which facilitates developers to efficiently implement various deep learning models. Here, PyTorch is selected in combination with the RTX 3090 hardware platform, which can well implement the described model training and related operations, and finally build a model that meets the requirements.

[0084] In step S6, the mine image to be reconstructed is input, and the super-resolution deep learning model is used to perform super-resolution reconstruction on the mine image to generate a high-quality image for intelligent analysis and processing of the mine environment.

[0085] Figure 2 A block diagram of the super-resolution reconstruction network model constructed by the present invention.

[0086] like Figure 2 As shown, the super-resolution reconstruction network includes an input image, which is then processed by a shallow feature extraction module, a high-level feature extraction module, and a reconstruction module in sequence to output an image.

[0087] The input image is represented by X LR indicates that LR means low resolution, X LR It can be represented as a three-dimensional array R H×W×3 , H represents the height of the image, W represents the width of the image, 3 represents that the image has three color channels, channel refers to the representation of independent color components or other characteristic components in the image, 3 usually refers to red, green, and blue channels, which are used to present color images.

[0088] exist Figure 2 In the illustrated embodiment, the number of high-level feature extraction modules is 4, and each high-level feature extraction module includes 3 dual-space feature extractions and 3 dual-channel feature extractions.

[0089] like Figure 2 As shown, in the reconstruction module, an upsampling method is used to upsample the deep features finally extracted by the shallow feature extraction module and the high-level feature extraction module, and a convolutional layer is used to aggregate features before and after the upsampling operation. The reconstruction module converts the low-resolution image data into high-resolution image data.

[0090] Before upsampling, the input deep features are further clustered and integrated. The convolution layer has powerful local feature extraction and feature fusion capabilities. By sliding the convolution kernel on the feature map for convolution operation, the features of adjacent areas can be weighted and combined, making the features more concentrated and semantically clearer, and extracting more valuable feature representations for subsequent upsampling and image reconstruction. In different embodiments, the parameters of the convolution layer are set according to actual needs, such as setting the convolution kernel size to control the range of local areas involved in each feature aggregation, setting the step size to control the change in the feature map size and the sparsity of feature extraction. Through the convolution operation of the convolution layer, the deep features are optimized. Feature representation is prepared for the subsequent upsampling, allowing upsampling to generate high-quality, high-resolution images based on better and more compact features.

[0091] Specifically, upsampling is the process of converting low-resolution data into high-resolution data. In an optional embodiment, the upsampling method may be a pixel rearrangement upsampling method. The pixel rearrangement upsampling method is a technique for improving image resolution by rearranging and combining the elements in the feature map to increase the resolution of the image, and while increasing the resolution, retaining the image semantic information contained in the deep features as much as possible, which helps to restore a higher quality image. After upsampling, a convolution operation is used to help fuse features of different scales to generate a higher quality high-resolution image.

[0092] Figure 3 for Figure 2 Block diagram of dual-space feature extraction in the super-resolution reconstruction network model.

[0093] Figure 4 for Figure 2 Block diagram of dual-channel feature extraction in the super-resolution reconstruction network model.

[0094] like Figure 3 Shown and Figure 4 As shown, both the dual-space feature extraction and the dual-channel feature extraction include a local convolution module LCB, an adaptive interaction module AIM, and a dual-group feedforward network DGFN, and include a spatial window attention SW-SA and a channel window attention CW-SA, respectively. The dual-space feature extraction performs LN normalization on the image data, and then after the spatial window attention SW-SA and the local convolution module LCB process the image data in parallel, the image data is normalized again by LN, and then the image data is processed using the dual-group feedforward network DGFN. The dual-channel feature extraction performs LN normalization on the image data, and then after the channel window attention CW-SA and the local convolution module LCB process the image data in parallel, the image data is normalized again by LN, and then the image data is processed using the dual-group feedforward network DGFN.

[0095] In the spatial window attention SW-SA, we input a dimension R H×W×3 The matrix X of the image, where H represents the height of the image, W represents the width of the image, and 3 represents that the image has three color channels, is linearly projected on X. The query Q matrix, key K matrix, and value V matrix are obtained by multiplying X with the corresponding linear projection matrix:

[0096] Q=XW Q , K = XW K , V = XW V ,

[0097] Among them, W Q , W K , W V is the linear projection matrix with the bias omitted.

[0098] Then the query Q matrix, key K matrix, and value V matrix are divided into non-overlapping windows and each window is flattened. Each window contains N w pixels, and after flattening, the reshaped projection matrices are represented as Q s ,K s ,V s ,and in Indicates the number of windows, N w represents the number of pixels in each window, C represents the number of channels, and then Q s ,K s ,V s Divided into h heads: Q s =[Q s 1 ,…,Q s h ],K s =[K s 1 ,…,K s h ], and V s =[V s 1 ,…,V s h ]. The dimension of each head is The output Y of the i-th head s i Defined as:

[0099]

[0100] The D in the formula represents the relative position encoding. Specifically, Q s ,K s ,V sIt is divided into h heads, and the features of the original C channels are evenly distributed to the h heads, so that each head can process a part of the channel features. Through the parallel processing of multiple heads, the richness and diversity of the feature relationship captured by the model are increased, and more comprehensive and detailed feature association information can be mined.

[0101] By s i Reshape and concatenate to get a dimension of R H×W×C The matrix Y s Finally, the spatial window attention SW-SA output is as follows:

[0102] Y s =concat(Y s 1 ,…,Y s h ),

[0103] SW-SA(X)=Y s W p

[0104] Among them, W p is a dimension R C×C The linear projection of all features is fused.

[0105] Spatial window attention (SW-SA) focuses on capturing features in the spatial dimension, focusing on the features of different local areas in space and the associations between them by dividing the input into different windows and other operations; channel window attention (CW-SA) focuses on the extraction and interaction of features in the channel dimension, mining the information connection between different channels. When they are executed alternately, they can capture spatial and channel features at the same time to a certain extent, and obtain relevant feature information in image data from different dimensions.

[0106] Figure 5 for Figure 3 and Figure 4 Block diagram of the local convolution module in the architecture.

[0107] like Figure 5As described, the local convolution module performs convolution, depthwise separable convolution, SE attention mechanism, and convolution again in sequence. It can be understood that the convolution operation is to extract local features of image data by sliding a defined convolution kernel on the image data. Depthwise separable convolution includes depthwise convolution and pointwise convolution. Depthwise convolution performs convolution operation on each channel of image data, thereby reducing the complexity of feature mixing between different channels of the image. Pointwise convolution performs cross-channel information fusion on the results of depthwise convolution output through a 1×1 convolution kernel, which plays a role in integrating features of different channels. A convolution kernel is a small matrix used to extract features from input data. A 1×1 convolution kernel means that the height of the convolution kernel is one pixel and the width is also one pixel. Depthwise separable convolution can not only ensure the extraction effect of local features of the image, but also make the super-resolution deep learning model run more efficiently. The SE attention mechanism compresses the input features to obtain a global feature description, and then generates different weights for different channels through an excitation operation based on the global feature description. Convolution is performed again to integrate and extract the features obtained after the above convolution, depthwise separable convolution, and SE attention mechanism processing, and the local convolution module outputs local features.

[0108] Figure 6 yes Figure 3 and Figure 4 Block diagram of the adaptive interaction module in the architecture.

[0109] like Figure 6 As shown, the adaptive interaction module space interaction and channel interaction, in the adaptive interaction module, input feature F1 and feature F2, the dimension of feature F1 and feature F2 is R H×W×C , feature F1 is the output of the spatial window attention, feature F2 is the output of the local convolution module, taking feature F1 as an example, the channel attention map of spatial interaction is calculated as follows:

[0110] S-Map(F1)=f(W2σ(W1F1))

[0111] C-Map(F1)=f(W4σ(W3H GP (F1)))

[0112] Among them, H GP represents global average pooling, f(.) represents the sigmoid function, σ(.) represents the GELU function. W(.) represents the point-by-point convolution weight used to reduce or enlarge the channel dimension. The scaling ratios of W1 and W2 are r1, C / r1 respectively, the scaling ratio of W3 is r2, and the expansion ratio of W4 is r2.

[0113] The calculation formula for spatial interaction is:

[0114] SI(F1,F2)=A⊙S-Map(F1)

[0115] The calculation formula for channel interaction is:

[0116] CI(F1,F2)=A⊙C-Map(F1)

[0117] Among them, ⊙ represents element-wise multiplication, which is used to adjust the features according to the weights of the attention map.

[0118] It can be understood that the channel attention map of spatial interaction is a map used to measure the importance of different channels of features in the process of spatial interaction. Through calculation, a result that can reflect the channel attention weight in the spatial dimension is finally obtained. The sigmoid function is a nonlinear activation function that is used to introduce nonlinearity in the network to help the network learn more complex feature representations. The GELU function is an activation function based on the Gaussian error function. It formally combines the characteristics of linear transformation and nonlinear activation, which helps the model better capture the feature information in the input data.

[0119] Figure 7 yes Figure 3 and Figure 4 Block diagram of the dual-group feedforward network in the structure.

[0120] The formula of the dual-group feedforward network DGFD is:

[0121] X exp1 =Exp1(X input )

[0122] X exp2 =DWConv(X exp1 )

[0123] X1,X2=Split(X exp2 )

[0124] X s-gate =X1.Gelu(X1)

[0125] X c-gate =X2.Gelu(X1)

[0126] X output =Sqz(Cat(X s-gate ,X c-gate )

[0127] Among them, X input and X output Represent the input and output features respectively, X (.) represents the intermediate features, Exp1 and Sqz are channel expansion and squeezing operations respectively, and DWConv is the spatial information extraction achieved through 3×3 depthwise separable convolution.

[0128] like Figure 7 As shown in Figure 1, the dual-group feedforward network consists of two groups of feedforward networks. A feedforward network refers to a network in which information is transmitted unidirectionally from the input layer to the output layer through the hidden layer. The dual-group feedforward network uses 1×1 convolution Conv and 3×3 deep DWConv to enrich local features.

[0129] The structure of the dual-group feedforward network includes:

[0130] The input feature map consists of three channels; after the convolution operation Conv with a convolution kernel size of 1×1, the information between channels is fused; the depth-wise separable convolution layer operation DWConv with a convolution kernel size of 3×3 is performed to further extract features; after the convolution operation with a convolution kernel size of 1×1, the channel information is integrated to obtain the output feature map.

[0131] The dual-group feed-forward network can recover the channel features lost during the spatial gating process, thereby enhancing the multi-level nature of feature representation.

[0132] Figure 8 It is an output test image of other reconstruction methods and the method of the present invention.

[0133] Common super-resolution reconstruction methods include Bicubic interpolation and SRCNN. Bicubic interpolation is a method based on polynomial interpolation. It calculates the interpolation result by considering the grayscale values ​​of 16 pixels around the interpolation point. SRCNN is a super-resolution reconstruction method based on convolutional neural networks. By learning the mapping relationship from low-resolution images to high-resolution images, it can restore richer details to a certain extent.

[0134] like Figure 8 As shown in the figure, the image reconstructed by the Bicubic method is relatively blurred. Although the resolution is improved to a certain extent, the detail loss is relatively serious. The SRCNN method recovers more details, but there is over-smoothing or distortion in some areas. In comparison, the mine image super-resolution reconstruction method based on deep learning of the present invention visually exhibits better detail recovery ability, clearer images, and richer details.

[0135] Furthermore, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the deep learning-based mine image super-resolution reconstruction method as in any of the aforementioned embodiments is implemented.

[0136] It should be noted that the computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0137] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages.

[0138] Furthermore, the present invention also provides a data processing device, which may include: a memory and a processor, wherein the memory can be used to store computer-executable instructions or computer programs, and the processor is used to execute the computer-executable instructions or computer programs stored in the memory, thereby implementing the deep learning-based mine image super-resolution reconstruction method as in any of the aforementioned embodiments.

[0139] The flow chart in the accompanying drawings illustrates the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present invention. In this regard, each square frame in the flow chart can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two square frames represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square frame in the flow chart, and the combination of the square frames in the flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0141] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0142] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A mine image super-resolution reconstruction method based on deep learning, characterized in that: The method comprises the following steps: S1: Start the mine image super-resolution reconstruction task; S2: Acquire image data in a mine environment; S3: preprocessing the image, using image denoising technology to denoise the image and enhance the image data so as to increase the number of the image by multiple times; S4: Build a super-resolution deep learning model; S5: Using the enhanced image data to train a super-resolution deep learning model; S6: Input a mine image to be reconstructed, and use the super-resolution deep learning model to perform super-resolution reconstruction on the mine image.

2. The method according to claim 1, characterized in that In step S2, the image is acquired from a mine dataset.

3. The method according to claim 1, characterized in that In step S3, the image denoising technique is Gaussian filtering, and the method of enhancing the image data includes transforming, cropping, randomly rotating 90°, 180° and 270°, and horizontally flipping the image.

4. The method according to claim 1, characterized in that In step S4, the super-resolution deep learning model includes: A shallow feature extraction module, wherein the shallow feature extraction module uses a convolution layer to extract shallow features of the image, wherein the shallow features include contours and edges, colors, textures, and shapes of the image; A high-level feature extraction module, the high-level feature extraction module includes dual-space feature extraction and dual-channel feature extraction, the high-level feature extraction module is used to extract high-level features of the image data, the high-level features include high-level semantic information of the image, contextual relationships in the image, spatial layout, event detection, and attribute recognition. A reconstruction module, wherein the reconstruction module uses an upsampling method to upsample the deep features finally extracted by the shallow feature extraction module and the high-level feature extraction module, and uses a convolutional layer to aggregate features before and after the upsampling operation. The reconstruction module converts the low-resolution image data into the high-resolution image data.

5. The method according to claim 4, characterized in that The super-resolution deep learning model includes 1 shallow feature extraction module, 4 high-level feature extraction modules and 1 reconstruction module, and each high-level feature extraction module includes 3 dual-space feature extractions and 3 dual-channel feature extractions.

6. The method according to claim 5, characterized in that The dual-space feature extraction includes a spatial window attention, a local convolution module, an adaptive interaction module and a dual-group feedforward network. The dual-space feature extraction normalizes the image data, and then after the spatial window attention and the local convolution module process the image data in parallel, the image data is normalized again, and then the image data is processed using the dual-group feedforward network; The dual-channel feature extraction includes channel attention, the local convolution module, the adaptive interaction module and the dual-group feedforward network. The dual-channel feature extraction normalizes the image data, and then the image data is processed in parallel by the channel window attention and the local convolution module, and then the image data is normalized again, and then the image data is processed using the dual-group feedforward network.

7. The method according to claim 6, characterized in that In the spatial window attention, the input dimension is R H×W×3 A matrix X of , wherein H represents the height of the image, W represents the width of the image, and 3 represents that the image has three color channels, linearly projecting the matrix X to obtain a linear projection matrix, and multiplying the matrix X with the corresponding linear projection matrix to obtain a query Q matrix, a key K matrix, and a value V matrix, Q=XW Q ,K=XW K ,V=XW V Among them, W Q , W K , W V is the linear projection matrix omitting the bias, Then the query Q matrix, key K matrix and value V matrix are divided into non-overlapping windows, and each window is flattened. Each window contains N w pixels, and after flattening, the reshaped projection matrices are represented as Q s ,K s ,V s , and Q s ,K s ,V s The dimension is in Indicates the number of windows, N w represents the number of pixels in each window, C represents the number of channels, and then Q s ,K s ,V s Divided into h heads, Q s =[Q s 1 ,…,Q s h ],K s =[K s 1 ,…,K s h ], and V s =[V s 1 ,…,V s h ], the dimension of each of the heads is The output Y of the i-th head s i Defined as: Where D represents the relative position code, The output Y for all the heads s i Reshape and concatenate to get a dimension of R H×W×C The matrix Y S , the spatial window attention output is as follows, AND s =concat(AND s 1 ,…,AND s h ), SW-SA(X)=Y s W p Among them, W p The dimension is R C×C The linear projection of all features is fused.

8. The method according to claim 6, characterized in that The local convolution module includes extracting local features of the image data through a convolution operation, using depthwise separable convolution to more effectively and deeply extract the local features, then using a compression and excitation attention mechanism to optimize the local features, and again using the convolution operation to integrate and extract the local features.

9. The method according to claim 6, characterized in that The adaptive interaction module includes spatial interaction and channel interaction. In the adaptive interaction module, feature F1 and feature F2 are input, and the dimensions of feature F1 and feature F2 are R H ×W×C , the feature F1 is the output of the spatial window attention, the feature F2 is the output of the local convolution module, and the channel attention map of the spatial interaction is calculated as follows: S-Map(F1)=f(W2σ(W1F1)) C-Map(F1)=f(W4σ(W3H GP (F1))) Among them, H GP represents global average pooling, f(.) represents the sigmoid function, σ(.) represents the GELU function. W(.) represents the point-by-point convolution weight used to reduce or enlarge the channel dimension. The scaling ratios of W1 and W2 are r1, C / r1 respectively, the scaling ratio of W3 is r2, and the expansion ratio of W4 is r2. The calculation formula of the spatial interaction is: SI(F1,F2)=A⊙S-Map(F1) The calculation formula for the channel interaction is: CI(F1,F2)=A⊙C-Map(F1) where ⊙ represents element-wise multiplication.

10. The method according to claim 6, characterized in that The formula of the dual-group feedforward network is: X exp1 =Exp1(X input ) X exp2 =DWConv(X exp1 ) X1,X2=Split(X exp2 ) X s-gate =X1.Gel(X1) X c-gate =X2.Frost(X1) X output =Sqz(Cat(X s-gate ,X c-gate ) Among them, X input and X output Represent the input and output features respectively, X (.) represents the intermediate features, Exp1 and Sqz are channel expansion and squeezing operations respectively, and DWConv is the spatial information extraction achieved through 3×3 depthwise separable convolution.

11. The method according to claim 4, characterized in that The upsampling method is a pixel rearrangement upsampling method.

12. The method according to claim 1, characterized in that In step S5, when the super-resolution deep learning model is trained, the image input into the super-resolution deep learning model is divided into image blocks of 64 pixels × 64 pixels, the number of iterations of the super-resolution deep learning model is 200,000, and the number of data samples simultaneously input into the super-resolution deep learning model in each iteration is 16.

13. The method according to claim 12, characterized in that The super-resolution deep learning model is trained by an optimizer, and the parameters β1 and β2 of the optimizer are set to 0.9 and 0.99, where β1 is used to estimate the homogeneity of the gradient and β2 is used to estimate the variance of the gradient. The super-resolution deep learning model is optimized by minimizing the mean absolute error loss function, and the initial learning rate of the super-resolution deep learning model is set to 5×10 -3 , and dynamically adjust the learning rate, which is halved based on the previous iteration with the number of iterations [50000, 100000, 150000, 175000] as the mark.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions or a computer program, and when the computer-executable instructions or the computer program are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

15. A data processing device, characterized in that: The data processing device comprises: A memory, wherein the memory is used to store computer executable instructions or computer programs; A processor, wherein the processor is configured to execute the computer executable instructions or the computer program stored in the memory to implement the method according to any one of claims 1 to 13.

Images