Image restoration method and device, electronic equipment and storage medium

Abstract

The application provides an image restoration method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. The image restoration method comprises the following steps: acquiring an initial image, performing feature extraction on the initial image, and obtaining a first feature set; inputting the first feature set into a U-shaped network for encoding processing and decoding processing, obtaining a target feature set after dimension reduction, and outputting a restored target image according to the target feature set and the initial image. The U-shaped network comprises a plurality of cascaded attention-like modules. Each attention-like module performs twice feature channel grouping processing on an input feature set input into the attention-like module, so that the downhole image can be efficiently restored.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, electronic device and storage medium. Background Technology

[0002] In the construction of modern intelligent mines, video surveillance systems have become a core infrastructure supporting safe production, intelligent inspection, and equipment operation and maintenance. The clarity of underground monitoring videos directly affects the accuracy of safety hazard identification and the reliability of equipment status monitoring. However, the complex environmental conditions in coal mines pose multiple challenges to video imaging quality: factors such as coal dust adhesion, ambient fog, and direct sunlight from equipment can easily cause dirt to accumulate on camera lenses, leading to imaging defects such as partial obstruction, blurred images, and glare interference. These image quality problems not only reduce the effective coverage of the monitoring system but may also lead to misjudgments of abnormal equipment conditions and missed detection of personnel behavior characteristics, seriously restricting the decision-making of safety early warning systems.

[0003] As mines accelerate their digital transformation towards intelligent operation, achieving video image enhancement and intelligent pollutant perception in multi-source interference environments has become a key challenge in overcoming the technological bottlenecks of intelligent sensing in mines. Currently, research on video enhancement algorithms integrating computer vision and deep learning technologies is becoming a crucial breakthrough for improving the level of visual monitoring in mines. In recent years, with the rapid development of deep learning technology, neural network-based image restoration methods have made significant progress in areas such as image inpainting, image dehazing, image reflection removal, image denoising, image super-resolution, and low-light enhancement. Compared to traditional convolutional networks, the Transformer model directly models the dependencies between any positions in a sequence through a self-attention mechanism, breaking the limitations of the local receptive field of convolutional networks, and has thus attracted increasing attention from researchers.

[0004] However, the self-attention mechanism employed by the Transformer architecture to obtain long-range dependencies requires traversing all spatial pixels, resulting in a quadratic relationship between its spatial and time complexity and image resolution. This not only significantly reduces the inference efficiency of the algorithm in practical applications but also makes it difficult to effectively deploy the algorithm on low-computing-power devices. Although a large number of studies have emerged in recent years to reduce the complexity of Transformer models, existing methods still mainly focus on optimizing the spatial attention mechanism. They fail to fully exploit the natural advantage of the channel attention mechanism in maintaining linear complexity and lack the ability to dynamically adjust parameters to adapt the model to different scales. Summary of the Invention

[0005] This application aims to effectively solve the aforementioned technical problems. Therefore, the first objective of this application is to propose an image restoration method to achieve efficient restoration of downhole images.

[0006] The second objective of this application is to provide an image restoration device.

[0007] The third objective of this application is to propose an electronic device.

[0008] The fourth objective of this application is to provide a computer-readable storage medium.

[0009] The fifth objective of this application is to provide a computer program product.

[0010] To achieve the above objectives, a first aspect of this application provides an image restoration method, comprising:

[0011] An initial image is acquired, and features are extracted from the initial image to obtain a first feature set;

[0012] The first feature set is input into a U-shaped network for encoding and decoding to obtain a dimensionality-reduced target feature set. Based on the target feature set and the initial image, the restored target image is output. The U-shaped network includes multiple cascaded attention modules, and each attention module performs feature channel grouping processing twice on its own input feature set.

[0013] To achieve the above objectives, a second aspect of this application provides an image restoration apparatus, comprising:

[0014] The first acquisition module is used to acquire an initial image and extract features from the initial image to obtain a first feature set;

[0015] The second acquisition module is used to input the first feature set into a U-shaped network for encoding and decoding to obtain a dimensionality-reduced target feature set, and output the restored target image based on the target feature set and the initial image. The U-shaped network includes multiple cascaded attention-like modules, and each attention-like module performs feature channel grouping processing twice on its own input feature set.

[0016] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory communicatively connected to the processor;

[0017] The memory stores computer-executed instructions;

[0018] The processor executes computer execution instructions stored in the memory to implement the method described in the first aspect embodiment.

[0019] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in the first aspect embodiment.

[0020] To achieve the above objectives, a fifth aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0021] The image restoration method, apparatus, electronic device, and storage medium provided in this application acquire an initial image collected underground, extract features from the initial image to obtain a first feature set, input the first feature set into a U-shaped network, the U-shaped network including multiple cascaded attention-like modules, each attention-like module performing feature channel grouping processing twice on its own input feature set to obtain the final target feature set, and fusing the target feature set with the initial image to obtain the restored target image, ensuring image restoration effect, while effectively reducing the number of module parameters, solving the pain points of parameter redundancy and high deployment cost of traditional Transformer models in underground coal mine scenarios. Attached Figure Description

[0022] Additional aspects and advantages of this application will become apparent and readily understood in conjunction with the accompanying drawings, in which:

[0023] Figure 1A A schematic diagram of a Transformer network module based on spatial attention mechanism provided in an embodiment of this application;

[0024] Figure 1B A schematic diagram of a Transformer network module based on channel attention mechanism provided in an embodiment of this application;

[0025] Figure 1C A schematic diagram of a Transformer network module based on a feature channel regrouping strategy provided in an embodiment of this application;

[0026] Figure 1D This is a schematic diagram of the NAFNet module provided in an embodiment of this application;

[0027] Figure 1 This is a schematic flowchart of an image restoration method provided in an embodiment of this application;

[0028] Figure 2 This is a schematic flowchart illustrating another image restoration method provided in an embodiment of this application;

[0029] Figure 2AA symmetrical U-shaped network structure diagram provided in the embodiments of this application;

[0030] Figure 2B This is an asymmetric U-shaped network structure diagram provided in an embodiment of this application;

[0031] Figure 3 A schematic diagram of a Transformer-like module provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application. Detailed Implementation

[0033] Embodiments of this application will be described in detail below, examples of which are illustrated in the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or functions. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0034] To address the issue of excessive complexity in Transformer models, this embodiment replaces the spatial attention mechanism in conventional Transformer network modules with a channel attention mechanism, such as... Figure 1A and Figure 1B As shown, where X represents the input feature, LN represents layer regularization, FFN represents the feedforward network, SA represents the spatial attention mechanism, and CA represents the channel attention mechanism. For ease of description, the modified Transformer module in this embodiment is referred to as the "Transformer-like module".

[0035] The Visual Geometry Group (VGG) network has played a crucial role in the development of deep learning and computer vision. One of its major contributions is the proposal to replace large kernel convolutions with cascaded small kernel convolutions. Under the premise of comparable model complexity, the VGG architecture offers greater network depth and better model performance. Based on this idea, this embodiment aims to construct a more flexible and efficient Transformer-like module. Referring to the feature channel grouping of the efficient convolutional architecture ShuffleNetV2, this application proposes a "feature channel regrouping strategy" to further reduce the module's complexity, such as... Figure 1C As shown, X1 and X2 both represent feature maps, and Split, Concat, and Shuffle correspond to feature grouping, feature concatenation, and random shuffling of feature channels, respectively.

[0036] Furthermore, replacing the multi-head self-attention mechanism in the Transformer module, the Nonlinear Activation Free Network (NAFNet) model uses a lightweight inverse residual module MobileNetV2, a channel attention module SCA, and an activation function SimpleGate to construct an efficient and lightweight Transformer-like image restoration module—the NAFNet module. Figure 1D As shown, this module is composed of two cascaded sub-modules, sub-module 1 (Block-1) and sub-module 2 (Block-2). Combining the feature channel grouping idea in the ShuffleNetV2 module, this application innovatively utilizes the "feature channel regrouping strategy" to construct a more flexible and efficient Transformer-like image restoration module. Since the proposed module integrates the advantages of both the ShuffleNetV2 module and the NAFNet module, for ease of understanding and description, this embodiment refers to this module as the SN module. The SN module is the core inventive point, and its specific structure and technical features are detailed in subsequent embodiments.

[0037] The image restoration method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.

[0038] Figure 1 This is a schematic flowchart illustrating an image restoration method provided in an embodiment of this application. Figure 1 As shown, the image restoration method includes the following steps:

[0039] S101, Obtain the initial image and extract features from the initial image to obtain the first feature set.

[0040] In some embodiments, the initial image is an image directly acquired by downhole equipment. This initial image may have low resolution or be blurry, requiring image restoration processing, which involves image repair, dehazing, or noise reduction to obtain a clear, high-resolution image.

[0041] Optionally, the method for feature extraction of the initial image includes, but is not limited to, the Harris corner detection algorithm or the scale-invariant feature transformation algorithm to extract features of the initial image; or, hierarchical feature extraction based on the convolutional layers of a convolutional neural network is performed to obtain multi-scale features and form a first feature set.

[0042] S102, the first feature set is input into the U-shaped network for encoding and decoding to obtain the dimensionality-reduced target feature set, and the restored target image is output based on the target feature set and the initial image.

[0043] It is understood that the U-shaped network is a symmetrical encoder-decoder structure fully convolutional neural network, including a shrinking path (corresponding to the encoder) and an expanding path (corresponding to the decoder). In this embodiment, the U-shaped network contains multiple cascaded attention-like modules, that is, SN modules, in which each attention-like module performs feature channel grouping processing twice on the input feature set.

[0044] In this embodiment, the U-shaped network includes an improved attention-based module that performs channel attention on grouped features. A first feature set is input into the U-shaped network, and multiple cascaded attention-based modules perform feature channel grouping processing on the input feature set. For example, the first feature set is input into a first-level attention-based module, which processes it to obtain a second feature set; the second feature set is input into a second-level attention-based module, which processes it to obtain a third feature set, and so on. Through the shrinking path (corresponding to encoding) and expanding path (corresponding to decoding) of the U-shaped network, a dimensionality-reduced target feature set is obtained.

[0045] Furthermore, the target image is reconstructed based on the target feature set and the initial image. This involves fusing the abstract features extracted by the U-shaped network with the original image information and reconstructing the target image through inverse mapping. This target image has higher resolution and clarity, resulting in a better presentation.

[0046] In this embodiment, an initial image collected underground is acquired, and features are extracted from this initial image to obtain a first feature set. The first feature set is then input into a U-shaped network for encoding and decoding to obtain a dimensionality-reduced target feature set. The U-shaped network includes multiple cascaded attention-like modules, each of which performs feature channel grouping processing twice on the input feature set to generate the final target feature set. The target feature set is then fused with the initial image to obtain the restored target image, ensuring image restoration effectiveness while effectively reducing the number of module parameters. This addresses the problems of parameter redundancy and high deployment costs associated with traditional Transformer models in underground coal mine scenarios.

[0047] Based on the above embodiments, Figure 2 This is a schematic flowchart illustrating another image restoration method provided in an embodiment of this application. Figure 2 As shown, the image restoration method includes the following steps:

[0048] S201: Obtain the initial image and extract features from the initial image to obtain the first feature set.

[0049] In the embodiments of this application, the implementation method of step S201 can be performed with reference to any of the disclosed embodiments, and will not be described again.

[0050] S202: If the U-shaped network is a symmetrical U-shaped network, the first feature set is input into the U-shaped network for encoding and decoding to obtain the target feature set after dimensionality reduction, and the restored target image is output according to the target feature set and the initial image.

[0051] In some embodiments, the symmetric U-shaped network comprises N levels (N being an integer greater than or equal to 2), wherein the Nth level comprises only a first-type attention module, while each of the other levels (levels 1 to N-1) comprises both a first-type attention module and a second-type attention module. For example, Figure 2A The symmetrical U-shaped network structure in this embodiment is shown.

[0052] For the i-th level (1≤i≤N-1), the output feature set of the first type of attention module is downsampled and then input into the (i+1)-th level first type of attention module. The output feature set of the (i+1)-th level second type of attention module is upsampled and then fused with the output feature set of the i-th level first type of attention module, serving as the input to the i-th level second type of attention module. This process continues until the output of the first level second type of attention module is downsampled to obtain the target feature set; where N is an integer greater than or equal to 2.

[0053] like Figure 2A As shown, each row represents one level, with a total of N=5 levels. The first square of each level represents the first type of attention module, and the second square represents the second type of attention module. The input of the first type of attention module in level 1 is the first feature set; its output, after downsampling, serves as the input of the first type of attention module in level 2. This process continues until the output of the first type of attention module in level N-1 is downsampled and used as the input of the first type of attention module in level N, thus obtaining its output feature set.

[0054] Furthermore, the output feature set of the first type of attention module at level N is upsampled and then fused with the output feature set of the first type of attention module at level N-1 through an adder, serving as the input feature set of the second type of attention module at level N-1. Correspondingly, the output feature set of the second type of attention module at level N-1 is upsampled and then fused with the output feature set of the first type of attention module at level N-2 through an adder, serving as the input feature set of the second type of attention module at level N-2, and so on, until the output feature set of the second type of attention module at level 2 is upsampled and then fused with the output feature set of the first type of attention module at level 1 through an adder, serving as the input feature set of the second type of attention module at level 1. The output feature set of the second type of attention module at level 1 is then downsampled to obtain the final target feature set.

[0055] In some embodiments, the process by which the attention module obtains the output feature set is as follows: the input feature set is grouped by feature channels to obtain a first feature subset and a second feature subset; optionally, the proportion of the first group corresponding to the feature channel group of the input feature set can be determined; based on the proportion of the first group, the input feature set is divided into a first feature subset and a second feature subset. For example, if the proportion of the first group is α, α∈(0,1], then the first feature subset is represented as X1=(1-α)*X, and the second feature subset is represented as X2=α*X, where X is the original input feature set.

[0056] In some embodiments, the second feature subset can also be processed by an attention mechanism to obtain a third feature subset. That is, the second feature subset is significantly enhanced by an attention mechanism to obtain a third feature subset that has been processed and optimized in depth.

[0057] Furthermore, feature concatenation and feature channel shuffling are performed on the first and third feature subsets to obtain a candidate feature set for the attention module; deep fusion of feature information between the first and third feature subsets is achieved through feature concatenation; to further enhance information flow and interaction between channels, a channel random shuffling mechanism is added, effectively breaking the boundaries of the original channels and promoting comprehensive information exchange and fusion.

[0058] Accordingly, the candidate feature set is grouped by feature channel to obtain a fourth feature subset and a fifth feature subset; optionally, the proportion of the second group of feature channel groups corresponding to the candidate feature set can be determined; based on the proportion of the second group, the candidate feature set is grouped by feature channel to obtain a fourth feature subset and a fifth feature subset; in this embodiment, the fourth feature subset and the fifth feature subset can be represented as: X4=(1-β)*X h With X5=β*X h Where X4 is the fourth feature subset, X5 is the fifth feature subset, and X... h Let be the candidate feature set, and β be the proportion of the second group when grouping the feature channels, β∈(0,1).

[0059] In some embodiments, the fifth feature subset can also be processed by a feedforward network to obtain the sixth feature subset. That is, the fifth feature subset is processed by a feedforward network to enhance the expressive power of the model and obtain the sixth feature subset.

[0060] Furthermore, the fourth and sixth feature subsets are concatenated and their channels are shuffled to obtain the output feature set of the attention module. By grouping and concatenating the feature channels twice, the information flow between channels is enhanced, enabling the entire network module to capture richer and more comprehensive feature information while maintaining its lightweight nature.

[0061] In this embodiment, the attention module consists of a first structural block and a second structural block connected in series:

[0062] First structural block:

[0063] The system consists of the first feature grouping unit (Split), the first layer normalization unit (LN), the channel attention mechanism unit (ChannelAttention, CA), the first feature concatenation unit (Concat), and the first feature channel shuffling unit (Shuffle), which are connected in sequence.

[0064] Second structural block:

[0065] The second feature grouping unit (Split), the second layer regularization unit (LN), the feedforward neural network unit (FFN), the second feature concatenation unit (Concat), and the second feature channel shuffling unit (Shuffle) are connected in sequence.

[0066] Specifically, such as Figure 1C As shown:

[0067] Split: First feature grouping unit and second feature grouping unit.

[0068] Concat: First feature concatenation unit and second feature concatenation unit.

[0069] Shuffle: First feature channel shuffling unit and second feature channel shuffling unit.

[0070] LN and CA in the first structural block: are the first-layer regularization unit and the channel attention mechanism unit, respectively.

[0071] LN and FFN in the second structural block: are the second-layer regularization unit and feedforward network unit, respectively.

[0072] Function Description:

[0073] The first feature grouping unit, the first LN unit, and the CA unit in the first structural block are used to process the second feature subset using an attention mechanism to obtain the third feature subset. Then, the first feature concatenation unit and the first feature channel shuffling unit perform feature concatenation and feature channel shuffling on the first and third feature subsets to obtain a candidate feature set.

[0074] The second feature grouping unit, the second LN unit, and the FFN unit in the second structural block are used to perform feedforward network processing on the fifth feature subset to obtain the sixth feature subset. Then, the second feature concatenation unit and the second feature channel shuffling unit perform feature concatenation and feature channel shuffling on the fourth and sixth feature subsets to obtain the output feature set.

[0075] In some embodiments, the computing power device associated with the downstream task can also be determined; the computing power device associated with the downstream task is the device that performs the task requirements in a specific application scenario, such as a central processing unit or a graphics processing unit.

[0076] Optionally, the computing power capability information of the computing devices is determined. Based on the computing power capability information, at least one of the proportion of the first group and the proportion of the second group is determined. The computing power capability information may include computing power indicators such as floating-point operations per second, trillion floating-point operations per second, trillion operations per second, and millions of instructions per second; if the computing power capability of the computing devices is high, the proportion of the first group or the proportion of the second group can be determined to be a larger value; if the computing power capability of the computing devices is low, the proportion of the first group or the proportion of the second group can be determined to be a smaller value.

[0077] Optionally, in this embodiment, the proportion of the first group is α, and the proportion of the second group is β. Conditions: When α = β = 0.5, the number of parameters and computational cost of the attention module is relatively large; when α = β = 0.25, the number of parameters and computational cost of the attention module is relatively small.

[0078] Understandably, the complexity of the SN module can be controlled by adjusting the feature channel grouping ratios α and β. To facilitate analysis of the SN module's complexity, let's take a single convolution with a bias term as an example; its parameter complexity (Params) and time complexity (FLOPs) are as follows:

[0079] params=K w *K h *C in *C out +C out ,

[0080] FLOPs = (K w *K h *C in *C out+C out )*W out *H out .

[0081] The parameter complexity (parameters) and time complexity (FLOPs) of a 3x3 depthwise separable convolution are as follows:

[0082] Params = 3 * 3 * C in +C out ,

[0083] FLOPs = (3 * 3 * C in +C out )*W out *H out .

[0084] Considering that the number of feature channels in a deep learning neural network is generally no less than 32, and therefore, with the same feature map resolution, the complexity of a 1*1 convolution is at least 3 times that of a 3*3 depthwise separable convolution.

[0085] In the SN module, the parameter complexity (Params) and time complexity (FLOPs) of a single convolution with a bias term are as follows:

[0086] Params = [K w *K h *(α*C in )*(α*C out )+α*C out ]+[K w *K h *(β*C in )*(β*C out )+β*C out ],

[0087] FLOPs = {[K w *K h *(α*C in )*(α*C out )+α*C out ]+[K w *K h *(β*C in )*β*Cout+β*Cout}*Wout*Hout.

[0088] Similarly, with the same feature map resolution, the complexity of a 1*1 convolution is at least three times that of a 3*3 depthwise separable convolution.

[0089] Based on the above formula, it can be seen that 1*1 convolutions account for the majority of the module's complexity in both the NAFNet and SN modules; the space complexity of a single module is O(C0). in *C out The time complexity is O(C). in *C out *W out *H out This means that the space and time complexity of a single module are both proportional to the product of the input and output channels; the SN module has approximately (α) times the complexity of the NAFNet module in both aspects. 2 +β 2 When α = β = 0.5, the SN module is about 25% of the baseline NAFNet module in both complexity aspects, which can significantly reduce the module's storage overhead and inference efficiency; and when the channel grouping ratio is expanded to 1:3 (corresponding to α = β = 0.25), the number of parameters and computation of the SN module can be further reduced to 6.25% of the baseline NAFNet module.

[0090] For a U-shaped network architecture with 32 initial channels, when α=β=0.5, the number of parameters in the SN-UNet-T model is 9.93MB and the computational cost is 35.72GB; when α=β=1, the SN-UNet-T model degenerates into NAFNet, at which point the number of parameters in the model is 17.11MB and the computational cost is 64.38GB.

[0091] S203: If the U-shaped network is an asymmetric U-shaped network, the first feature set is input into the U-shaped network for encoding and decoding to obtain the target feature set after dimensionality reduction, and the restored target image is output based on the target feature set and the initial image.

[0092] In some embodiments, the asymmetric U-shaped network comprises N levels, each containing only one attention-like module. For example, Figure 2B The asymmetric U-shaped network structure in this embodiment is shown.

[0093] For the i-th level (2≤i≤N), the output feature set of its attention-like module is downsampled and used as the input of the (i+1)-th level attention-like module; simultaneously, this output is downsampled to obtain the dimensionality-reduced output feature set. Here, N is an integer greater than or equal to 2. For example, the output of the 1st level is downsampled and used as the input of the 2nd level, resulting in the dimensionality-reduced output of the 1st level; the output of the 2nd level is downsampled and used as the input of the 3rd level, resulting in the dimensionality-reduced output of the 2nd level, and so on, until the output feature set and dimensionality-reduced output feature set of the Nth level are obtained.

[0094] Understandably, the input to the first-level attention module is the first feature set.

[0095] Furthermore, the output feature set of level 1 is concatenated with all the dimensionality-reduced output feature sets of levels 2 to N to obtain a concatenated feature set. This concatenated feature set is then subjected to attention processing and downsampling through an attention-like module to obtain the target feature set.

[0096] For existing Nonlinear Activation-Free Networks (NAFNet), a five-layer encoder-decoder structure is employed, with each layer distinguished by resolution. To ensure that the network architecture of this embodiment has a similar overall complexity to NAFNet, and considering that the complexity of the proposed SN module is 1 / 4 that of NAFNet (based on the default feature channel grouping ratio), this embodiment employs a 4x repetition cascading strategy to design the number of encoder-decoder modules. Specifically:

[0097] The number of modules in each stage of the NAFNet encoder is [1,1,1,28,1], and the number of decoders is [1,1,1,1].

[0098] The number of modules in each stage of the symmetric U-shaped network architecture SN-UNet encoder is [4,4,4,112,4], and the number of decoders is [4,4,4,4].

[0099] The number of modules in each stage of the asymmetric U-shaped SN-Net encoder is [4,4,4,112,4], and the number of modules in the decoder network stage is [1]. The decoder has only one SN module (instead of four), because when the model initializes more than 32 feature channels, a single SN module is sufficient to achieve feature fine-tuning.

[0100] In some embodiments, the U-shaped structure is initialized with 32 or 64 feature channels. Tables 1 and 2 show the network structure and complexity when the number of initialized channels is 32 and 64, respectively. For SN-UNet and SN-Net with an initialization of 32 channels: the number of feature channels at each level of the encoder is [32, 64, 128, 256, 512], and the number of decoder channels is set to [256, 128, 64, 32] and

[32] , respectively. Referring to Table 1, when the complexity of the U-shaped network architecture is comparable to that of the benchmark NAFNet, the U-shaped network is deeper: SN-UNet_B_W32 has 18.08MB of parameters, 68.16GB of floating-point computation, and 144 SN modules for the encoder-decoder; SN-Net_B_W32 has 17.13MB of parameters, 60.57GB of floating-point computation, and 129 SN modules for the encoder-decoder. The total number of layers in the proposed architectures all exceeds 120. Correspondingly, the baseline model NAFNet has 17.11MB of parameters, 64.38GB of computation, 36 NAFNet encoder-decoder modules, and a total of no more than 45 layers (including 4 downsampling and 4 upsampling).

[0101] In some embodiments, by adjusting the distribution of the number of modules in the encoder-decoder, a more efficient and lightweight architecture can be constructed, promoting flexible deployment and efficient inference of the model. When α = β = 0.5, the complexity of the lightweight model SN-UNet-T is only about half that of the baseline model NAFNet, specifically [9.93MB, 35.72GB] and [17.11MB, 64.38GB], respectively. The distribution of SN modules in the encoder-decoder of SN-UNet-T is [1, 1, 1, 28, 1] and [1, 1, 1, 1]. The complexity of the simplified model SN-UNet-S is only 2 / 3 that of NAFNet, specifically [10.09MB, 37.54GB] and [17.11MB, 64.38GB], respectively. The distribution of SN modules in the encoder-decoder of SN-UNet-S is [2, 2, 2, 56, 2] and [2, 2, 2, 2].

[0102] Compared to the encoder-decoder symmetric SN-UNet, the encoder-decoder asymmetric SN-Net has less complexity. Specifically, the lightweight model SN-Net-T has a lower complexity [5.61MB, 20.11GB] than the SN-UNet-T model [9.93MB, 35.72GB]; the simplified model SN-Net-S has a lower complexity [9.45MB, 33.59GB] than the SN-UNet-S model [10.09MB, 37.54GB]; and the standard model SN-Net-B has a lower complexity [17.13MB, 60.57GB] than the SN-UNet-B model [18.08MB, 68.16GB].

[0103] Table 1

[0104]

[0105] Table 2

[0106]

[0107] This embodiment follows the hyperparameter settings used in the NAFNet network for optimization. For example, the Adam optimizer and PSNR loss are used to train the model, where β1 = 0.9, β2 = 0.9, and the weight decay coefficient is 10. -3 The model underwent a total of 400,000 iterations, with the learning rate starting at 10%. -3 Gradually decrease to 10 -7 The decay strategy is cosine annealing; during training, models initialized with 32 channels use 32 as the image batch size, and models initialized with 64 channels use 64 as the image batch size; the random seed is set to a constant of 10 to reduce random errors in the experiment.

[0108] In this embodiment, the first feature set is encoded and decoded based on a symmetric U-shaped network or an asymmetric U-shaped network to obtain a dimensionality-reduced target feature set. During the output of the target feature set in the symmetric U-shaped network or the asymmetric U-shaped network, the feature set is downsampled and concatenated based on multiple cascaded attention-like modules in the network to finally obtain the output feature combination, which yields the dimensionality-reduced target feature set. Based on the target feature set and the initial image, the restored target image is output, for example, by reconstructing the restored target image through inverse mapping. The target image has higher resolution and higher clarity, resulting in a better presentation effect and solving the pain points of parameter redundancy and high deployment cost of traditional Transformer models in underground coal mine scenarios.

[0109] Based on the above embodiments, Figure 3 This is a schematic diagram of a Transformer-like module provided in an embodiment of this application. After inputting feature X, feature channels are grouped, spliced ​​and fused, and feature channels are shuffled. Then, a second feature channel grouping, splicing and fusion, and feature channel shuffling are performed. Feature fusion is fully realized through two feature channel groupings. Based on feature channel shuffling, the information flow between channels is enhanced, and richer and more comprehensive feature information can be captured.

[0110] To achieve the above embodiments, this application also proposes an image restoration apparatus.

[0111] Figure 4 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application. Figure 4 As shown, the image restoration device 400 includes:

[0112] First acquisition module 401: used to acquire an initial image and extract features from the initial image to obtain a first feature set;

[0113] The second acquisition module 402 is used to input the first feature set into the U-shaped network for encoding and decoding processing to obtain the dimensionality-reduced target feature set, and output the restored target image based on the target feature set and the initial image. The U-shaped network includes multiple cascaded attention modules, and each attention module performs feature channel grouping processing twice on its own input feature set.

[0114] In some implementations, the second acquisition module 402 includes:

[0115] The input feature set is grouped by feature channels to obtain a first feature subset and a second feature subset. The second feature subset is then processed by an attention mechanism to obtain a third feature subset.

[0116] The first and third feature subsets are concatenated and their channels are shuffled to obtain a candidate feature set for the attention module.

[0117] The candidate feature set is grouped by feature channel to obtain the fourth feature subset and the fifth feature subset;

[0118] The fifth feature subset is processed by a feedforward network to obtain the sixth feature subset;

[0119] The fourth and sixth feature subsets are concatenated and their feature channels are shuffled to obtain the output feature set of the attention module.

[0120] In some implementations, the attention module includes a first structure block and a second structure block connected in series. The first structure block includes a first feature grouping unit, a first-layer regularization LN unit, a channel attention mechanism (CA) unit, a first feature splicing unit, and a first feature channel scrambling unit connected in sequence. The second structure block includes a second feature grouping unit, a second LN unit, a feedforward network (FFN) unit, a second feature splicing unit, and a second feature channel scrambling unit connected in sequence.

[0121] In some implementations, the first feature grouping unit, the first LN unit, and the CA unit perform attention mechanism processing on the second feature subset to obtain the third feature subset, and then perform feature concatenation and feature channel scrambling on the first feature subset and the third feature subset through the first feature concatenation unit and the first feature channel scrambling unit to obtain the candidate feature set;

[0122] The second feature grouping unit, the second LN unit, and the FFN unit perform feedforward network processing on the fifth feature subset to obtain the sixth feature subset. The second feature concatenation unit and the second feature channel shuffling unit then perform feature concatenation and feature channel shuffling on the fourth and sixth feature subsets to obtain the output feature set.

[0123] In some implementations, the U-shaped network is a symmetric U-shaped network, which includes N levels. Each level except the Nth level includes two types of attention modules, namely the first type of attention module and the second type of attention module. The Nth level includes the first type of attention module.

[0124] For the i-th level (1≤i≤N-1, where N is an integer greater than or equal to 2), the output feature set of the first type of attention module at the i-th level is downsampled and used as the input feature set of the first type of attention module at the (i+1)-th level.

[0125] The output feature set of the (i+1)th level second type attention module is upsampled and then concatenated with the output feature set of the ith level first type attention module in an adder to serve as the input feature set of the ith level second type attention module. This process continues until the output feature set of the ith level second type attention module is downsampled to obtain the target feature set.

[0126] In some implementations, the U-shaped network is an asymmetric U-shaped network, which consists of N levels, where each level includes an attention-like module.

[0127] For the i-th level (2≤i≤N, where N is an integer greater than or equal to 2), the output feature set of the attention module of the i-th level is downsampled and used as the input feature set of the attention module of the (i+1)-th level.

[0128] The output feature set of the i-th level attention module is downsampled to obtain the dimensionality-reduced output feature set of the i-th level attention module.

[0129] The output feature set of the first-level attention module and the dimensionality-reduced output feature set of each of the remaining levels are concatenated to obtain the concatenated feature set. Then, an attention module is used to perform attention processing and downsampling processing on the concatenated feature set to obtain the target feature set.

[0130] In some implementations, the second acquisition module 402 includes:

[0131] Determine the proportion of the first group in the feature channel grouping corresponding to the input feature set;

[0132] Based on the proportion of the first group, the input feature set is grouped by feature channel to obtain the first feature subset and the second feature subset.

[0133] In some implementations, the second acquisition module 402 includes:

[0134] Determine the proportion of the second group in the feature channel grouping corresponding to the candidate feature set;

[0135] Based on the proportion of the second group, the candidate feature set is grouped by feature channel to obtain the fourth feature subset and the fifth feature subset.

[0136] In some implementations, device 400 also includes:

[0137] Identify the computing devices associated with downstream tasks;

[0138] Determine the computing power capacity information of the computing equipment;

[0139] Based on the computing power information, determine at least one of the proportions of the first group and the second group.

[0140] In some implementations, the first group has a proportion of α, and the second group has a proportion of β, where α = β = 0.5 or α = β = 0.25. The number of parameters and computational cost of the attention module when α = β = 0.5 is greater than the number of parameters and computational cost of the attention module when α = β = 0.25.

[0141] It should be noted that the foregoing explanation of the image restoration method embodiment also applies to the image restoration device of this embodiment, and will not be repeated here.

[0142] In this embodiment, the encoding and decoding of the first feature set are performed based on a symmetric U-shaped network or an asymmetric U-shaped network to obtain a dimensionality-reduced target feature set. During the output of the target feature set in the symmetric U-shaped network or the asymmetric U-shaped network, the feature set is downsampled and concatenated based on multiple cascaded attention-like modules in the network to finally obtain the output feature combination, which yields the dimensionality-reduced target feature set. Based on the target feature set and the initial image, the restored target image is output, for example, the restored target image is obtained through inverse mapping reconstruction. The target image has higher resolution and higher clarity, and the presentation effect is better, solving the pain points of parameter redundancy and high deployment cost of traditional Transformer models in underground coal mine scenarios.

[0143] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0144] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0145] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0146] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0147] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0148] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0149] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0150] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0151] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0152] The logic and / or steps represented or otherwise described in a flowchart can be considered as a sequence of executable instructions that implement logical functions. These instructions can be implemented in any computer-readable medium for use or in combination by an instruction execution system (such as a computer-based system, a processor-based system, etc.).

[0153] In this specification, "computer-readable medium" means any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system. Specific examples (not an exhaustive list) include: electrical connections having one or more wires (electronic devices), portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM).

[0154] In addition, computer-readable media may include paper or other media on which programs can be printed; programs can be electronically acquired and stored in computer memory through optical scanning, editing, interpretation, and other processing.

[0155] It should be understood that various parts of this application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware, which are stored in memory and executed by a suitable instruction execution system. If hardware implementation is used, one or a combination of the following techniques known in the art can be used: discrete logic circuits (having logic gates for implementing data signal logic functions), application-specific integrated circuits (having suitable combinational logic gates), programmable gate arrays (PGAs), or field-programmable gate arrays (FPGAs), etc.

[0156] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0158] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An image restoration method, characterized in that, The method includes: An initial image is acquired, and features are extracted from the initial image to obtain a first feature set; The first feature set is input into a U-shaped network for encoding and decoding to obtain a dimensionality-reduced target feature set. Based on the target feature set and the initial image, the restored target image is output. The U-shaped network includes multiple cascaded attention modules, and each attention module performs feature channel grouping processing twice on its own input feature set. The attention module performs two feature channel grouping processes on the input feature set itself, including: The input feature set is grouped by feature channels to obtain a first feature subset and a second feature subset, and the second feature subset is processed by an attention mechanism to obtain a third feature subset; The first feature subset and the third feature subset are concatenated and their feature channels are shuffled to obtain the candidate feature set of the attention module. The candidate feature set is grouped by feature channels to obtain a fourth feature subset and a fifth feature subset; The fifth feature subset is processed by a feedforward network to obtain the sixth feature subset; The fourth feature subset and the sixth feature subset are concatenated and their feature channels are shuffled to obtain the output feature set of the attention module. The attention module includes a first structure block and a second structure block connected in series. The first structure block includes a first feature grouping unit, a first layer regularization LN unit, a channel attention mechanism (CA) unit, a first feature splicing unit, and a first feature channel scrambling unit connected in sequence. The second structure block includes a second feature grouping unit, a second layer regularization LN unit, a feedforward network (FFN) unit, a second feature splicing unit, and a second feature channel scrambling unit connected in sequence. The first feature grouping unit, the first LN unit, and the CA unit perform attention mechanism processing on the second feature subset to obtain the third feature subset, and then perform feature splicing and feature channel scrambling on the first feature subset and the third feature subset through the first feature splicing unit and the first feature channel scrambling unit to obtain the candidate feature set. The second feature grouping unit, the second LN unit, and the FFN unit perform feedforward network processing on the fifth feature subset to obtain the sixth feature subset. The second feature splicing unit and the second feature channel shuffling unit then perform feature splicing and feature channel shuffling on the fourth feature subset and the sixth feature subset to obtain the output feature set.

2. The method according to claim 1, characterized in that, The U-shaped network is a symmetrical U-shaped network, which contains an N-level structure; except for the Nth level, each of the other levels contains a first type of attention module and a second type of attention module; the Nth level contains only a first type of attention module. For the i-th level, 1≤i≤N-1, where N is an integer greater than or equal to 2, the output feature set of the first type of attention module in the i-th level is downsampled and used as the input feature set of the first type of attention module in the (i+1)-th level. The output feature set of the second type of attention module at level i+1 is upsampled and then fused with the output feature set of the first type of attention module at level i in the feature splicer. The fusion result serves as the input feature set of the second type of attention module at level i. This recursive process continues to level 1, and finally the output feature set of the second type of attention module is downsampled to generate the target feature set.

3. The method according to claim 1, characterized in that, The U-shaped network is an asymmetric U-shaped network, which includes N levels, each level including an attention-like module; For the i-th level, 2≤i≤N, where N is an integer greater than or equal to 2, the output feature set of the attention module of the i-th level is downsampled and used as the input feature set of the attention module of the (i+1)-th level. The output feature set of the i-th level attention module is downsampled to obtain the dimensionality-reduced output feature set of the i-th level attention module. The output feature set of the first-level attention module and the dimensionality-reduced output feature set of each of the remaining levels are concatenated to obtain a concatenated feature set. Then, the concatenated feature set is subjected to attention processing and downsampling processing by an attention module to obtain the target feature set.

4. The method according to claim 1, characterized in that, The step of grouping the input feature set by feature channels to obtain a first feature subset and a second feature subset includes: Determine the proportion of the first group of feature channel groups corresponding to the input feature set; Based on the first grouping ratio, the input feature set is grouped by feature channel to obtain a first feature subset and a second feature subset.

5. The method according to claim 1, characterized in that, The step of grouping the candidate feature set by feature channels to obtain a fourth feature subset and a fifth feature subset includes: Determine the proportion of the second group in the feature channel grouping corresponding to the candidate feature set; Based on the second grouping ratio, the candidate feature set is grouped by feature channel to obtain a fourth feature subset and a fifth feature subset.

6. The method according to claim 4 or 5, characterized in that, The method further includes: Identify the computing devices associated with downstream tasks; Determine the computing power capability information of the computing device; Based on the computing power information, determine at least one of the proportions of the first group and the proportion of the second group.

7. The method according to claim 6, characterized in that, The proportion of the first group is The proportion of the second group is ,in, ,or, ; in, The number of parameters and computational cost of the attention module mentioned above are greater than... The number of parameters and computational cost of the attention module mentioned above.

Images