Abnormal weather image restoration method and device
Through the deep perception model and transfer learning network combined with pyramid feature extraction and SE attention mechanism network, the problem of poor image recovery performance in the prior art is solved, and the image recovery effect with higher quality and real-time performance is achieved.
Patent Information
- Application Number
- CN202411107940.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-15
AI Technical Summary
The existing abnormal weather image recovery methods have poor performance, poor information recovery effect and poor real-time performance in practical applications.
By building a deep perception model, transfer learning network and image recovery network, real image collections are obtained for abnormal weather synthesis, feature transfer and training are performed, and combined with pyramid feature extraction network and SE attention mechanism network, image removal and recovery are achieved.
It improves the quality of image restoration, balances the acquisition of global information and local detailed information, simplifies the network structure, and improves the real-time and effect of image restoration.
Smart Images

Figure CN118967489A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a method and device for restoring an abnormal weather image. Background Art
[0002] In recent years, computer vision has been widely used in the field of transportation. However, images or videos taken in bad weather are easily disturbed by scattering caused by rain, snow, and fog. When it rains or snows, raindrops or snow particles of different sizes and shapes often adhere to glass windows or camera lenses, resulting in occlusion and blurring of images collected by vehicle-mounted or outdoor smart devices. This not only reduces the quality of the collected images, but also interferes with subsequent computer vision tasks, such as road detection and vehicle tracking. Therefore, in bad weather, reducing the impact of weather and improving image quality are of great research significance for improving the practicality and stability of computer vision systems.
[0003] Improving hardware conditions can improve the quality of acquired images, but it still cannot solve the blur and occlusion problems caused by abnormal climate. With the rapid development of software technology, researchers have proposed different methods to obtain clear images for different degradation factors, among which the most widely used is the method based on deep learning. This type of method restores a clear image by learning the restoration rules and feature representation of the image. Compared with improving hardware technology, the method of improving image quality through software technology is more economical, convenient and fast, and has been more widely used.
[0004] At present, neural networks built using deep learning have become the mainstream method in the field of abnormal weather image restoration. For example, in the defogging algorithm, Chen et al. proposed a gated context aggregation network to directly learn the residual relationship between the fog image and the clear image, and the defogging effect is good. In the rain and snow removal algorithm, Yang et al. first proposed a rain synthesis model, which aims to more effectively deal with the problems of heavy rain and rain accumulation. On this basis, they constructed a multi-task convolutional network JORDER, which uses paired rain streak labels and clean rain-free images to learn the nonlinear mapping from rain to no rain, thereby realizing the detection and removal of rain.
[0005] Great progress has been made in the field of image restoration under abnormal weather conditions. However, both model-driven and data-driven methods are prone to problems such as overly smooth background or residual rain streaks, which means that abnormal weather image restoration still has problems such as poor practical application performance, poor overall information recovery effect, and poor real-time performance. Summary of the invention
[0006] In view of this, the purpose of the present invention is to provide an abnormal weather image restoration method and device to solve the problems of poor practical application performance, poor overall information recovery effect and poor real-time performance of the existing abnormal weather image restoration methods.
[0007] The present invention provides an abnormal weather image restoration method, comprising the steps of:
[0008] S1: Obtain a set of real images, build a depth perception model, and perform abnormal weather synthesis on the real image set through the depth perception model to obtain a set of synthetic images;
[0009] S2: Construct a transfer learning network, and perform feature transfer on the synthetic image set through the transfer learning network to obtain a training image set;
[0010] S3: construct an image restoration network, train the image restoration network through a training image set, and obtain a trained image restoration network;
[0011] S4: The abnormal weather image to be restored is removed through the trained image restoration network to obtain the restored real image.
[0012] Preferably, step S2 specifically comprises:
[0013] S21: Obtain a depth image of the synthesized image, the depth image passes through the first Conv module, the second Conv module and the third Conv module in sequence, and outputs a first feature φ;
[0014] S22: The feature φ passes through the fourth Conv module and the fifth Conv module in sequence, and outputs the feature γ; the feature φ passes through the sixth Conv module and the seventh Conv module in sequence, and outputs the second feature β;
[0015] S23: Input migration feature F, and obtain training image s through γ, β and F calculation c , the calculation formula is:
[0016] s c =SFT(F|γ,β)=γ⊙F+γ⊙β
[0017] Among them, ⊙ represents the multiplication of elements;
[0018] S24: Repeat steps S21 to S23 to obtain all training images to form a training image set.
[0019] Preferred:
[0020] The image restoration network consists of a patch embedding module, a pyramid feature extraction network, a backbone network, and a SE attention mechanism network connected sequentially;
[0021] The patch embedding module is connected to the pyramid feature extraction network and the SE attention mechanism network, the pyramid feature extraction network is connected to the backbone network, and the backbone network is connected to the SE attention mechanism network.
[0022] Preferred:
[0023] The pyramid feature extraction network includes: Pooling module, eighth Conv module, ninth Conv module, tenth Conv module, upsampling module, Concat module and Conv5 module;
[0024] The patch embedding module is connected to the Pooling module and the Concat module, the Pooling module is connected to the eighth Conv module, the ninth Conv module and the tenth Conv module, the eighth Conv module, the ninth Conv module and the tenth Conv module are connected to the upsampling module, the upsampling module is connected to the Concat module, and the Concat module is connected to the Conv5 module.
[0025] Preferred:
[0026] The backbone network includes N RTB modules connected in sequence, and each RTB module has the same structure;
[0027] The nth RTB module includes: a first STB module, a second STB module, a third Add module and an eleventh Conv module;
[0028] The first STB module and the second STB module have the same structure;
[0029] The first STB module includes a first Layer Norm module, a WMSA module, a first Add module, a second Layer Norm module, an MLP module and a second Add module connected in sequence;
[0030] The pyramid feature extraction network is connected to the first RTB module;
[0031] The n-1th RTB module is connected to the first Layer Norm module, the first Add module and the third Add module of the nth RTB module, the first Add module is connected to the second Add module, the second Add module is connected to the second STB module, the second STB module is connected to the third Add module, the third Add module is connected to the first Layer Norm module and the eleventh Conv module, and the eleventh Conv module is connected to the n+1th RTB module;
[0032] The Nth RTB module is connected with the SE attention mechanism network.
[0033] Preferably, step S3 specifically comprises:
[0034] S31: training image s c Input the image restoration network, extract features of the training image through the patch embedding module, and obtain the initial feature R embed , the calculation formula is:
[0035] R embed =f1(R input )=PatchEmbed(Conv(R input ))+Conv(R input )
[0036] Among them, R input For training images, PatchEmbed() means processed by the patch embedding module, and Conv() means processed by the convolutional layer;
[0037] S32: R through pyramid feature extraction network and backbone network embed Perform deep feature extraction to obtain deep feature R deep , the calculation formula is:
[0038] R deep =f2(R embed )=RTB1…RTB N (Conv(R embed ))+R embed
[0039] Among them, RTB N () indicates that it has been processed by the Nth RTB module;
[0040] S33: R deep Input the SE attention mechanism network to obtain the weight image u c , the weight image u c With training images s c Multiply to get the final image
[0041] S34: repeating steps S31 to S33 for a preset number of times, and calculating the average loss of the image restoration network;
[0042] S35: Select a weight subset, and prune the image restoration network through the weight subset;
[0043] S36: If the average loss is less than the preset value, a trained image restoration network is obtained; otherwise, return to step S31.
[0044] A storage medium, wherein the storage medium stores instructions and data for implementing the abnormal weather image restoration method.
[0045] An abnormal weather image restoration device comprises: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the abnormal weather image restoration method.
[0046] The present invention has the following beneficial effects:
[0047] The feature transfer of the synthetic image set is performed through the transfer learning network. Through transfer learning, the domain shift problem is taken into account, and the difference between the synthetic data set and the real data set is further reduced. By introducing the pyramid feature extraction network in the image restoration network, the network can extract abnormal weather features at different scales, not limited to local information, enrich the structure and texture information of the features, balance the acquisition of global information and local detail information, and improve the quality of the restored image. The image restoration network is lightweighted by using the network pruning method, which simplifies the network structure and network parameters. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;
[0049] Figure 2 This is the structure diagram of the transfer learning network;
[0050] Figure 3 Recover the network structure diagram for the image;
[0051] Figure 4 This is the network structure diagram of the SE attention mechanism;
[0052] Figure 5 This is a system structure diagram of an embodiment of the present invention;
[0053] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0054] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0055] Reference Figure 1 The present invention provides an abnormal weather image restoration method, comprising the steps of:
[0056] S1: Obtain a set of real images, build a depth perception model, and perform abnormal weather synthesis on the real image set through the depth perception model to obtain a set of synthetic images;
[0057] Further, step S1 includes the following steps:
[0058] Step S11, select a depth perception model, and further associate the comprehensive model with the scene depth d to create a depth perception model. The comprehensive model is used to simulate abnormal weather phenomena in the video, including degradation factors such as rain streaks, snowflakes falling, and fog gathering, and adds the time characteristics of abnormal weather scenes. This visible intensity change along the time dimension is a particle accumulation flow. The comprehensive model and the depth perception model are expressed as follows:
[0059]
[0060] Among them, S t , α are related to the scene depth, as shown below:
[0061] S t (i,j)=S pattern (i,j)·t r (i,j)
[0062]
[0063] A(i,j)=1-e -βd(i,j)
[0064] Among them, S pattern (i, j) represents the intensity map of uniformly distributed particles in the spatial image; t r (i, j) represents the depth-dependent particle intensity map; d(i, j) represents the depth; α represents the fringe intensity; β represents the particle concentration, and the larger β is, the greater the particle concentration is. Represents a point-wise multiplication operation.
[0065] S2: Construct a transfer learning network, and perform feature transfer on the synthetic image set through the transfer learning network to obtain a training image set;
[0066] Furthermore, the structure of the transfer learning network is as follows Figure 2 As shown in the figure, the feature map of the synthetic dataset is taken as input, and the depth information of the real dataset is merged into the network. First, three convolutional layers are applied to extract the conditional mapping φ from the depth map, and then φ is mapped to another two convolutional layers to predict the modulation parameters γ and β, respectively. Finally, the spatial feature transformation is used to output the input synthetic data features to obtain the displaced features.
[0067] Step S2 is specifically as follows:
[0068] S21: Obtain a depth image of the synthesized image, the depth image passes through the first Conv module, the second Conv module and the third Conv module in sequence, and outputs a first feature φ;
[0069] S22: The feature φ passes through the fourth Conv module and the fifth Conv module in sequence, and outputs the feature γ; the feature φ passes through the sixth Conv module and the seventh Conv module in sequence, and outputs the second feature β;
[0070] S23: Input migration feature F, and obtain training image s through γ, β and F calculation c , the calculation formula is:
[0071] s c =SFT(F|γ,β)=γ⊙F+γ⊙β
[0072] Among them, ⊙ represents the multiplication of elements;
[0073] Specifically, in the transfer learning network, the present invention is guided by the depth map and uses SFT to perform feature transformation, and the synthesized image after transfer is relatively closer to the real image;
[0074] S24: Repeat steps S21 to S23 to obtain all training images to form a training image set.
[0075] S3: construct an image restoration network, train the image restoration network through a training image set, and obtain a trained image restoration network;
[0076] Furthermore, the structure of the image restoration network is as follows Figure 3 As shown;
[0077] The image restoration network consists of a patch embedding module, a pyramid feature extraction network, a backbone network, and a SE attention mechanism network connected sequentially;
[0078] The patch embedding module is connected to the pyramid feature extraction network and the SE attention mechanism network, the pyramid feature extraction network is connected to the backbone network, and the backbone network is connected to the SE attention mechanism network.
[0079] Furthermore, the pyramid feature extraction network includes: a Pooling module, an eighth Conv module, a ninth Conv module, a tenth Conv module, an upsampling module, a Concat module and a Conv5 module;
[0080] The patch embedding module is connected to the Pooling module and the Concat module, the Pooling module is connected to the eighth Conv module, the ninth Conv module and the tenth Conv module, the eighth Conv module, the ninth Conv module and the tenth Conv module are connected to the upsampling module, the upsampling module is connected to the Concat module, and the Concat module is connected to the Conv5 module.
[0081] Further, the backbone network includes N RTB modules connected in sequence, and each RTB module has the same structure;
[0082] The nth RTB module includes: a first STB module, a second STB module, a third Add module and an eleventh Conv module;
[0083] The first STB module and the second STB module have the same structure;
[0084] The first STB module includes a first Layer Norm module, a WMSA module, a first Add module, a second Layer Norm module, an MLP module and a second Add module connected in sequence;
[0085] The pyramid feature extraction network is connected to the first RTB module;
[0086] The n-1th RTB module is connected to the first Layer Norm module, the first Add module and the third Add module of the nth RTB module, the first Add module is connected to the second Add module, the second Add module is connected to the second STB module, the second STB module is connected to the third Add module, the third Add module is connected to the first Layer Norm module and the eleventh Conv module, and the eleventh Conv module is connected to the n+1th RTB module;
[0087] The Nth RTB module is connected with the SE attention mechanism network.
[0088] Furthermore, step S3 is specifically as follows:
[0089] S31: training image s c Input the image restoration network, extract features of the training image through the patch embedding module, and obtain the initial feature R embed , the calculation formula is:
[0090] R embed =f1(R input )=PatchEmbed(Conv(R input ))+Conv(R input )
[0091] Among them, R input For training images, PatchEmbed() means processed by the patch embedding module, and Conv() means processed by the convolutional layer;
[0092] Specifically, the patch embedding module is Figure 3 In the Patch Embedding module, the patch embedding module first performs convolution operation to extract features, and then converts them into patch embedding form to perform local feature extraction and input construction;
[0093] S32: R through pyramid feature extraction network and backbone network embed Perform deep feature extraction to obtain deep feature R deep , the calculation formula is:
[0094] R deep =f2(R embed )=RTB1…RTB N (Conv(R embed ))+R embed
[0095] Among them, RTB N () indicates that it has been processed by the Nth RTB module;
[0096] Specifically, the number of layers of the Transformer-based recursive network is shallow, and the corresponding receptive field is small, which makes the network's ability to obtain the global context information of the image weak, resulting in the restored image only focusing on local details and ignoring the recovery of the overall information. A pyramid feature extraction network is embedded in the original feature extraction network, and three pyramids of different scales are used for feature fusion. First, the feature map extracted by the network is pooled at three scales to obtain feature pyramids of 1×1, 2×2, and 4×4. A 1×1 depth convolution is used for each layer of the pooled feature map to obtain a feature map with 1 channel; secondly, the feature map of each layer is upsampled by bilinear interpolation filling to restore the feature map to the size of the original feature map; then, the generated feature map is spliced with the original feature map in the channel dimension to obtain a composite feature map with double the number of channels. Finally, a 3×3 convolution kernel is used to convolve the composite feature map, and the output result of the convolution layer is the value to be estimated.
[0097] A network consisting of multiple recursive transformer blocks (RTBs) performs deep feature extraction. There is no parameter sharing between each RTB, and because it is based on Transformer, the output dimension is equal to the input. Finally, a residual connection is used to add the input of the deep feature extraction stage to its output;
[0098] Each recursive block RTB uses recursive and residual connections to stack STBs (Swin Transformer Blocks). A single STB first passes the input through the layer norm (LN) and then through the window multi-head self-attention (WMSA) mechanism. The output of the self-attention mechanism is combined with the initial input to form the first residual connection. The output then passes through a multi-layer perceptron (MLP) and then another LN. The final output is combined again with the output of the self-attention mechanism to form the second residual connection. STB can be described as:
[0099]
[0100] MultiHead(Q,K,V)=Concat(h1…h n )W O
[0101]
[0102] Where W is the weight, the inputs Q, K, and V are often referred to as query, key, and value, and K refers to the dimension. Each STB in the residual connection can have different parameters, but the recursive call always calls the same STB;
[0103] S33: R deep Input the SE attention mechanism network to obtain the weight image u c , the weight image u c With training images s c Multiply to get the final image
[0104] Specifically, the structure of the SE attention mechanism network is as follows Figure 4 As shown;
[0105] Step S331: The Squeeze operation compresses the feature value of each channel in the input feature map into a 1×1×C global descriptor, which reflects the global perceptual information of each channel, such as the average value and the maximum value. This process is expressed as:
[0106]
[0107] Step S332: This global descriptor is converted into a vector through the excitation operation. The size of the vector is the same as the number of channels of the input feature map. The function is to assign corresponding weights to each channel of the input feature map according to the importance of different channels. This process is expressed by the formula:
[0108] s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))
[0109] Step S333: The weights are weighted and multiplied with the original feature map channels to obtain the final feature map. In order to cover more features, the SE attention mechanism network is added to the feature extraction network. After the network fully extracts the features, it can obtain better enhanced feature expression with only a small amount of computational cost, helping the model to better focus on important information related to image restoration and improve the effect of image restoration. It can be expressed as:
[0110]
[0111] S34: repeating steps S31 to S33 for a preset number of times, and calculating the average loss of the image restoration network;
[0112] S35: Select a weight subset, and prune the image restoration network through the weight subset;
[0113] Specifically, the current sparsity rate is obtained according to the number of training rounds, that is, dynamic weight thresholding, gradually increasing the ratio of weights set to zero during training, and balancing the zeroed weights among layers;
[0114] In each pruning cycle, the parameters are sorted according to their weight magnitudes, and the weight parameters are reset to zero according to the sparsity percentage. Secondly, the pruned model is re-trained using the training set, and the back-propagation gradient is calculated through generalized direct estimation to give the zeroed weights a chance to regress.
[0115] Determine whether the current sparsity rate meets the set requirements. If not, repeat the training steps. Otherwise, end the pruning and save the currently obtained sparse weight parameters.
[0116] S36: If the average loss is less than the preset value, a trained image restoration network is obtained; otherwise, return to step S31.
[0117] Furthermore, in model training, evaluation indicators (also called performance indicators or metrics) play a vital role, which helps us measure and understand the performance and effect of the model during the training process. The present invention adopts PSNR and SSIM as evaluation indicators.
[0118] PSNR (Peak Signal-to-Noise Ratio) is a common indicator for evaluating the quality of digital images. It is usually used to compare the difference between the original image and the compressed or reconstructed image. It is calculated by comparing the difference between the original pixel value of the image and the processed pixel value. The calculation formula is as follows:
[0119]
[0120] in Indicates the maximum possible pixel value, with a dynamic range of 2 b -1. MSE stands for mean square error. When the difference between two images is smaller, it means the noise is smaller, the MSE is smaller, the PSNR is larger, and the image quality is better.
[0121] SSIM (Structural Similarity Index) is also an indicator used to evaluate image quality. SSIM does not directly calculate the average and variance of the entire image, but uses a Gaussian kernel of size 11×11, standard deviation 1.5, and sum 1 as a sliding window to calculate the local feature, and finally calculates the average value of the local feature SSIM MSSIM. The Gaussian kernel can bring a certain degree of blur, which helps to capture high-level features so that it does not just stay at the pixel scale. Therefore, compared with the traditional PSNR, it is more in line with the perception of the human visual system. SSIM not only takes into account the brightness information of the image, but also the structure and contrast information of the image, so it can better reflect the real perception of the image in the human eye. The calculation formula is as follows:
[0122]
[0123] S4: The abnormal weather image to be restored is removed through the trained image restoration network to obtain the restored real image.
[0124] Specifically, the image restoration stage processes the deep features. The height and width of the input image are first reshaped through the convolutional layer to process the features. Finally, another residual connection is used between the input and output of the network so that the features extracted by the network can be removed from the input. The whole can be expressed as:
[0125] f3(R deep )=Conv(PatchUnembed(R deep ))+R input
[0126] See also Figure 5 , Figure 5 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: an abnormal weather image recovery device 401, a processor 402 and a storage medium 403.
[0127] An abnormal weather image restoration device 401: The abnormal weather image restoration device 401 implements the abnormal weather image restoration method.
[0128] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the abnormal weather image restoration method.
[0129] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the abnormal weather image restoration method.
[0130] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0131] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.
[0132] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for restoring abnormal weather images, characterized in that: Includes steps: S1: Obtain a set of real images, build a depth perception model, and perform abnormal weather synthesis on the real image set through the depth perception model to obtain a set of synthetic images; S2: Construct a transfer learning network, and perform feature transfer on the synthetic image set through the transfer learning network to obtain a training image set; S3: construct an image restoration network, train the image restoration network through a training image set, and obtain a trained image restoration network; S4: The abnormal weather image to be restored is removed through the trained image restoration network to obtain the restored real image.
2. The abnormal weather image restoration method according to claim 1, characterized in that: Step S2 is specifically as follows: S21: Obtain a depth image of the synthesized image, the depth image passes through the first Conv module, the second Conv module and the third Conv module in sequence, and outputs a first feature φ; S22: The feature φ passes through the fourth Conv module and the fifth Conv module in sequence, and outputs the feature γ; the feature φ passes through the sixth Conv module and the seventh Conv module in sequence, and outputs the second feature β; S23: Input migration feature F, and obtain training image s through γ, β and F calculation c , the calculation formula is: s c =SFT(F|γ,β)=γ⊙F+γ⊙β Among them, ⊙ represents the multiplication of elements; S24: Repeat steps S21 to S23 to obtain all training images to form a training image set.
3. The abnormal weather image restoration method according to claim 1, characterized in that: The image restoration network consists of a patch embedding module, a pyramid feature extraction network, a backbone network, and a SE attention mechanism network connected sequentially; The patch embedding module is connected to the pyramid feature extraction network and the SE attention mechanism network, the pyramid feature extraction network is connected to the backbone network, and the backbone network is connected to the SE attention mechanism network.
4. The abnormal weather image restoration method according to claim 3 is characterized in that: The pyramid feature extraction network includes: Pooling module, eighth Conv module, ninth Conv module, tenth Conv module, upsampling module, Concat module and Conv5 module; The patch embedding module is connected to the Pooling module and the Concat module, the Pooling module is connected to the eighth Conv module, the ninth Conv module and the tenth Conv module, the eighth Conv module, the ninth Conv module and the tenth Conv module are connected to the upsampling module, the upsampling module is connected to the Concat module, and the Concat module is connected to the Conv5 module.
5. The abnormal weather image restoration method according to claim 3 is characterized in that: The backbone network includes N RTB modules connected in sequence, and each RTB module has the same structure; The nth RTB module includes: a first STB module, a second STB module, a third Add module and an eleventh Conv module; The first STB module and the second STB module have the same structure; The first STB module includes a first Layer Norm module, a WMSA module, a first Add module, a second Layer Norm module, an MLP module and a second Add module connected in sequence; The pyramid feature extraction network is connected to the first RTB module; The n-1th RTB module is connected to the first Layer Norm module, the first Add module and the third Add module of the nth RTB module, the first Add module is connected to the second Add module, the second Add module is connected to the second STB module, the second STB module is connected to the third Add module, the third Add module is connected to the first Layer Norm module and the eleventh Conv module, and the eleventh Conv module is connected to the n+1th RTB module; The Nth RTB module is connected with the SE attention mechanism network.
6. The abnormal weather image restoration method according to claim 3, characterized in that: Step S3 is specifically as follows: S31: training image s c Input the image restoration network, extract features of the training image through the patch embedding module, and obtain the initial feature R embed , the calculation formula is: R embed =f1(R input )=PatchEmbed(Conv(R input ))+Conv(R input ) Among them, R input For training images, PatchEmbed() means processed by the patch embedding module, and Conv() means processed by the convolutional layer; S32: R through pyramid feature extraction network and backbone network embed Perform deep feature extraction to obtain deep feature R deep , the calculation formula is: R deep =f2(R embed )=RTB1…RTB N (Conv(R embed ))+R embed Among them, RTB N () indicates that it has been processed by the Nth RTB module; S33: R deep Input the SE attention mechanism network to obtain the weight image u c , the weight image u c With training images s c Multiply to get the final image S34: repeating steps S31 to S33 for a preset number of times, and calculating the average loss of the image restoration network; S35: Select a weight subset, and prune the image restoration network through the weight subset; S36: If the average loss is less than the preset value, a trained image restoration network is obtained; otherwise, return to step S31.
7. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the abnormal weather image restoration method described in any one of claims 1 to 6.
8. An abnormal weather image restoration device, characterized in that: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement the abnormal weather image restoration method described in any one of claims 1 to 6.
Citation Information
Patent Citations
Image defogging method based on image depth information guidance for transfer learning
CN111489301A
Neural network image defogging method based on pyramid channel feature attention
CN111539888A
Sand and dust image restoration method based on visual attention mechanism
CN117689548A
Image-processing apparatus for removing haze contained in video, and method therefof
US20150071563A1
