Image Restoration Method, Device, Equipment and Medium under Severe Weather Conditions

Abstract

The present invention discloses an image restoration method, device, equipment and medium under adverse weather conditions, relating to the technical field of computer vision. Among them, the image restoration method under adverse weather conditions includes: obtaining an image to be restored, obtaining weather feature parameters and estimated stage features of the image to be restored, determining a predicted residual feature of the image to be restored based on the weather feature parameters and the estimated stage features, and processing the image to be restored according to the predicted residual feature to obtain a clear image corresponding to the image to be restored. In the embodiments of the present invention, by determining the weather feature parameters and estimated stage features of the image to be restored to perform image restoration on the image to be restored, while realizing the restoration of images under different adverse weather conditions using the weather feature parameters, the estimated stage features are fused to remove noise in the image to be restored, and the accuracy of image restoration is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to an image restoration method, device, equipment and medium under harsh weather conditions. Background Art

[0002] Various harsh weather conditions (such as rain, fog, snow, etc.) in the natural environment will affect the image quality. For example, rainfall will cause image blurring, reduced contrast and stripe noise interference. Fog will cause a decrease in image brightness and color distortion. Snow will cause changes in image brightness and color deviation. Under different weather conditions, the nature and severity of image quality problems vary.

[0003] Existing general weather condition-related algorithms aim to use the same deep network model to process images under various weather conditions. Such algorithms usually adopt a strategy of unifying the modeling of image features under different weather conditions. Some of these methods use multiple network modules, each module specifically dealing with one weather condition, and then integrating them in some way. For example, some models use a query mechanism for weather types to adjust the behavior of the network by learning queries for different weather types. There are also some methods that adopt strategies such as knowledge distillation and contrast learning to improve the generalization ability of the model under different weather conditions. However, there are still some problems with existing algorithms. Some methods do not deeply model the specific imaging features under different weather conditions, resulting in poor performance when dealing with certain weather conditions. In addition, some methods require additional weather type classification in the inference stage, which increases the computational cost and introduces unnecessary complexity. Therefore, how to restore images under various harsh weather conditions has become an urgent problem to be solved currently. Summary of the Invention

[0004] The present invention provides an image restoration method, device, equipment and medium under harsh weather conditions to solve the problem of poor image restoration effect under various harsh weather conditions.

[0005] According to one aspect of the present invention, there is provided an image restoration method under harsh weather conditions, wherein the method includes:

[0006] Obtain the image to be restored;

[0007] Obtain the weather feature parameters and the estimation stage features of the image to be restored;

[0008] Determine the predicted residual features of the image to be restored based on the weather feature parameters and the estimation stage features, and process the image to be restored according to the predicted residual features to obtain the clear image corresponding to the image to be restored.

[0009] According to another aspect of the present invention, there is provided an image restoration device under bad weather conditions, wherein the device includes:

[0010] An image acquisition module for acquiring an image to be restored;

[0011] A feature determination module for acquiring weather feature parameters and estimation stage features of the image to be restored;

[0012] An image restoration module for determining a prediction residual feature of the image to be restored based on the weather feature parameters and the estimation stage features, and processing the image to be restored according to the prediction residual feature to obtain a clear image corresponding to the image to be restored.

[0013] According to another aspect of the present invention, there is provided an electronic device, which includes:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image restoration method under bad weather conditions according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the image restoration method under bad weather conditions according to any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention realizes the restoration of images under different bad weather conditions by using weather feature parameters and improving the accuracy of image restoration by fusing estimation stage features through acquiring an image to be restored, acquiring weather feature parameters and estimation stage features of the image to be restored, then determining a prediction residual feature of the image to be restored based on the weather feature parameters and the estimation stage features, and processing the image to be restored according to the prediction residual feature to obtain a clear image corresponding to the image to be restored.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Description of the Drawings

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0021] Figure 1 is a flowchart of an image restoration method under bad weather conditions provided in Embodiment 1 of the present invention;

[0022] Figure 2 is an example diagram of the relationship between visibility and distance provided in Embodiment 1 of the present invention;

[0023] Figure 3 is a flowchart of an image restoration method under bad weather conditions provided in Embodiment 2 of the present invention;

[0024] Figure 4 is an example diagram of a target image restoration model provided in Embodiment 3 of the present invention;

[0025] Figure 5 is an example diagram of a weather-aware cross-attention module provided in Embodiment 3 of the present invention;

[0026] Figure 6 is an example diagram of a weather-aware cross-attention block provided in Embodiment 3 of the present invention;

[0027] Figure 7 is an example diagram of a transmission-guided global attention block provided in Embodiment 3 of the present invention;

[0028] Figure 8 is an example diagram of an occlusion-guided local attention block provided in Embodiment 3 of the present invention;

[0029] Figure 9 is an example diagram of a weather-aware fusion module provided in Embodiment 3 of the present invention;

[0030] Figure 10 is a structural schematic diagram of an image restoration device under bad weather conditions provided in Embodiment 4 of the present invention;

[0031] Figure 11 is a structural schematic diagram of an electronic device for implementing the image restoration method under bad weather conditions in the embodiments of the present invention. Detailed implementation manners

[0032] To enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solution in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 is a flowchart of an image restoration method under adverse weather conditions according to Embodiment 1 of the present invention. This embodiment is applicable to the situation of restoring images under adverse weather. This method can be executed by an image restoration device under adverse weather conditions. The image restoration device under adverse weather conditions can be implemented in the form of hardware and / or software, and the image restoration device under adverse weather conditions can be configured in an electronic device. As Figure 1 shown, the method includes:

[0036] S110. Obtain the image to be restored.

[0037] Among them, the image to be restored can be understood as the image waiting to be restored. In the actual operation process, the image to be restored may include a scene image taken under adverse weather. Exemplarily, the image to be restored may be an image with blurred, high-noise or low-contrast due to weather such as rain, fog, snow, etc.

[0038] In the embodiments of the invention, the image to be restored can be collected by an image acquisition device, or alternatively, the image to be restored can be obtained from the data source of the image to be restored.

[0039] S120. Obtain the weather feature parameters and the estimated stage features of the image to be restored.

[0040] Among them, the weather feature parameter can be understood as a parameter indicating the weather-related features of the scene in the image to be restored, and the weather condition at the time of shooting the image to be restored can be determined according to the weather feature parameter. Exemplarily, the weather feature parameter can at least include global atmospheric light, medium transmittance, occlusion transparency, and brightness.

[0041] The estimated stage feature can be understood as an image feature parameter estimated for the image to be restored. In the actual operation process, the estimated stage feature of the image to be restored can be determined by a pre-trained weather-related prior estimation network, and the estimated stage feature can be a feature map. Exemplarily, the pre-trained weather-related prior estimation network can be composed of two encoders and two decoders connected in series, and the weather-related prior estimation network can be constructed based on the U-Net network.

[0042] In the embodiment of the invention, the image to be restored can be input into the pre-trained weather-related prior estimation network, and the weather feature parameter and the estimated stage feature of the image to be restored can be determined through the weather-related prior estimation network. In the actual operation process, the global environmental information of the image to be restored is extracted by the encoding layer of the weather-related prior estimation network, and the pixel-level information of the image to be restored is extracted by the decoding layer to determine the estimated stage feature of the image to be restored. In practical applications, the global atmospheric light can be considered to contain more global environmental information, while the medium transmittance, occlusion transparency, brightness, and the estimated stage feature focus on pixel-level information. Therefore, the prediction of the global atmospheric light is achieved by pooling and estimating the last feature map of the encoder of the U-Net, while the medium transmittance, occlusion transparency, brightness, and the estimated stage feature are obtained from the decoder. In one embodiment, both the weather feature parameter and the estimated stage feature have a separate prediction structure in the form of a multi-layer perceptron (MLP).

[0043] S130. Determine the predicted residual feature of the image to be restored based on the weather feature parameter and the estimated stage feature, and process the image to be restored according to the predicted residual feature to obtain a clear image corresponding to the image to be restored.

[0044] Among them, the predicted residual feature refers to the image feature used for image restoration. The predicted residual feature can be an image feature related to the noise of the image to be restored generated by fusing the image features of each stage. According to the predicted residual feature, the noise in the image to be restored can be removed to generate a clear image. The clear image refers to the noise-free image after processing the image to be restored.

[0045] In an embodiment of the invention, the global atmospheric light, medium transmittance, occlusion transparency, and brightness in the weather feature parameters can be extracted to determine an initial restored image according to the weather feature parameters. Then, the weather feature parameters, the estimated stage features, the initial restored image, and the image to be restored are input into a pre-trained scene refinement restoration network. The predicted residual features of the image to be restored are generated according to the scene refinement restoration network, and the image to be restored is processed through the predicted residual features to obtain a clear image corresponding to the image to be restored.

[0046] In the actual application process, the weather-related prior estimation network and the scene refinement restoration network can form a target image restoration model. Inputting the image to be restored into the target image restoration model can achieve the restored clear image. Among them, the scene refinement restoration network can be built by a deep learning network, such as a U-Net network. The scene refinement restoration network is composed of four weather-aware cross-attention modules connected in series. The weather-aware cross-attention module is composed of at least two residual neural network modules and a weather-aware cross-attention block connected in series. The weather-aware cross-attention block consists of a transmission-guided global attention block, an occlusion-guided local attention block, and a weather-aware fusion module.

[0047] In the actual operation process, after inputting the weather feature parameters, the estimated stage features, the initial restored image, and the image to be restored into the pre-trained scene refinement restoration network, the first refinement stage features can be extracted through the residual neural network module in the weather-aware cross-attention module. The weather-aware cross-attention block can fuse and generate the second refinement stage features according to the first refinement stage features and the estimated stage features. Then, the first refinement stage features and the second refinement stage features are superimposed, and the predicted residual features can be determined by iteratively processing the superimposed features according to multiple weather-aware cross-attention modules. The image to be restored is restored according to the predicted residual features to obtain a clear image corresponding to the image to be restored.

[0048] In the embodiment of the present invention, by obtaining the image to be restored, obtaining the weather feature parameters and the estimated stage features of the image to be restored, and then determining the predicted residual features of the image to be restored based on the weather feature parameters and the estimated stage features, and processing the image to be restored according to the predicted residual features to obtain a clear image corresponding to the image to be restored, it is possible to restore images in different bad weather conditions using the weather feature parameters, and at the same time, fuse the estimated stage features to remove the noise in the image to be restored, improving the accuracy of image restoration under bad weather conditions.

[0049] In one embodiment, Figure 2 is an example diagram of the relationship between visibility and distance provided according to Embodiment 1 of the present invention, as Figure 2As shown, in the region 0 < z < z1, the visibility of the particles is affected by the camera exposure time and does not depend on z, where z1 = 2fa, a is the landing radius, and f is the focal length. In the region z1 < z < z2, the visibility of the particles decreases as the distance increases, proportional to 1 / z, where z2 = Rz1 and R is a constant. In the region z > z2, the appearance of a single particle is too small for the camera to capture, while a cluster of particles produces a fog-like aggregation scattering effect. From the above analysis, the visual phenomena caused by rain and snow can be classified as follows: in the region z < z2, we call it visible particle occlusion, while in the region z > z2, we call it fog-like scattering effect. In addition, in the region z < z2, visible particles at multiple depths may project onto the same pixel position on the image plane. Therefore, the scene depth of the pixel is larger, affected by more particles, and the visibility of each particle is different, which can also be called the volume effect.

[0050] In one embodiment, the method for image restoration under adverse weather conditions further includes:

[0051] Obtain the clear scene radiation, global atmospheric light, scene depth, atmospheric scattering coefficient, brightness, near transparency, and transparency corresponding to each depth of the target image;

[0052] Determine the product of the scene depth and the atmospheric scattering coefficient as the first data, and take the negative power of the first data with the natural constant as the medium transmittance;

[0053] Determine 1 minus the medium transmittance as the second data, and sum the product of the second data and the transparency corresponding to each depth as the depth transparency;

[0054] Take the sum of the near transparency and the depth transparency as the occlusion transparency;

[0055] Determine the product of the clear scene radiation and the medium transmittance as the third data, determine the product of the global atmospheric light and the second data as the fourth data, and take the sum of the third data and the fourth data as the background image affected by the aggregation scattering effect;

[0056] Determine the product of the brightness and the occlusion transparency as the fifth data, determine the difference between 1 and the occlusion transparency as the sixth data, determine the product of the background image affected by the aggregation scattering effect and the sixth data as the seventh data, and take the sum of the fifth data and the seventh data as the adverse weather image;

[0057] Form a training image pair with the target image and the corresponding adverse weather image, and train the target image restoration model according to the training image pair, where the target image restoration model is constructed by a weather-related prior estimation network and a scene refinement restoration network.

[0058] Among them, the clear-scene radiation can be understood as the radiation value corresponding to each pixel index in the target image. To capture the above volume effect, the volume containing visible particles can be divided into several slender layers, and the transparency corresponding to each depth can be understood as the transparency contribution of each layer, simulating Figure 2 the visible occlusion in the region z1 < z < z2 in Figure 2 ; the adjacent transparency is generally a fixed value, simulating the visible occlusion in the region 0 < z < z1 in

[0059] to determine the occlusion transparency together with the transparency corresponding to each depth. In one embodiment, the brightness is usually a constant value in the same target image.

[0060] In the implementation of the invention, the target image restoration model is constructed by a weather-related prior estimation network and a scene refinement restoration network, and is used to restore the image to be restored to the corresponding clear image. The image to be restored can first extract data features through the weather-related prior estimation network, and then fuse the features through the scene refinement restoration network for restoration. Training image pairs can be constructed to train the target image restoration model. Clear-scene radiation, global atmospheric light, scene depth, atmospheric scattering coefficient, brightness, adjacent transparency, and transparency corresponding to each depth can be extracted for the target image. The depth transparency can be determined first, and the occlusion transparency can be determined according to the depth transparency and the adjacent transparency. Then, for the fog-like scattering effect, the background image affected by the aggregated scattering effect can be determined, and then the bad-weather image can be determined based on the background image affected by the aggregated scattering effect and weather-related parameters such as brightness and occlusion transparency. The target image and the corresponding bad-weather image are formed into a training image pair, and the target image restoration model is trained according to the training image pair.

[0061] In one embodiment, for images captured under general adverse weather conditions, such as fog, rain, and snow, it can be regarded as a combination of foggy scattering effects and (optional) visible particle occlusion.

[0062] First, the foggy scattering effect can be considered as follows:

[0063] B(x) = J(x)t(x) + A(1 - t(x))

[0064] Where B is the background image affected by the aggregated scattering effect, J is the clear scene radiation, t is the medium transmittance, A is the global atmospheric light, and x is the pixel index. Among them, the medium transmittance t = e -βd(x) Describes the part of light scattering, where is the scene depth and β is the atmospheric scattering coefficient.

[0065] Then, consider the visible occlusion of rain and snow:

[0066] I(x) = O(x)α(x) + B(x)(1 - α(x))

[0067] Where the image observed under adverse weather conditions, that is, the adverse weather image; α and O respectively represent the occlusion transparency and brightness, B is the background image affected by the aggregated scattering effect, x is the pixel index, and the brightness is usually constant in the photo.

[0068] At the same time, consider the volume effect of visible rain and snow. To capture these volume effects, the volume containing visible particles can be divided into multiple layers, and these layers are combined, expressed as:

[0069] α(x) = α near (x) + α far (x)

[0070]

[0071] Where α near and α far respectively simulate the visible occlusion in the regions 0 < z < z1 and z1 < z < z2. N represents the number of layers, d is the scene depth, β is the atmospheric scattering coefficient, and x is the pixel index. Since there are more visible raindrops in the far - distance region, its scene depth d is larger. In one embodiment, α near can also include one or more layers.

[0072] In one embodiment, when there is no visible rain or snow blocking, i.e., I(x) = J(x)t(x) + A(1 - t(x)), where α = 0, this is the case of only fog. When the scattering effect can be ignored, i.e., I(x) = O(x)α(x) + J(x)(1 - α(x)), where t = 1, this corresponds to the case of less rain and snow. When regenerating a severe weather image, the specific physical visual process of the image captured under common severe weather conditions is considered. The effects of visible particles (such as raindrops and snowflakes) at close range and the fog-like scattering effect in the distance on the image are comprehensively considered to ensure higher restoration accuracy of the target image restoration model, and then image restoration for the mixed weather condition is realized.

[0073] Embodiment 2

[0074] Figure 3 It is a flowchart of an image restoration method under severe weather conditions provided according to Embodiment 2 of the present invention. This embodiment is further optimized and extended based on the above-mentioned implementation manner and can be combined with each optional technical solution in the above-mentioned implementation manner. As Figure 3 shown, the method includes:

[0075] S201. Obtain the image to be restored.

[0076] S202. Input the image to be restored into the weather-related prior estimation network.

[0077] Among them, the weather-related prior estimation network is built based on a deep learning network. Exemplarily, the weather-related prior estimation network is built based on U-Net. The weather-related prior estimation network is composed of two encoders and two decoders connected in series.

[0078] The weather-related prior estimation network can be understood as a network for estimating weather feature parameters and generating estimated-stage features by estimating the features of the image to be restored, and can extract parameters such as estimated-stage features, medium transmittance, occlusion transparency, and brightness in the image to be restored.

[0079] S203. Generate first data features through the encoder, pool to determine the global atmospheric light from the first data features, and input the first data features into the decoder.

[0080] Among them, the first data features refer to the feature data determined by encoding the image to be restored through the encoder, and can be the feature map corresponding to the image to be restored.

[0081] In an embodiment of the invention, after the image to be restored is input into the weather-related prior estimation network, the encoder generates the first data feature. Since the global atmospheric light contains more global environmental information, the first data feature can be pooled and estimated to obtain the global atmospheric light. In the actual operation process, the prediction of the global atmospheric light is obtained by pooling and estimating the last feature map of the encoder. After determining the global atmospheric light, the first data feature can be input into the decoder.

[0082] S204. The decoder processes the first data feature to generate the estimated-phase feature, as well as the medium transmittance, occlusion transparency, and brightness.

[0083] In an embodiment of the invention, after the first data feature is input into the decoder, the decoder processes the first data feature to obtain the estimated-phase feature. Feature estimation is performed on the image to be restored to generate the estimated-phase feature. At the same time, since the estimation of the medium transmittance and occlusion transparency focuses on pixel-level information, therefore, the medium transmittance, occlusion transparency, and brightness in the image to be restored can be obtained through the decoder.

[0084] S205. The global atmospheric light, medium transmittance, occlusion transparency, and brightness are used as weather feature parameters.

[0085] In an embodiment of the invention, when the global atmospheric light, medium transmittance, occlusion transparency, and brightness are determined, the global light, medium transmittance, occlusion transparency, and brightness can be used as weather feature parameters.

[0086] S206. The global atmospheric light, medium transmittance, occlusion transparency, and brightness in the weather feature parameters are extracted, and the initial restored image is determined according to the weather feature parameters.

[0087] In an embodiment of the invention, the global atmospheric light, medium transmittance, occlusion transparency, and brightness in the weather feature parameters can be obtained, and the initial restored image is restored based on the global atmospheric light, medium transmittance, occlusion transparency, and brightness. In the actual operation process, since the severe weather image is determined in a preset manner during the training process of the weather-related prior estimation network, correspondingly, the initial restored image can be restored in a corresponding preset manner.

[0088] In one embodiment, determining the initial restored image according to the weather feature parameters includes:

[0089] Determining the product of the occlusion transparency and brightness corresponding to each pixel index in the image to be restored as the first result;

[0090] Determining the difference between the radiation value corresponding to each pixel index of the image to be restored and the first result as the second result;

[0091] Determine the difference between 1 and the occlusion transparency corresponding to each pixel index as the third result, and take the ratio of the second result to the third result as the radiance value corresponding to the unoccluded image;

[0092] Determine the difference between 1 and the medium transmittance corresponding to each pixel index as the fourth result, and take the product of the fourth result and the global atmospheric light as the fifth result;

[0093] Determine the difference between the radiance value corresponding to the unoccluded image and the fifth result as the sixth result, take the ratio of the sixth result to the medium transmittance as the radiance value corresponding to the initial restored image, and determine the initial restored image.

[0094] In an embodiment of the invention, an image of an unoccluded background (without fog) can be determined first for the image to be restored, and then the initial restored image can be determined according to the image of the unoccluded background. In the actual operation process, the product of the occlusion transparency and the brightness corresponding to each pixel index in the image to be restored can be determined first as the first result, and then the difference between the radiance value corresponding to each pixel index in the image to be restored and the first result can be determined as the second result. At the same time, the difference between 1 and the occlusion transparency corresponding to each pixel index is determined as the third result, and the ratio of the second result to the third result is taken as the radiance value corresponding to the unoccluded image. Then, the difference between 1 and the medium transmittance corresponding to each pixel index is determined as the fourth result, and the product of the fourth result and the global atmospheric light is taken as the fifth result; the difference between the radiance value corresponding to the occluded image and the fifth result is taken as the sixth result, and then the ratio of the result to the medium transmittance is taken as the radiance value corresponding to the initial restored image, and the initial restored image is determined.

[0095] In one embodiment, the initial restored image can be determined according to the following formula:

[0096]

[0097]

[0098] Wherein, and are the unoccluded (without fog) background and the initial restored image calculated based on the estimated information respectively, t is the medium transmittance, A is the global atmospheric light, x is the pixel index, α is the occlusion transparency, and O is the brightness.

[0099] S207. Input the weather feature parameters, the estimated stage features, the initial restored image, and the image to be restored into the weather-aware cross-attention module in the scene refinement restoration network.

[0100] Among them, the scene refinement and restoration network is built based on a deep learning network. The scene refinement and restoration network is composed of four weather perception cross-attention modules connected in series. The weather perception cross-attention module is composed of at least two residual neural network modules and a weather perception cross-attention block connected in series. The weather perception cross-attention block consists of a transmission-guided global attention block, an occlusion-guided local attention block, and a weather perception fusion module.

[0101] Among them, the transmission-guided global attention block is Transmission-guided global attention (TGGA); the occlusion-guided local attention block is occlusion-guided local attention (OGLA). Since the medium transmission describes part of the fog dissipation effect and depends on the scene depth. And the medium transmission is global information, where the medium transmission within a local area is almost uniform. Due to similar depths, some non-local regions in the scene have similar medium transmissions. These regions suffer from similar visibility reduction due to the fog effect. Therefore, the transmission-guided global attention (TGGA) uses non-local features with similar transmission values in the estimation stage to enhance the refined features at a certain position. Since rain and snow present particles that occlude local regions in the input image, the textures in the affected regions are highly correlated. The occlusion-guided local attention (OGLA) can propagate the estimated features to the refined features locally by referring to less occluded regions.

[0102] In an embodiment of the invention, after determining the initial restored image, the weather feature parameters, the features in the estimation stage, the initial restored image, and the image to be restored can be input into the weather perception cross-attention module in the scene refinement and restoration network.

[0103] S208. Extract the first refinement stage features through the residual neural network module in the weather perception cross-attention module, and input the first refinement stage features and the features in the estimation stage into the weather perception cross-attention block.

[0104] Among them, the first refinement stage features refer to the refined feature data of the image to be restored extracted by the residual neural network module in the weather perception cross-attention module. In the actual operation process, the first refinement stage features can be feature maps.

[0105] In an embodiment of the invention, through the weather perception cross-attention module, the feature data of the image to be restored can be refined and extracted according to the weather feature parameters, the features in the estimation stage, and the initial restored image as the first refinement stage features, and then the first refinement stage features and the features in the estimation stage are input into the weather perception cross-attention block.

[0106] S209. Obtain the second refinement stage features through the weather perception cross-attention block.

[0107] Among them, the second refinement stage feature refers to the refined feature data obtained by further fusing the first refinement stage feature extracted by the weather perception cross-attention block in the weather perception cross-attention module and the estimation stage feature. The weather perception cross-attention block consists of a transmission-guided global attention block, an occlusion-guided local attention block, and a weather perception fusion module, which can extract corresponding first features and second features for medium transmission and occlusion transparency respectively according to the transmission-guided global attention block and the occlusion-guided local attention block, and then fuse them with both the first feature and the second feature according to the weather feature parameters to generate the second refinement stage feature.

[0108] In one embodiment, obtaining the second refinement stage feature through the weather perception cross-attention block includes:

[0109] Inputting the medium transmission, the estimation stage feature, and the first refinement stage feature into the transmission-guided global attention block to generate the first feature, and inputting the occlusion transparency, the estimation stage feature, and the first refinement stage feature into the occlusion-guided local attention block to generate the second feature;

[0110] Inputting the occlusion transparency and the medium transmission into the weather perception fusion module to generate the first weight of the first feature and the second weight of the second feature; among them, the weather perception fusion module consists of a convolutional layer and an activation function layer;

[0111] Taking the sum of the product of the first feature and the first weight and the product of the second feature and the second weight as the second refinement stage feature.

[0112] Among them, the first feature refers to the data feature extracted by the transmission-guided global attention block according to the medium transmission, the estimation stage feature, and the first refinement stage feature; the second feature refers to the data feature extracted by the occlusion-guided local attention block according to the occlusion transparency, the estimation stage feature, and the first refinement stage feature. The weather perception fusion module is to integrate the enhanced features generated by the transmission-guided global attention block and the occlusion-guided local attention block. Among them, the weather perception fusion module takes the first feature and the second feature the estimated medium transmission and the estimated occlusion transparency as inputs. Then, the connected inputs are fed into a depth convolution, 1×1 convolution, and Sigmoid activation sequence to generate the fusion attention weights α t , α o ∈[0,1] h×w×1 . Among them, α t is the first weight corresponding to the first feature, and α o is the second weight corresponding to the second feature. Finally, the aggregated feature is X r′ = α t ·X t + α o ·X o , where · represents element-wise multiplication. The second refinement stage feature refers to the data feature generated by fusing through the weather-aware cross-attention block.

[0113] In an embodiment of the invention, the medium transmittance, the estimated stage feature, and the first refinement stage feature can be input into the transmittance-guided global attention block to generate the first feature. At the same time, the occlusion transparency, the estimated stage feature, and the first refinement stage feature are input into the occlusion-guided local attention block to generate the second feature. The occlusion transparency and the medium transmittance are input into the weather-aware fusion module to generate the first weight of the first feature and the second weight of the second feature. The sum of the product of the first feature and the first weight and the product of the second feature and the second weight is used as the second refinement stage feature. By combining the transmittance-guided global attention and the occlusion-guided local attention, it helps to more effectively inject weather feature parameters during the scene refinement process and establish global and local dependencies, improving the adaptability and performance of the network.

[0114] In one embodiment, inputting the medium transmittance, the estimated stage feature, and the first refinement stage feature into the transmittance-guided global attention block to generate the first feature, and inputting the occlusion transparency, the estimated stage feature, and the first refinement stage feature into the occlusion-guided local attention block to generate the second feature includes:

[0115] Downsample the estimated stage feature and the medium transmittance respectively according to the transmittance-guided global attention block to obtain the downsampled estimated stage feature and the downsampled medium transmittance;

[0116] Determine the similarity between the medium transmittance and the downsampled medium transmittance according to the transmittance-guided global attention block;

[0117] Extract the data features of the first refinement stage feature, the estimated stage feature, and the downsampled estimated stage feature respectively according to the transmittance-guided global attention block, and determine the product of the data feature corresponding to the first refinement stage feature and the data feature corresponding to the estimated stage feature as the first product;

[0118] Normalize the superposition result of the similarity and the first product according to the transmittance-guided global attention block as the first initial feature, and use the product of the first initial feature and the data feature of the downsampled estimated stage feature as the first feature;

[0119] Perform bilinear interpolation, downsampling convolution, and window division on the occlusion transparency, the first refinement stage feature, and the estimated stage feature respectively according to the occlusion-guided local attention block to generate the window occlusion transparency, the window first refinement stage feature, and the window estimated stage feature;

[0120] Extract the data features of the first refinement stage features of the window and the first - dimensional data features and the second - dimensional data features of the window estimation stage features respectively according to the occlusion - guided local attention block;

[0121] Determine the product of the data features of the first refinement stage features and the first - dimensional data features as the second product according to the occlusion - guided local attention block;

[0122] Determine the difference between 1 and the window occlusion transparency according to the occlusion - guided local attention block, and normalize the superposition result of the difference and the second product as the second initial feature, and take the product of the second initial feature and the second - dimensional data features as the second feature.

[0123] In the invention embodiment, the estimation - stage features and the medium transmittance can be downsampled respectively according to the transmission - guided global attention block to obtain the downsampled estimation - stage features and the downsampled medium transmittance, and at the same time, the similarity between the medium transmittance and the downsampled medium transmittance is determined. Then, extract the data features of the first refinement stage features, the estimation - stage features, and the downsampled estimation - stage features respectively, determine the product of the data features corresponding to the first refinement stage features and the data features corresponding to the estimation - stage features as the first product, normalize the superposition result of the similarity and the first product as the first initial feature, and take the product of the first initial feature and the data features of the downsampled estimation - stage features as the first feature.

[0124] Perform bilinear interpolation, downsampling convolution, and window division on the occlusion transparency, the first refinement stage features, and the estimation - stage features respectively through the occlusion - guided local attention block to generate the window occlusion transparency, the window first refinement stage features, and the window estimation stage features. Extract the data features of the window first refinement stage features and the first - dimensional data features and the second - dimensional data features of the window estimation stage features respectively according to the occlusion - guided local attention block, determine the product of the data features of the first refinement stage features and the first - dimensional data features as the second product, and determine the difference between 1 and the window occlusion transparency, and normalize the superposition result of the difference and the second product as the second initial feature, and take the product of the second initial feature and the second - dimensional data features as the second feature.

[0125] S210. Superimpose the first refinement stage features and the second refinement stage features to generate initial features, and input the initial features and the weather feature parameters into the next - day weather perception cross - attention module for iterative processing to determine the prediction residual features.

[0126] Among them, the initial features refer to the feature data generated by the first weather perception cross - attention module through the fusion of the first refinement stage features and the second refinement stage features.

[0127] In the invention embodiment, the first refinement stage feature and the second refinement stage feature can be superimposed to generate an initial feature, and the results of each weather perception cross-attention module and the weather feature parameters are sequentially input into the next perception cross-attention module for iterative processing to obtain a predicted residual feature.

[0128] S211. Restore the image to be restored according to the predicted residual feature to obtain a clear image corresponding to the image to be restored.

[0129] In the embodiment of the present invention, by inputting the image to be restored into the weather-related prior estimation network, the first data feature is generated by the encoder, the global atmospheric light is determined by pooling the first data feature, and the first data feature is input into the decoder. The decoder processes the first data feature to generate an estimation stage feature, as well as medium transmittance, occlusion transparency, and brightness, so as to estimate the weather feature parameters and the estimation stage feature according to the weather-related prior estimation network. By determining the initial restored image according to the weather feature parameters, inputting the weather feature parameters, the estimation stage feature, the initial restored image, and the image to be restored into the weather perception cross-attention module in the scene refinement restoration network, the first refinement stage feature is extracted by the residual neural network module in the weather perception cross-attention module, and the first refinement stage feature and the estimation stage feature are input into the weather perception cross-attention block. The second refinement stage feature is obtained through the weather perception cross-attention block, the first refinement stage feature and the second refinement stage feature are superimposed to generate an initial feature, and the initial feature and the weather feature parameters are input into the next weather perception cross-attention module for iterative processing to determine the predicted residual feature, which realizes the combination of transmission-guided global attention and occlusion-guided local attention, helps to more effectively inject weather priors during the scene refinement process, and establishes global and local dependencies, improves the adaptability and performance of the network, restores the image to be restored according to the predicted residual feature, and obtains a clear image corresponding to the image to be restored, realizes the automatic restoration of the image to be restored, and improves the accuracy and convenience of restoring the image to be restored.

[0130] Embodiment III

[0131] This embodiment is further optimized and extended based on the above implementation manner and can be combined with each optional technical solution in the above implementation manner. Taking the generation of a bad weather image corresponding to a target image through an imaging model as an example, a further description of an image restoration method under bad weather conditions is given.

[0132] As Figure 2 shown in the example diagram of the relationship between visibility and distance, in the prior art, for different bad weathers such as fog, rain, and snow, there are existing imaging models for fog, rain, and snow respectively. The process for describing the formation of a fog image is as follows:

[0133] I(x) = J(x)t(x) + A(1 - t(x)), (1)

[0134] Wherein, I and J are respectively the bad weather image of the observed bad weather as an object and the clear scene radiation, t is the medium transmittance, A is the global atmospheric light, and x is the pixel index. The medium transmittance t = e -βd(x) Describes the part of light scattering, where d is the scene depth and β is the atmospheric scattering coefficient.

[0135] The formation of the rain image can be simplified to an additional composite model:

[0136]

[0137] Wherein, is the bad weather image caused by the noise of rain, and O(x) is the color intensity map of scattered rain streaks. In addition, O(x) can contain one or more layers of rain streaks, that is, where n is the number of layers. Considering the scattering effect, the rain model can be described as:

[0138]

[0139] Wherein, I is the bad weather image affected by the noise of rain and fog, and t and A are as shown in equation (1), is the rain image without scattering effect, as shown in equation (2).

[0140] The formation of the snow image can be described as:

[0141]

[0142] Wherein, is the bad weather image affected by snowflakes, α is the transparency of snow, and O is the color intensity of snow. Considering the scattering effect, the snow model can be expressed as:

[0143]

[0144] Wherein, I is the bad weather image affected by snowflakes and fog, and t and A are as shown in equation (1), is the snow image without scattering effect.

[0145] Generally speaking, the rain and snow imaging models have similar footprints in recent deep learning-based research, that is, only rain streaks or snowflakes in equations (2) and (4), or with scattering effects in equations (3) and (5), except for some minor deviations in details.

[0146] In the present invention, images taken under general bad weather conditions, such as fog, rain, and snow, can be regarded as a combination of foggy scattering effects and (optional) visible particle occlusions.

[0147] First, we consider the fog-like scattering effect as follows:

[0148] B(x) = J(x)t(x) + A(1 - t(x)), (6)

[0149] where B is the background image affected by the aggregation scattering effect, similar to Equation (1).

[0150] Then, we consider the visible occlusion of rain and snow:

[0151] I(x) = O(x)α(x) + B(x)(1 - α(x)), (7)

[0152] where I is the image observed under adverse weather conditions; α and O represent the occlusion transparency and brightness respectively. Among them, O is usually more constant in the photo.

[0153] Meanwhile, our method takes into account the volume effect of visible rain and snow. To capture these volume effects, we divide the volume containing visible particles into multiple layers and combine these layers, denoted as

[0154] α(x) = α near (x) + α far (x)

[0155]

[0156] where α near and α far simulate the visible occlusion in the regions 0 < z < z1 and z1 < z < z2 respectively, and N represents the number of layers. Importantly, our method makes the number of visible raindrops in the far - distance region larger, with a larger scene depth d. Meanwhile, α near can also contain one or more layers, although the visibility change with depth is not considered.

[0157] In one embodiment, when there is no visible rain or snow obstruction, i.e., I(x) = J(x)t(x) + A(1 - t(x)), where α = 0, this is the case only containing fog. When the scattering effect can be ignored, i.e., I(x) = O(x)α(x) + J(x)(1 - α(x)), where t = 1, this corresponds to the case with less rain and snow. When regenerating a bad weather image, the specific physical visual process of the images captured under common bad weather conditions is considered, and the effects of visible particles (such as raindrops and snowflakes) at close range and fog-like scattering effects in the distance on the image are comprehensively considered to ensure higher restoration accuracy of the target image restoration model, and then image restoration for mixed weather conditions is achieved. A physics-based unified imaging model is implemented, which considers the specific physical visual process of the images captured under common bad weather conditions. This model comprehensively considers the effects of visible particles (such as raindrops and snowflakes) at close range and fog-like scattering effects in the distance on the image.

[0158] In one embodiment, Figure 4 is an example diagram of a target image restoration model provided according to Embodiment 3 of the present invention. As Figure 4 shown, the target image restoration model is a two-stage framework, including a weather-related prior estimation stage and a scene refinement restoration stage based on weather prior. The target image restoration model receives an image (image to be restored) captured under any type (including mixed types) of bad weather conditions as input and generates a clean and clear image as output.

[0159] First, the image I to be restored is sent to the weather-related prior estimation stage, whose goal is to estimate weather-related prior information (weather feature parameters), including medium transmittance t, occlusion transparency α, and brightness O, as well as global atmospheric light A. Therefore, the initially restored scene is obtained by solving equations (9) and (10):

[0160]

[0161]

[0162] where and are the unoccluded (fog-free) background and the initially restored image calculated based on the estimated information, respectively. Then, the initially restored image and the image I to be restored are sent together to the scene refinement restoration stage based on weather prior to generate the final noise-free scene image. In addition, the estimated prior information in the first stage, i.e., medium transmittance t and occlusion transparency α, as well as the feature of the estimation stage, are also injected into the second stage to assist in scene restoration.

[0163] In one embodiment, Figure 5It is an example diagram of a weather-aware cross-attention module provided according to Embodiment 3 of the present invention. As Figure 5 shown, the weather-aware cross-attention module is composed of at least two residual neural network modules and weather-aware cross-attention blocks connected in series. The weather-aware cross-attention (WACA) module propagates cross-stage features through the cross-attention mechanism, enhancing the fusion of the first refinement stage feature Xr and the estimation stage feature Xe. The implementation of the two stages is based on U-Net, where the WACA block is used in the second stage, i.e., the scene refinement and restoration stage.

[0164] In one embodiment, Figure 6 It is an example diagram of a weather-aware cross-attention block provided according to Embodiment 3 of the present invention. As Figure 6 shown, the weather-aware cross-attention block consists of a transmission-guided global attention block, an occlusion-guided local attention block, and a weather-aware fusion module.

[0165] In addition, in the weather-related prior estimation stage, the global atmospheric light A can be considered to contain more global environmental information, while the medium transmission t, occlusion transparency α, luminance O, and the estimation stage feature focus on pixel-level information. Therefore, the prediction of the global atmospheric light is achieved by pooling the last feature map of the encoder of the estimation U-Net, while the medium transmission, occlusion transparency, luminance, and the estimation stage feature are obtained from the decoder. In one embodiment, there are separate MLP-style prediction structures for both the weather feature parameters and the estimation stage feature.

[0166] In one embodiment, Figure 7 It is an example diagram of a transmission-guided global attention block provided according to Embodiment 3 of the present invention. As Figure 7 shown, the medium transmission t describes the part of the fog dissipation effect and depends on the scene depth d. The medium transmission is global information, where the medium transmission t within a local region is almost uniform, and due to similar depths, some non-local regions in the scene have similar medium transmission t. These regions suffer from similar visibility reduction due to the fog effect. Therefore, the transmission-guided global attention (TGGA) uses non-local features with similar transmission values in the estimation stage to enhance the refinement feature at a certain position.

[0167] Specifically, the estimation stage feature Xe and the medium transmission t are respectively downsampled to obtain the downsampled estimation stage feature and the downsampled medium transmission t ds , and the similarity β between the medium transmission t and the downsampled medium transmission t ds is determined. t, extract the data feature Q of the first refinement stage feature, the data feature K of the estimation stage feature, and the data feature V of the downsampled estimation stage feature respectively, and determine the product of the data feature Q corresponding to the first refinement stage feature and the data feature K corresponding to the estimation stage feature as the first product QK T , the similarity β t and the first product QK T of the superposition result is normalized as the first initial feature, and the product of the first initial feature and the data feature of the downsampled estimation stage feature is used as the first feature

[0168] In one embodiment, Figure 8 is an example diagram of an occlusion-guided local attention block provided by Embodiment 3 of the present invention. As Figure 8 shown, rain and snow present particles that occlude local regions in the input image, and the textures in the affected regions are highly correlated. The occlusion-guided local attention (OGLA) propagates the estimated features to the refined features locally by referring to less occluded regions

[0169] Specifically, perform bilinear interpolation, downsampling convolution, and window partitioning on the occlusion transparency α, the first refinement stage feature Xr, and the estimation stage feature Xe according to the occlusion-guided local attention block to generate the window occlusion transparency α w , the window first refinement stage feature and the window estimation stage feature Extract the data feature q of the window first refinement stage feature, the first-dimensional data feature k and the second-dimensional data feature v of the window estimation stage feature respectively; determine the product of the data feature q of the first refinement stage feature and the first-dimensional data feature k as the second product qK T , determine the difference β between 1 and the window occlusion transparency o , and the difference β o and the second product qK T of the superposition result is normalized as the second initial feature, and the product of the second initial feature and the second-dimensional data feature is used as the second feature

[0170] In one embodiment, Figure 9 is an example diagram of a weather-aware fusion module provided by Embodiment 3 of the present invention. As Figure 9 shown, the weather-aware fusion module is to integrate the enhanced features of TGGA and OGLA using the guidance of transmission and occlusion. Among them, the weather-aware fusion module combines the first feature and the second feature The estimated medium transmittance t and the estimated occlusion transparency α are used as inputs. Then, the concatenated inputs are fed into a sequence of depth convolution, 1×1 convolution, and Sigmoid activation to generate the fused attention weight α t , α o ∈[0,1] h×w×1 . Where α t is the first weight; α o is the second weight. Finally, the aggregated feature for X r ′ = α t ·X t + α o ·X o , where · is the element-wise multiplication.

[0171] The restoration of the image to be restored under various different adverse weather conditions, including mixed weather conditions, is achieved through two stages: weather-related prior estimation and scene restoration based on weather priors.

[0172] Example 4

[0173] Figure 10 is a schematic structural diagram of an image restoration device under adverse weather conditions provided by the fourth embodiment of the present invention. As Figure 10 shown, the device includes: an image acquisition module 101, a feature determination module 102, and an image restoration module 103.

[0174] Among them, the image acquisition module 101 is used to acquire the image to be restored.

[0175] The feature determination module 102 is used to acquire the weather feature parameters of the image to be restored and the features in the estimation stage.

[0176] The image restoration module 103 is used to determine the predicted residual features of the image to be restored based on the weather feature parameters and the features in the estimation stage, and process the image to be restored according to the predicted residual features to obtain the clear image corresponding to the image to be restored.

[0177] In the embodiment of the present invention, the image acquisition module acquires the image to be restored, the feature determination module acquires the weather feature parameters of the image to be restored and the features in the estimation stage, and the image restoration module determines the predicted residual features of the image to be restored based on the weather feature parameters and the features in the estimation stage, and processes the image to be restored according to the predicted residual features to obtain the clear image corresponding to the image to be restored, so as to realize the restoration of images under different adverse weather conditions by using weather feature parameters, and at the same time, fuse the features in the estimation stage to remove the noise in the image to be restored, and improve the accuracy of image restoration under adverse weather conditions.

[0178] In one embodiment, the feature determination module 102 includes:

[0179] A first data input unit for inputting an image to be restored into a weather-related prior estimation network; wherein, the weather-related prior estimation network is built based on a deep learning network, and the weather-related prior estimation network is composed of two encoders and two decoders connected in series;

[0180] An encoding unit for generating first data features through an encoder, pooling to determine the global atmospheric light from the first data features, and inputting the first data features into the decoder;

[0181] A decoding unit for processing the first data features through a decoder to generate estimated-stage features, as well as medium transmittance, occlusion transparency, and brightness;

[0182] A parameter determination unit for using the global atmospheric light, medium transmittance, occlusion transparency, and brightness as weather feature parameters.

[0183] In one embodiment, the image restoration module 103 includes:

[0184] An image determination unit for extracting the global atmospheric light, medium transmittance, occlusion transparency, and brightness in the weather feature parameters, and determining an initial restored image according to the weather feature parameters;

[0185] A second data input unit for inputting the weather feature parameters, estimated-stage features, initial restored image, and the image to be restored into the weather-aware cross-attention module in the scene refinement restoration network; wherein, the scene refinement restoration network is built based on a deep learning network, the scene refinement restoration network is composed of four weather-aware cross-attention modules connected in series, and the weather-aware cross-attention module is composed of at least two residual neural network modules and a weather-aware cross-attention block connected in series;

[0186] A first feature extraction module for extracting first refinement-stage features through the residual neural network modules in the weather-aware cross-attention module, and inputting the first refinement-stage features and the estimated-stage features into the weather-aware cross-attention block;

[0187] A second feature extraction module for the feature extraction module to obtain second refinement-stage features through the weather-aware cross-attention block;

[0188] A feature fusion unit for superimposing the first refinement-stage features and the second refinement-stage features to generate initial features, and inputting the initial features and the weather feature parameters into the next weather-aware cross-attention module for iterative processing to determine prediction residual features;

[0189] An image restoration unit for restoring the image to be restored according to the prediction residual features to obtain a clear image corresponding to the image to be restored.

[0190] In one embodiment, the image determination unit is specifically configured to:

[0191] Determine the product of the occlusion transparency and the brightness corresponding to each pixel index in the image to be restored as the first result;

[0192] Determine the difference between the radiance value corresponding to each pixel index of the image to be restored and the first result as the second result;

[0193] Determine the difference between 1 and the occlusion transparency corresponding to each pixel index as the third result, and use the ratio of the second result to the third result as the radiance value corresponding to the unoccluded image;

[0194] Determine the difference between 1 and the medium transmittance corresponding to each pixel index as the fourth result, and use the product of the fourth result and the global atmospheric light as the fifth result;

[0195] Determine the difference between the radiance value corresponding to the unoccluded image and the fifth result as the sixth result, and use the ratio of the sixth result to the medium transmittance as the radiance value corresponding to the initial restored image, and determine the initial restored image.

[0196] In one embodiment, the weather perception cross-attention block consists of a transmission-guided global attention block, an occlusion-guided local attention block, and a weather perception fusion module; the second feature extraction module is specifically configured to:

[0197] Input the medium transmittance, the estimated stage features, and the first refinement stage features into the transmission-guided global attention block to generate the first feature, and input the occlusion transparency, the estimated stage features, and the first refinement stage features into the occlusion-guided local attention block to generate the second feature;

[0198] Input the occlusion transparency and the medium transmittance into the weather perception fusion module to generate the first weight of the first feature and the second weight of the second feature; wherein, the weather perception fusion module consists of a convolutional layer and an activation function layer;

[0199] Use the sum of the product of the first feature and the first weight and the product of the second feature and the second weight as the second refinement stage features.

[0200] In one embodiment, the second feature extraction module is further configured to:

[0201] Downsample the estimated stage features and the medium transmittance respectively according to the transmission-guided global attention block to obtain the downsampled estimated stage features and the downsampled medium transmittance;

[0202] Determine the similarity between the medium transmittance and the downsampled medium transmittance according to the transmission-guided global attention block;

[0203] Extract the data features of the first refinement stage features, the estimation stage features, and the downsampled estimation stage features respectively according to the transmission-guided global attention block, and determine the product of the data features corresponding to the first refinement stage features and the data features corresponding to the estimation stage features as the first product;

[0204] Normalize the superposition result of the similarity and the first product according to the transmission-guided global attention block as the first initial feature, and take the product of the first initial feature and the data features of the downsampled estimation stage features as the first feature;

[0205] Perform bilinear interpolation, downsampled convolution, and window division on the occlusion transparency, the first refinement stage features, and the estimation stage features respectively according to the occlusion-guided local attention block to generate window occlusion transparency, window first refinement stage features, and window estimation stage features;

[0206] Extract the data features of the window first refinement stage features and the first-dimensional data features and second-dimensional data features of the window estimation stage features respectively according to the occlusion-guided local attention block;

[0207] Determine the product of the data features of the first refinement stage features and the first-dimensional data features as the second product according to the occlusion-guided local attention block;

[0208] Determine the difference between 1 and the window occlusion transparency according to the occlusion-guided local attention block, and normalize the superposition result of the difference and the second product as the second initial feature, and take the product of the second initial feature and the second-dimensional data features as the second feature.

[0209] In one embodiment, the image restoration device further includes:

[0210] A data extraction module, configured to obtain the clear scene radiation, global atmospheric light, scene depth, atmospheric scattering coefficient, brightness, adjacent transparency, and transparency corresponding to each depth of the target image;

[0211] A medium transmission determination module, configured to determine the product of the scene depth and the atmospheric scattering coefficient as the first data, and take the negative power of the first data with the natural constant as the base as the medium transmission;

[0212] A depth transparency determination module, configured to determine 1 minus the medium transmission as the second data, and sum the product of the second data and the transparency corresponding to each depth as the depth transparency;

[0213] An occlusion transparency determination module, configured to take the sum of the adjacent transparency and the depth transparency as the occlusion transparency;

[0214] A background image determination module, configured to determine the product of the clear scene radiation and the medium transmittance as the third data, determine the product of the global atmospheric light and the second data as the fourth data, and use the sum of the third data and the fourth data as the background image affected by the aggregated scattering effect;

[0215] A weather image determination module, configured to determine the product of the brightness and the occlusion transparency as the fifth data, determine the difference between 1 and the occlusion transparency as the sixth data, determine the product of the background image affected by the aggregated scattering effect and the sixth data as the seventh data, and use the sum of the fifth data and the seventh data as the bad weather image;

[0216] A model training module, configured to form a training image pair by using a target image and the corresponding bad weather image, and train a target image restoration model according to the training image pair, wherein the target image restoration model is constructed by a weather-related prior estimation network and a scene refinement restoration network.

[0217] The image restoration device provided by the embodiment of the present invention can execute the image restoration method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0218] Embodiment 5

[0219] Figure 11 It is a schematic structural diagram of an electronic device for implementing an image restoration method under bad weather conditions according to an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0220] As Figure 11As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0221] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0222] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the image restoration method.

[0223] In some embodiments, the image restoration method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image restoration method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the image restoration method by any other appropriate means (e.g., by means of firmware).

[0224] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0225] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0226] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0227] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input received from the user can be in any form (including acoustic input, voice input, or tactile input).

[0228] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0229] The computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0230] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0231] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for image restoration under adverse weather conditions, characterized in that: include: Obtaining the image to be restored; Obtain weather characteristic parameters and estimation stage characteristics of the image to be restored; Determining a prediction residual feature of the image to be restored based on the weather characteristic parameter and the estimation stage feature, and processing the image to be restored according to the prediction residual feature to obtain a clear image corresponding to the image to be restored; The step of determining the predicted residual feature of the image to be restored based on the weather characteristic parameter and the estimated stage feature, and processing the image to be restored according to the predicted residual feature to obtain a clear image corresponding to the image to be restored includes: Extracting global atmospheric light, medium transmittance, occlusion transparency and brightness from the weather characteristic parameters, and determining an initial restored image according to the weather characteristic parameters; Input the weather characteristic parameters, the estimated stage characteristics, the initial restored image and the image to be restored into the weather-aware cross-attention module in the scene refinement and restoration network; wherein the scene refinement and restoration network is built based on a deep learning network, the scene refinement and restoration network is composed of four weather-aware cross-attention modules connected in series, and the weather-aware cross-attention module is composed of at least two residual neural network modules and a weather-aware cross-attention block connected in series; Extracting first refinement stage features through the residual neural network module in the weather-aware cross-attention module, and inputting the first refinement stage features and the estimation stage features into the weather-aware cross-attention block; Obtaining the second refinement stage features through the weather-aware cross-attention block; Superimposing the first refinement stage feature and the second refinement stage feature to generate an initial feature, and inputting the initial feature and the weather feature parameter into the next weather perception cross attention module for iterative processing to determine the prediction residual feature; The image to be restored is restored according to the prediction residual features to obtain a clear image corresponding to the image to be restored.

2. The method according to claim 1, characterized in that The step of obtaining weather characteristic parameters of the image to be restored and estimating phase characteristics includes: Inputting the image to be restored into a weather-related prior estimation network; wherein the weather-related prior estimation network is built based on a deep learning network, and the weather-related prior estimation network is composed of two encoders and two decoders connected in series; Generate a first data feature through two serially connected encoders, pool the first data feature to obtain global atmospheric light, and input the first data feature into two serially connected decoders; Processing the first data feature through two serially connected decoders to generate the estimation stage feature, as well as medium transmission, occlusion transparency and brightness; The global atmospheric light, the medium transmittance, the occlusion transparency and the brightness are used as the weather characteristic parameters.

3. The method according to claim 1, characterized in that The determining of the initial restored image according to the weather characteristic parameters comprises: Determine the product of the occlusion transparency and the brightness corresponding to each pixel index in the image to be restored as a first result; Determine a difference between the radiation value corresponding to each pixel index of the image to be restored and the first result as a second result; Determine 1 minus the difference of the occlusion transparency corresponding to each pixel index as a third result, and take the ratio of the second result to the third result as the radiation value corresponding to the unoccluded image; Determine 1 minus the difference of the medium transmittance corresponding to each pixel index as a fourth result, and multiply the fourth result by the global atmospheric light as a fifth result; The difference between the radiation value corresponding to the unobstructed image and the fifth result is determined as a sixth result, and the ratio of the sixth result to the medium transmission is used as the radiation value corresponding to the initial restored image to determine the initial restored image.

4. The method according to claim 1, characterized in that The weather-aware cross-attention block consists of a transmission-guided global attention block, an occlusion-guided local attention block, and a weather-aware fusion module; The second refinement stage features are obtained by the weather-aware cross-attention block, including: Input the medium transmittance, the estimation stage feature and the first refinement stage feature into the transmission-guided global attention block to generate a first feature, and input the occlusion transparency, the estimation stage feature and the first refinement stage feature into the occlusion-guided local attention block to generate a second feature; Inputting the occlusion transparency and the medium transmittance into the weather-aware fusion module to generate a first weight of the first feature and a second weight of the second feature; wherein the weather-aware fusion module is composed of a convolution layer and an activation function layer; The sum of the product of the first feature and the first weight and the product of the second feature and the second weight is used as the second refinement stage feature.

5. The method according to claim 4, characterized in that The step of inputting the medium transmittance, the estimation stage feature, and the first refinement stage feature into the transmission-guided global attention block to generate a first feature, and inputting the occlusion transparency, the estimation stage feature, and the first refinement stage feature into the occlusion-guided local attention block to generate a second feature comprises: Downsampling the estimation stage features and the medium transmission respectively according to the transmission-guided global attention block to obtain downsampled estimation stage features and downsampled medium transmission; determining a similarity between the medium transmission and the downsampled medium transmission according to the transmission-guided global attention block; Extracting data features of the first refinement stage feature, the estimation stage feature, and the downsampled estimation stage feature respectively according to the transmission-guided global attention block, and determining a product of the data feature corresponding to the first refinement stage feature and the data feature corresponding to the estimation stage feature as a first product; Normalizing the superposition result of the similarity and the first product according to the transmission-guided global attention block as a first initial feature, and taking the product of the first initial feature and the data feature of the downsampling estimation stage feature as a first feature; According to the occlusion-guided local attention block, bilinear interpolation, downsampling convolution and window partitioning are performed on the occlusion transparency, the first refinement stage feature and the estimation stage feature to generate window occlusion transparency, window first refinement stage feature and window estimation stage feature; Extracting data features of the first refinement stage features of the window and first dimensional data features and second dimensional data features of the window estimation stage features respectively according to the local attention block guided by the occlusion; Determine, according to the occlusion-guided local attention block, a product of the data feature of the first refinement stage feature and the first dimensional data feature as a second product; According to the occlusion-guided local attention block, the difference of 1 minus the window occlusion transparency is determined, and the superposition result of the difference and the second product is normalized as the second initial feature, and the product of the second initial feature and the second dimensional data feature is used as the second feature.

6. The method according to claim 1, characterized in that The method further comprises: Obtain the clear scene radiation, global atmospheric light, scene depth, atmospheric scattering coefficient, brightness, adjacent transparency and transparency corresponding to each depth of the target image; Determine the product of the scene depth and the atmospheric scattering coefficient as first data, and use the negative power of the first data with the natural constant as the base as medium transmission; Determine 1 minus the medium transmittance as second data, and sum the products of the second data and the transparency corresponding to each depth as depth transparency; The sum of the proximity transparency and the depth transparency is taken as the occlusion transparency; Determine the product of the clear scene radiation and the medium transmittance as third data, determine the product of the global atmospheric light and the second data as fourth data, and use the sum of the third data and the fourth data as a background image affected by the aggregate scattering effect; Determine the product of the brightness and the occlusion transparency as fifth data, determine the difference of 1 minus the occlusion transparency as sixth data, determine the product of the background image affected by the aggregate scattering effect and the sixth data as seventh data, and use the sum of the fifth data and the seventh data as a severe weather image; The target image and the corresponding severe weather image are combined into a training image pair, and the target image restoration model is trained according to the training image pair, wherein the target image restoration model is constructed by a weather-related prior estimation network and a scene refinement restoration network.

7. An image restoration device under adverse weather conditions, characterized in that: include: An image acquisition module, used for acquiring the image to be restored; A feature determination module is used to obtain weather feature parameters and estimation stage features of the image to be restored; An image restoration module, used to determine the prediction residual features of the image to be restored based on the weather characteristic parameters and the estimation stage features, and process the image to be restored according to the prediction residual features to obtain a clear image corresponding to the image to be restored; Wherein, the image restoration module comprises: An image determination unit, used to extract global atmospheric light, medium transmission, occlusion transparency and brightness from the weather characteristic parameters, and determine an initial restored image according to the weather characteristic parameters; A second data input unit is used to input the weather characteristic parameters, the estimation stage characteristics, the initial restored image and the image to be restored into a weather-aware cross-attention module in a scene refinement and restoration network; wherein the scene refinement and restoration network is built based on a deep learning network, the scene refinement and restoration network is composed of four weather-aware cross-attention modules connected in series, and the weather-aware cross-attention module is composed of at least two residual neural network modules and a weather-aware cross-attention block connected in series; A first feature extraction module, configured to extract first refinement stage features through a residual neural network module in a weather-aware cross-attention module, and input the first refinement stage features and the estimation stage features into the weather-aware cross-attention block; A second feature extraction module, configured to obtain second refinement stage features through the weather-aware cross-attention block; A feature fusion unit, used for superimposing the first refinement stage feature with the second refinement stage feature to generate an initial feature, and inputting the initial feature and the weather feature parameter into a next weather perception cross attention module for iterative processing to determine a prediction residual feature; The image restoration unit is used to restore the image to be restored according to the prediction residual feature to obtain a clear image corresponding to the image to be restored.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the image restoration method under severe weather conditions as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image restoration method under severe weather conditions according to any one of claims 1 to 6 when executed.

Images