Image processing method, device, electronic device and storage medium

Abstract

The present disclosure provides an image processing method, apparatus, electronic device and storage medium, the method comprising: acquiring an image to be processed; the image to be processed comprising a blurred image to be processed and / or a clear image to be processed; determining a degradation feature map corresponding to the image to be processed, wherein the degradation feature map is used to characterize an image degradation pattern of spatial feature changes between the blurred image and a clear image corresponding to the blurred image; generating a target image based on the image to be processed and the degradation feature map; wherein, in a case where the image to be processed comprises a blurred image to be processed, the target image comprises a deblurred image corresponding to the blurred image to be processed; and in a case where the image to be processed comprises a clear image to be processed, the target image comprises a blurred image corresponding to the clear image to be processed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] During the process of image formation, recording, processing, and transmission, the image quality may deteriorate due to imperfections in the imaging system, recording equipment, transmission media, and processing methods. This phenomenon is called image degradation; for example, motion degradation, blur degradation, and nonlinear degradation.

[0003] Generally, degraded images are blurry or unclear, so deblurring can be performed. Alternatively, a clear image can be degraded, for example by blurring the background to highlight the foreground. Therefore, a method for image processing is needed. Summary of the Invention

[0004] In view of this, the present disclosure at least provides an image processing method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, the present disclosure provides an image processing method, comprising:

[0006] Acquire an image to be processed; the image to be processed includes a blurred image to be processed and / or a clear image to be processed;

[0007] Determining a degradation feature map corresponding to the image to be processed, wherein the degradation feature map is used to characterize an image degradation pattern of spatial feature changes between a blurred image and a clear image corresponding to the blurred image;

[0008] A target image is generated based on the image to be processed and the degradation feature map; wherein, when the image to be processed includes a blurred image to be processed, the target image includes a deblurred image corresponding to the blurred image to be processed; and when the image to be processed includes a clear image to be processed, the target image includes a blurred image corresponding to the clear image to be processed.

[0009] In the above method, for the acquired image to be processed, a degradation feature map corresponding to the feature image to be processed can be first determined. Since the degradation feature map can be used to characterize the image degradation pattern of the spatial feature changes between the blurred image and the clear image corresponding to the blurred image, the image quality of the target image can be improved when the target image is generated based on the image to be processed and the degradation feature map corresponding to the image to be processed. For example, when the image to be processed includes a blurred image to be processed, the method can more accurately restore the deblurred image corresponding to the blurred image to be processed by utilizing the determined degradation feature map, resulting in a better deblurring effect. When the image to be processed includes a clear image to be processed, by superimposing the degradation feature map onto the clear image to be processed, the resulting blurred image is more natural, and the display effect of the blurred image is better.

[0010] In a possible implementation, determining the degradation feature map corresponding to the image to be processed includes:

[0011] In a case where the image to be processed includes the blurred image to be processed but does not include the clear image to be processed, performing feature coding processing on the image to be processed to obtain a degradation feature map corresponding to the image to be processed;

[0012] In a case where the image to be processed includes the clear image to be processed and does not include the blurred image to be processed, determining a degradation feature map corresponding to the image to be processed from a plurality of degradation feature maps; or performing feature encoding processing on any acquired reference blurred image to obtain a degradation feature map, and determining the degradation feature map as the degradation feature map corresponding to the image to be processed;

[0013] In the case that the image to be processed includes the clear image to be processed and the blurred image to be processed, feature encoding processing is performed on the blurred image to be processed to obtain a degradation feature map, and the degradation feature map is determined as the degradation feature map corresponding to the clear image to be processed.

[0014] In the embodiment of the present disclosure, a corresponding method of determining the degradation feature map can be flexibly selected according to the type of images included in the images to be processed.

[0015] In a possible implementation, generating a target image based on the image to be processed and the degradation feature map includes:

[0016] Performing feature decoding processing on the image to be processed and the degradation feature map to generate a target feature map corresponding to the image to be processed; wherein the target feature map includes blur pattern information that matches a feature map of at least one resolution of the image to be processed;

[0017] The target feature map is used to process the image to be processed to generate the target image.

[0018] Here, since the target feature map includes blur pattern information that matches a feature map of at least one resolution of the image to be processed, the target feature map can be used to more accurately process the image to be processed and generate a target image with better image quality.

[0019] In one possible implementation, when the image to be processed includes the blurred image to be processed and does not include the clear image to be processed, the target image is generated by a trained first target neural network; the first target neural network includes: an encoder and a first decoder; the encoder is connected to the first decoder;

[0020] In a case where the image to be processed includes the clear image to be processed and does not include the blurred image to be processed, the target image is generated by a trained second target neural network; the second target neural network includes: the encoder and a second decoder; the encoder is connected to the second decoder;

[0021] In the case where the image to be processed includes the clear image to be processed and the blurred image to be processed, the target image is generated by a trained third target neural network; the third target neural network includes: the encoder, a first decoder and a second decoder; the encoder is connected to the first decoder and the second decoder respectively.

[0022] In one possible implementation, the first decoder includes m feature extraction subnetworks, m+1 parameter modulation subnetworks, and m feature fusion subnetworks; wherein m is a positive integer;

[0023] The i-th feature extraction subnetwork includes the i-th first residual subnetwork and the i-th downsampling feature layer; the i-th feature fusion subnetwork includes the i-th upsampling feature layer, the i-th connection layer, and the i-th second residual subnetwork;

[0024] The i-1th feature extraction subnetwork is connected to the i-2th feature extraction subnetwork and the i-th feature extraction subnetwork respectively; the i-1th feature fusion subnetwork is connected to the i-2th feature fusion subnetwork and the i-th feature fusion subnetwork respectively;

[0025] The i-th parameter modulation subnetwork is connected to the i+1-th parameter modulation subnetwork, the i-th first residual subnetwork in the i-th feature extraction subnetwork, and the i-th connection layer in the i-th feature fusion subnetwork respectively; where i is a positive integer greater than or equal to 1 and less than or equal to m;

[0026] The m+1th parameter modulation sub-network is connected to the mth downsampling feature layer in the mth feature extraction sub-network and the mth upsampling feature layer in the mth feature fusion sub-network respectively.

[0027] In the above embodiment, through a two-stage training scheme, the trained target neural network can better learn the degradation representation of the image (i.e., the image degradation pattern), so that the learned image degradation representation can be used to more accurately achieve image deblurring, and the performance of the target neural network is better.

[0028] In one possible implementation, the first target neural network is trained according to the following steps:

[0029] Acquire a first training sample, wherein the first training sample includes a plurality of first sample image pairs, and the first sample image pairs include a first blurred image and a first clear image corresponding to the first blurred image;

[0030] Using an encoder in a first neural network to be trained, performing feature encoding processing on the first blurred image to generate a sample degradation feature map corresponding to the first blurred image;

[0031] Using a first decoder in the first neural network to be trained, performing feature decoding processing on the sample degradation feature map and the first blurred image to generate a first sample feature map corresponding to the first blurred image;

[0032] Performing feature fusion processing on the first sample feature map and the first blurred image to generate a predicted clear image corresponding to the first blurred image;

[0033] Based on the predicted clear image and the first clear image, the first neural network to be trained is trained until the first neural network to be trained meets a first training cutoff condition, thereby obtaining the first target neural network.

[0034] In the above embodiment, a degradation representation map is obtained by performing feature encoding processing on the first blurred image; and the degradation feature map is used to perform deblurring processing on the first blurred image to obtain a predicted clear image corresponding to the first blurred image; and then, the first target neural network trained by the predicted clear image and the first clear image corresponding to the first blurred image can better learn the degradation representation of the image (i.e., the image degradation pattern), so that the image degradation representation can be used to more accurately realize image deblurring, and the performance of the first target neural network is better.

[0035] In one possible implementation, the second target neural network is trained according to the following steps:

[0036] Acquire a second training sample, wherein the second training sample includes a plurality of second sample image pairs, and the second sample image pairs include a second blurred image and a second clear image corresponding to the second blurred image;

[0037] Using an encoder in a second neural network to be trained, performing feature encoding processing on the second blurred image to generate a sample degradation feature map corresponding to the second blurred image;

[0038] Using a second decoder in the second neural network to be trained, performing feature decoding processing on the sample degradation feature map and the second clear image to generate a second sample feature map corresponding to the second clear image;

[0039] Performing feature fusion processing on the second sample feature map and the second clear image to generate a predicted blurred image corresponding to the second clear image;

[0040] Based on the predicted blurred image and the second blurred image, the second neural network to be trained is trained until the second neural network to be trained meets a second training cutoff condition, thereby obtaining the second target neural network.

[0041] In the above embodiment, a degradation representation map is obtained by performing feature encoding processing on the second blurred image; and the degradation feature map is used to blur the second clear image to obtain a predicted blurred image corresponding to the second clear image; and then, the second target neural network trained by the predicted blurred image and the second blurred image can better learn the degradation representation of the image (i.e., the image degradation pattern), so that the image degradation representation can be used to more accurately realize image blurring, and the performance of the second target neural network is better.

[0042] In one possible implementation, the third target neural network is trained according to the following steps:

[0043] Acquire a third training sample, wherein the third training sample includes a plurality of third sample image pairs, and the third sample image pairs include a third blurred image and a third clear image corresponding to the third blurred image;

[0044] Using an encoder in a third neural network to be trained, performing feature encoding processing on the third blurred image to generate a sample degradation feature map corresponding to the third blurred image;

[0045] performing feature decoding processing on the sample degradation feature map and the third blurred image using the first decoder in the third neural network to be trained to generate a third sample feature map corresponding to the third blurred image; and performing feature fusion processing on the third sample feature map and the third blurred image to generate a predicted clear image corresponding to the third blurred image;

[0046] and performing feature decoding processing on the sample degradation feature map and the third clear image using the second decoder in the third neural network to be trained to generate a fourth sample feature map corresponding to the third clear image; and performing feature fusion processing on the fourth sample feature map and the third clear image to generate a predicted blurred image corresponding to the third clear image;

[0047] Based on a first predicted image pair consisting of the predicted blurred image and the third blurred image, and a second predicted image pair consisting of the predicted clear image and the third clear image, the third neural network to be trained is trained until the third neural network to be trained meets a third training cutoff condition, thereby obtaining the third target neural network.

[0048] In the above embodiment, the predicted blurred image and the third blurred image are used to form a first predicted image pair, and the predicted clear image and the third clear image are used to form a second predicted image pair. The third neural network to be trained is trained using an adversarial training method, so that the third neural network to be trained can better learn the image degradation representation, and help the encoder in the third neural network to be trained to extract the degradation representation that is irrelevant to the image content from the blurred image; and then after the third target neural network is trained, the degradation feature map generated by the encoder in the third target neural network can more accurately represent the spatial feature changes between the blurred image and the clear image corresponding to the blurred image.

[0049] In one possible implementation, the step of performing feature decoding processing on the sample degradation feature map and the third blurred image using the first decoder in the third neural network to be trained to generate a third sample feature map corresponding to the third blurred image includes:

[0050] Using the first decoder in the third neural network to be trained, perform feature decoding processing on the third blurred image to generate a multi-level first feature map;

[0051] Determining, based on the sample degradation feature map, a degradation feature map corresponding to the first feature map at each level;

[0052] generating a second feature map corresponding to each level of the first feature map based on the multi-level first feature map and the degradation feature map corresponding to each level of the first feature map;

[0053] The second feature maps corresponding to the multi-level first feature maps are subjected to feature fusion processing to generate a third sample feature map corresponding to the third blurred image.

[0054] In the above embodiment, degradation feature maps corresponding to the first feature maps of each level are determined based on the sample degradation feature map. For example, the sample degradation feature map can be upsampled to obtain degradation feature maps corresponding to the first feature maps of different levels, so that degradation features of different levels can represent different image degradation patterns; then, the first feature map of each level is fused with the degradation feature map corresponding to the first feature map of that level to generate a second feature map corresponding to the first feature map of each level, so that the generated second feature maps of different levels can learn the features of different image degradation patterns; and then, by performing feature fusion processing on the second feature maps of multiple levels to obtain a third sample feature map, the third sample feature map can learn different image degradation patterns, so that the third sample feature map can generate a target image with better image quality using the third sample feature map.

[0055] In one possible implementation, generating the second feature map corresponding to each level of the first feature map based on the multi-level first feature map and the degradation feature map corresponding to each level of the first feature map includes:

[0056] For each level of the first feature map, performing a first convolution process on the degraded feature map corresponding to the first feature map of the level to obtain a first convolution feature map corresponding to the first feature map of the level; and performing a second convolution process on the degraded feature map corresponding to the first feature map of the level to obtain a second convolution feature map corresponding to the first feature map of the level;

[0057] Performing batch normalization on the first feature map of the level to obtain the processed first feature map of the level;

[0058] The first convolution feature map, the second convolution feature map and the first feature map after processing at this level are fused to generate a second feature map corresponding to the first feature map of this level; wherein the first convolution feature map is used to adjust the variance of the processed first feature map; and the second convolution feature map is used to adjust the average value of the processed first feature map.

[0059] Here, by performing batch normalization processing on the first feature map of this level, the features of the first feature map after processing at this level can be made more stable; and the degradation feature map corresponding to the first feature map of this level is subjected to the first convolution processing and the second convolution processing to obtain the first convolution feature map and the second convolution feature map; then the first convolution feature map, the second convolution feature map and the first feature map after processing at this level are fused, for example, the variance of the first feature map after processing at this level is modulated by the first convolution feature map, and the average value of the first feature map after processing at this level is modulated by the second convolution feature map to obtain the second feature map, thereby realizing the modulation of the first feature map by the degradation feature map, and enabling the second feature map to learn the image degradation pattern in the degradation feature map.

[0060] In a possible implementation, the image to be processed is an original image acquired by an image acquisition device. After generating the target image, the method further includes:

[0061] Control the image acquisition device to display the target image.

[0062] Here, when the image to be processed is the original image captured by the image acquisition device, the target image can be generated more quickly, and the image acquisition device can be controlled to display the target image. For example, the original image can be deblurred to generate a clearer target image, thereby improving the image display effect of the image acquisition device.

[0063] In a possible implementation manner, the image to be processed is any blurred video frame in video data. After generating the target image, the method further includes:

[0064] Deblurred video data is generated based on the target images respectively corresponding to the video frames in the video data.

[0065] Here, after obtaining the video data, deblurring processing may be performed on the video data to obtain deblurred video data, thereby improving the quality of the video data.

[0066] The description of the effects of the following devices, electronic equipment, etc. can be found in the description of the above method and will not be repeated here.

[0067] In a second aspect, the present disclosure provides an image processing apparatus, comprising:

[0068] An acquisition module, configured to acquire an image to be processed; wherein the image to be processed includes a blurred image to be processed and / or a clear image to be processed;

[0069] A determination module, configured to determine a degradation feature map corresponding to the image to be processed; wherein the degradation feature map is used to characterize an image degradation pattern of spatial feature changes between a blurred image and a corresponding clear image;

[0070] The first generation module is used to generate a target image based on the image to be processed and the degradation feature map; wherein, when the image to be processed includes a blurred image to be processed, the target image includes a deblurred image corresponding to the blurred image to be processed; when the image to be processed includes a clear image to be processed, the target image includes a blurred image corresponding to the clear image to be processed.

[0071] In a third aspect, the present disclosure provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method described in the first aspect or any one of the embodiments are performed.

[0072] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image processing method as described in the first aspect or any one of the embodiments above are executed.

[0073] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0075] Figure 1 A schematic diagram showing a flow chart of an image processing method provided by an embodiment of the present disclosure is shown;

[0076] Figure 2a A schematic diagram of the structure of a first target neural network in an image processing method provided by an embodiment of the present disclosure is shown;

[0077] Figure 2b A schematic diagram of the structure of a second target neural network in an image processing method provided by an embodiment of the present disclosure is shown;

[0078] Figure 2c A schematic diagram of the structure of a third target neural network in an image processing method provided by an embodiment of the present disclosure is shown;

[0079] Figure 3 A schematic structural diagram of a first decoder in an image processing method provided by an embodiment of the present disclosure is shown;

[0080] Figure 4 A schematic diagram of the structure of a parameter modulation subnetwork in an image processing method provided by an embodiment of the present disclosure is shown;

[0081] Figure 5 A schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure is shown;

[0082] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0084] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0085] During the process of image formation, recording, processing, and transmission, the image quality may deteriorate due to imperfections in the imaging system, recording equipment, transmission media, and processing methods. This phenomenon is called image degradation. For example, image degradation may be caused by the relative motion between the acquisition device and the object, or by the focusing of the acquisition device's lens.

[0086] Degraded images may be blurry or unclear, so deblurring can be performed on them. Alternatively, degrading a clear image can be performed, such as blurring the background area of ​​a clear image to highlight the foreground area.

[0087] Based on this, embodiments of the present disclosure provide an image processing method, apparatus, electronic device, and storage medium.

[0088] To facilitate understanding of the embodiments of the present disclosure, an image processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the image processing method provided in the embodiment of the present disclosure is generally a computer device with a certain computing capability, and the computer device includes, for example: a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the image processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0089] See also Figure 1 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present disclosure, the method comprising: S101-S103, specifically:

[0090] S101, obtaining an image to be processed; the image to be processed includes a blurred image to be processed and / or a clear image to be processed.

[0091] S102 : Determine a degradation feature map corresponding to the image to be processed, wherein the degradation feature map is used to characterize an image degradation pattern of spatial feature changes between a blurred image and a clear image corresponding to the blurred image.

[0092] S103, generating a target image based on the image to be processed and the degradation feature map; wherein, when the image to be processed includes a blurred image to be processed, the target image includes a deblurred image corresponding to the blurred image to be processed; when the image to be processed includes a clear image to be processed, the target image includes a blurred image corresponding to the clear image to be processed.

[0093] In the above method, for the acquired image to be processed, a degradation feature map corresponding to the feature image to be processed can be first determined. Since the degradation feature map can be used to characterize the image degradation pattern of the spatial feature changes between the blurred image and the clear image corresponding to the blurred image, the image quality of the target image can be improved when the target image is generated based on the image to be processed and the degradation feature map corresponding to the image to be processed. For example, when the image to be processed includes a blurred image to be processed, the method can more accurately restore the deblurred image corresponding to the blurred image to be processed by utilizing the determined degradation feature map, resulting in a better deblurring effect. When the image to be processed includes a clear image to be processed, by superimposing the degradation feature map onto the clear image to be processed, the resulting blurred image is more natural, and the display effect of the blurred image is better.

[0094] S101 to S103 are described in detail below.

[0095] For S101 and S102:

[0096] The image to be processed can be acquired through an image acquisition device. For example, the image to be processed can be an original image acquired by the image acquisition device, or can be any video frame in the video data acquired by the image acquisition device. The image acquisition device can be a mobile phone, a computer, a camera, etc.

[0097] The image to be processed may include a blurred image to be processed and / or a clear image to be processed, and a degradation feature map corresponding to the image to be processed may be determined, wherein the degradation feature map is used to characterize the image degradation pattern of the spatial feature changes between the blurred image and the clear image corresponding to the blurred image; for example, the image degradation pattern includes but is not limited to a motion degradation pattern, a blur degradation pattern, a nonlinear degradation pattern, etc.

[0098] For example, a trained neural network can be used to extract a degradation feature map from any blurred image, and the extracted degradation feature map can be used as the degradation feature map corresponding to the image to be processed. Alternatively, the degradation feature map corresponding to the image to be processed can be determined from a plurality of pre-determined degradation feature maps.

[0099] Determine the degradation feature map corresponding to the image to be processed, including the following methods:

[0100] Method 1: When the image to be processed includes a blurred image to be processed but does not include a clear image to be processed, feature coding processing is performed on the image to be processed to obtain a degradation feature map corresponding to the image to be processed.

[0101] During implementation, when the image to be processed includes a blurred image to be processed but does not include a clear image to be processed, the image to be processed (i.e., the blurred image to be processed) can be input into the trained encoder, and the encoder performs feature encoding processing on the image to be processed to obtain a degraded feature map corresponding to the image to be processed; so that the blurred image to be processed can be deblurred based on the obtained degraded feature map.

[0102] Method 2: When the image to be processed includes a clear image to be processed and does not include a blurred image to be processed, the degradation feature map corresponding to the image to be processed is determined from multiple degradation feature maps that are set; or, feature encoding processing is performed on any obtained reference blurred image to obtain a degradation feature map, and the degradation feature map is determined as the degradation feature map corresponding to the image to be processed.

[0103] During implementation, when the image to be processed includes a clear image to be processed and does not include a blurred image to be processed, the degradation feature map corresponding to the image to be processed can be determined from the set multiple degradation feature maps; for example, an encoder can be used to perform feature encoding processing on the pre-set multiple blurred images to obtain multiple degradation feature maps.

[0104] Alternatively, any reference blurred image can be obtained first, and then the reference blurred image can be input into an encoder. The encoder performs feature encoding processing on the reference blurred image to obtain a degradation feature map, and the degradation feature map is determined as the degradation feature map corresponding to the image to be processed (i.e., the clear image to be processed); so that the clear image to be processed can be blurred based on the obtained degradation feature map.

[0105] Method three: when the image to be processed includes a clear image to be processed and a blurred image to be processed, feature coding is performed on the blurred image to be processed to obtain a degradation feature map, and the degradation feature map is determined as the degradation feature map corresponding to the clear image to be processed.

[0106] During implementation, when the image to be processed includes a clear image to be processed and a blurred image to be processed, the blurred image to be processed can be input into an encoder, and the encoder performs feature encoding processing on the blurred image to be processed to obtain a degradation feature map, and the degradation feature map is determined as the degradation feature map corresponding to the clear image to be processed; so that the blurred image to be processed can be deblurred or the clear image to be processed can be blurred based on the obtained degradation feature map.

[0107] In the embodiment of the present disclosure, a corresponding method of determining the degradation feature map can be flexibly selected according to the type of images included in the images to be processed.

[0108] For S103:

[0109] During implementation, a target image can be generated based on a target image and a degradation feature map corresponding to the target image. If the target image includes a blurred target image, the blurred target image can be deblurred using the degradation feature map to generate a deblurred image corresponding to the blurred target image. For example, degradation feature information matching the image degradation pattern represented by the degradation feature map can be extracted from the blurred target image based on the degradation feature map, and the deblurred image can be obtained by fusing the degradation feature information with the blurred target image.

[0110] When the image to be processed includes a clear image to be processed, the clear image to be processed is defocused (e.g., blurred) using the degradation feature map to generate a defocused image corresponding to the clear image to be processed. For example, the degradation feature map can be fused with the clear image to be processed to obtain the defocused image.

[0111] In one possible implementation, generating a target image based on the image to be processed and the degradation feature map includes:

[0112] Step A1: perform feature decoding processing on the image to be processed and the degradation feature map to generate a target feature map corresponding to the image to be processed; wherein the target feature map includes blur pattern information that matches the feature map of at least one resolution of the image to be processed.

[0113] Step A2: Process the image to be processed using the target feature map to generate a target image.

[0114] During implementation, feature decoding processing can be performed on the image to be processed and the degraded feature map corresponding to the image to be processed to generate a target feature map corresponding to the image to be processed. The target feature map may include blur pattern information that matches the feature map of at least one resolution of the image to be processed.

[0115] Perform feature extraction on the image to be processed at least once to obtain at least one feature map corresponding to the image to be processed. When there are multiple feature maps, the resolutions of the multiple feature maps are different, and different resolutions correspond to different blur pattern information. A target feature map can then be generated based on the multiple feature maps and the degraded feature maps. The target feature map can include blur pattern information that matches the feature map of at least one resolution. For example, the multiple feature maps can include a feature map with a resolution of 512×512, a feature map with a resolution of 256×256, and so on. Each feature map of each resolution matches a blur pattern information. The feature values ​​of the same feature position in the target feature map and the image to be processed can then be added to obtain the target image.

[0116] Here, since the target feature map includes blur pattern information that matches a feature map of at least one resolution of the image to be processed, the target feature map can be used to more accurately process the image to be processed and generate a target image with better image quality.

[0117] In the disclosed embodiments, the target image can be generated by a trained target neural network. The target neural network may include an encoder and a decoder. When determining the degradation feature map corresponding to the image to be processed, the image to be processed can be input into the encoder, which performs feature encoding on the image to be processed to obtain the degradation feature map corresponding to the image to be processed. Furthermore, the image to be processed and the degradation feature map can be input into the decoder, which decodes the image to be processed and the degradation feature map to generate the target image.

[0118] Among them, since the images to be processed include different types of images, target neural networks with different network structures can be used to determine the target image.

[0119] The structure of the target neural network is described in detail below.

[0120] Case 1: When the image to be processed includes a blurred image to be processed but does not include a clear image to be processed, the target image is generated by a trained first target neural network; the first target neural network includes: an encoder and a first decoder; the encoder is connected to the first decoder.

[0121] For example, see Figure 2a In an image processing method shown, a structural schematic diagram of the first target neural network; the first target neural network includes: an encoder and a first decoder; the encoder is connected to the first decoder; the encoder can be used to perform feature encoder processing on the blurred image to be processed, obtain a degraded feature map corresponding to the blurred image to be processed, and input the degraded feature map to the first decoder; the first decoder can be used to perform feature decoding processing on the degraded feature map and the blurred image to be processed, generate a first target feature map corresponding to the blurred image to be processed, and then fuse the first target feature map with the blurred image to be processed to obtain a deblurred image.

[0122] Case 2: When the image to be processed includes a clear image to be processed and does not include a blurred image to be processed, the target image is generated by the trained second target neural network; the second target neural network includes: an encoder and a second decoder; the encoder is connected to the second decoder.

[0123] For example, see Figure 2b A schematic diagram of the structure of a second target neural network in an image processing method is shown. The second target neural network includes an encoder and a second decoder, with the encoder connected to the second decoder. The encoder can be used to perform feature encoding processing on any reference blurred image to obtain a degraded feature map corresponding to the reference blurred image (i.e., a degraded feature map corresponding to the clear image to be processed), and this degraded feature map is input to the second decoder. The second decoder can be used to perform feature decoding processing on the degraded feature map and the clear image to be processed to generate a second target feature map corresponding to the clear image to be processed. The second target feature map is then fused with the clear image to obtain a blurred image.

[0124] Case 3: When the images to be processed include clear images to be processed and blurred images to be processed, the target image is generated by the trained third target neural network; the third target neural network includes: an encoder, a first decoder and a second decoder; the encoder is connected to the first decoder and the second decoder respectively.

[0125] For example, see Figure 2cA schematic diagram of the structure of a third target neural network in an image processing method shown; the third target neural network includes: an encoder, a first decoder, and a second decoder; the encoder is connected to the first decoder and the second decoder, respectively. The encoder can be used to perform feature encoding processing on the blurred image to be processed to obtain a degraded feature map, and the degraded feature map is input to the first decoder and the second decoder, respectively; the first decoder can be used to perform feature decoding processing on the degraded feature map and the blurred image to be processed to generate a third target feature map corresponding to the blurred image to be processed; the third feature map is then fused with the blurred image to be processed to obtain a deblurred image. Alternatively, the second decoder can be used to perform feature decoding processing on the degraded feature map and the clear image to be processed to generate a fourth target feature map corresponding to the clear image to be processed; the fourth feature map is then fused with the clear image to be processed to obtain a blurred image.

[0126] The image contents of the clear image to be processed and the blurred image to be processed may be completely identical, partially identical, or completely different.

[0127] In specific implementation, when the blurred image to be processed is input into the first decoder, the third target neural network can generate a deblurred image corresponding to the blurred image to be processed; when the clear image to be processed is input into the second decoder, the third target neural network can generate a blurred image corresponding to the clear image to be processed; when the blurred image to be processed is input into the first decoder and the clear image to be processed is input into the second decoder, the third target neural network can generate a deblurred image corresponding to the blurred image to be processed and a blurred image corresponding to the clear image to be processed.

[0128] In a specific implementation, the first decoder includes m feature extraction subnetworks, m+1 parameter modulation subnetworks, and m feature fusion subnetworks; where m is a positive integer. The i-th feature extraction subnetwork includes the i-th first residual subnetwork and the i-th downsampling feature layer; the i-th feature fusion subnetwork includes the i-th upsampling feature layer, the i-th connection layer, and the i-th second residual subnetwork; the i-1-th feature extraction subnetwork is connected to the i-2-th feature extraction subnetwork and the i-th feature extraction subnetwork respectively; the i-1-th feature fusion subnetwork is connected to the i-2-th feature fusion subnetwork and the i-th feature fusion subnetwork respectively; the i-th parameter modulation subnetwork is connected to the i+1-th parameter modulation subnetwork, the i-th first residual subnetwork in the i-th feature extraction subnetwork, and the i-th connection layer in the i-th feature fusion subnetwork respectively; where i is a positive integer greater than or equal to 1 and less than or equal to m; the m+1-th parameter modulation subnetwork is connected to the m-th downsampling feature layer in the m-th feature extraction subnetwork and the m-th upsampling feature layer in the m-th feature fusion subnetwork respectively.

[0129] See also Figure 3 The structural diagram of the first decoder shown in FIG. 1 includes 4 feature extraction subnetworks, 5 parameter modulation subnetworks, and 4 feature fusion subnetworks; that is, m is 4. Among them, the first feature extraction subnetwork (i.e., the feature extraction subnetwork in the first stage) is connected to the second feature extraction subnetwork, the second feature extraction subnetwork is connected to the third feature extraction subnetwork, and the third feature extraction subnetwork is connected to the fourth feature extraction subnetwork; the first feature fusion subnetwork is connected to the second feature fusion subnetwork, the second feature fusion subnetwork is connected to the third feature fusion subnetwork, and the third feature fusion subnetwork is connected to the fourth feature fusion subnetwork.

[0130] Each feature extraction subnetwork includes a first residual subnetwork and a downsampling feature layer; each feature fusion subnetwork includes an upsampling feature layer, a connection layer, and a second residual subnetwork. The first parameter modulation subnetwork (i.e., the parameter modulation subnetwork in the first level) is respectively connected to the second parameter modulation subnetwork, the first first residual subnetwork in the first feature extraction subnetwork, and the first connection layer in the first feature fusion subnetwork; the second parameter modulation subnetwork is respectively connected to the third parameter modulation subnetwork, the second first residual subnetwork in the second feature extraction subnetwork, and the second connection layer in the second feature fusion subnetwork; the third parameter modulation subnetwork is respectively connected to the fourth parameter modulation subnetwork. The fourth parameter modulation subnetwork is connected to the fifth parameter modulation subnetwork, the fourth first residual subnetwork in the fourth feature extraction subnetwork, and the fourth connection layer in the fourth feature fusion subnetwork; the fifth parameter modulation subnetwork is connected to the fourth downsampling feature layer in the fourth feature extraction subnetwork and the fourth upsampling feature layer in the fourth feature fusion subnetwork.

[0131] The second decoder may include n feature extraction subnetworks, n+1 parameter modulation subnetworks, and n feature fusion subnetworks; wherein n is a positive integer; n may be greater than m, less than m, or equal to m; since the structure of the second decoder is similar to that of the first decoder, the specific structure of the second decoder and the connection status of each subnetwork can refer to the above description and will not be repeated here.

[0132] The following exemplifies the training process of the first target neural network, the training process of the second target neural network, and the training process of the third target neural network.

[0133] During implementation, the first target neural network can be trained according to the following steps:

[0134] S201 : Acquire a first training sample, wherein the first training sample includes a plurality of first sample image pairs, and the first sample image pair includes a first blurred image and a first clear image corresponding to the first blurred image.

[0135] S202 : Perform feature encoding processing on the first blurred image using an encoder in the first neural network to be trained, to generate a sample degradation feature map corresponding to the first blurred image.

[0136] S203 , using a first decoder in the first neural network to be trained, performing feature decoding processing on the sample degradation feature map and the first blurred image to generate a first sample feature map corresponding to the first blurred image.

[0137] S204: Perform feature fusion processing on the first sample feature map and the first blurred image to generate a predicted clear image corresponding to the first blurred image.

[0138] S205 , training the first neural network to be trained based on the predicted clear image and the first clear image until the first neural network to be trained meets a first training cutoff condition, thereby obtaining a first target neural network.

[0139] During implementation, the acquired first training sample may include multiple first sample image pairs, each of which may include a first blurred image and a first clear image corresponding to the first blurred image. The acquired first blurred image may be input into an encoder in a first neural network to be trained, and the encoder may perform feature encoding processing on the first blurred image to generate a sample degradation feature map corresponding to the first blurred image. The sample degradation feature map and the first blurred image may then be input into a first decoder in the first neural network to be trained, and the first decoder may perform feature decoding processing on the sample degradation feature map and the first blurred image to generate a first sample feature map corresponding to the first blurred image. Feature values ​​at the same feature position in the first sample feature map and the first blurred image may then be added together to generate a predicted clear image corresponding to the first blurred image.

[0140] Furthermore, a first loss value can be determined based on the predicted clear image and the first clear image, and the network parameters of the first neural network to be trained can be adjusted using the first loss value until the first neural network to be trained meets a first training cutoff condition, such as the first training cutoff condition including the first loss value being less than a set first loss threshold, the first neural network to be trained converging, the number of training times of the first neural network to be trained being greater than a set first quantity threshold, etc.

[0141] In the above embodiment, a degradation representation map is obtained by performing feature encoding processing on the first blurred image; and the degradation feature map is used to perform deblurring processing on the first blurred image to obtain a predicted clear image corresponding to the first blurred image; and then, the first target neural network trained by the predicted clear image and the first clear image corresponding to the first blurred image can better learn the degradation representation of the image (i.e., the image degradation pattern), so that the image degradation representation can be used to more accurately realize image deblurring, and the performance of the first target neural network is better.

[0142] During implementation, the second target neural network can be trained according to the following steps:

[0143] S301 : Acquire a second training sample, wherein the second training sample includes a plurality of second sample image pairs, and the second sample image pairs include a second blurred image and a second clear image corresponding to the second blurred image.

[0144] S302, using an encoder in a second neural network to be trained, performing feature encoding processing on the second blurred image to generate a sample degradation feature map corresponding to the second blurred image;

[0145] S303 , using a second decoder in a second neural network to be trained, performing feature decoding processing on the sample degradation feature map and the second clear image to generate a second sample feature map corresponding to the second clear image.

[0146] S304: Perform feature fusion processing on the second sample feature map and the second clear image to generate a predicted blurred image corresponding to the second clear image.

[0147] S305 , training the second neural network to be trained based on the predicted blurred image and the second blurred image until the second neural network to be trained meets a second training cutoff condition, thereby obtaining a second target neural network.

[0148] During implementation, the acquired second training sample may include multiple second sample image pairs, each of which may include a second blurred image and a second clear image corresponding to the second blurred image. The acquired second blurred image may be input into an encoder in a second neural network to be trained, and feature encoding processing may be performed on the second blurred image by the encoder to generate a sample degradation feature map corresponding to the second blurred image. The sample degradation feature map and the second clear image may then be input into a second decoder in the second neural network to be trained, and feature decoding processing may be performed on the sample degradation feature map and the second clear image by the second decoder to generate a second sample feature map corresponding to the second clear image. Feature values ​​at the same feature position in the second sample feature map and the second clear image may then be added to generate a predicted blurred image corresponding to the second clear image.

[0149] Furthermore, a second loss value can be determined based on the predicted blurred image and the second blurred image, and the network parameters of the second neural network to be trained can be adjusted using the second loss value until the second neural network to be trained meets the second training cutoff condition, such as the second training cutoff condition including the second loss value being less than a set second loss threshold, the second neural network to be trained converging, the number of training times of the second neural network to be trained being greater than a set second quantity threshold, etc.

[0150] In the above embodiment, a degradation representation map is obtained by performing feature encoding processing on the second blurred image; and the degradation feature map is used to blur the second clear image to obtain a predicted blurred image corresponding to the second clear image; and then, the second target neural network trained by the predicted blurred image and the second blurred image can better learn the degradation representation of the image (i.e., the image degradation pattern), so that the image degradation representation can be used to more accurately realize image blurring, and the performance of the second target neural network is better.

[0151] During implementation, the third target neural network can be trained according to the following steps:

[0152] S401 : Acquire a third training sample, wherein the third training sample includes a plurality of third sample image pairs, and the third sample image pair includes a third blurred image and a third clear image corresponding to the third blurred image.

[0153] S402 : Perform feature encoding processing on the third blurred image using an encoder in the third neural network to be trained, to generate a sample degradation feature map corresponding to the third blurred image.

[0154] S403, using the first decoder in the third neural network to be trained, performing feature decoding processing on the sample degradation feature map and the third blurred image to generate a third sample feature map corresponding to the third blurred image; and performing feature fusion processing on the third sample feature map and the third blurred image to generate a predicted clear image corresponding to the third blurred image.

[0155] S404, using the second decoder in the third neural network to be trained, perform feature decoding processing on the sample degradation feature map and the third clear image to generate a fourth sample feature map corresponding to the third clear image; and perform feature fusion processing on the fourth sample feature map and the third clear image to generate a predicted blurred image corresponding to the third clear image.

[0156] S405, based on the first predicted image pair consisting of the predicted blurred image and the third blurred image, and the second predicted image pair consisting of the predicted clear image and the third clear image, train the third neural network to be trained until the third neural network to be trained meets the third training cutoff condition, thereby obtaining a third target neural network.

[0157] During implementation, the acquired third training samples may include multiple third sample image pairs, each of which may include a third blurred image and a third clear image corresponding to the third blurred image. The acquired third blurred image may be input into an encoder in the third neural network to be trained, where the encoder performs feature encoding processing on the third blurred image to generate a sample degradation feature map corresponding to the third blurred image.

[0158] The sample degradation feature map and the third blurred image can be input into a first decoder in a third neural network to be trained, and the first decoder is used to perform feature decoding processing on the sample degradation feature map and the third blurred image to generate a third sample feature map corresponding to the third blurred image; the third sample feature map and the feature values ​​of the same feature position in the third blurred image can then be added together to generate a predicted clear image corresponding to the third blurred image; and the sample degradation feature map and the third clear image can be input into a second decoder in a third neural network to be trained, and the second decoder is used to perform feature decoding processing on the sample degradation feature map and the third clear image to generate a fourth sample feature map corresponding to the third clear image; the fourth sample feature map and the feature values ​​of the same feature position in the third clear image can then be added together to generate a predicted blurred image corresponding to the third clear image. The predicted blurred image and the third blurred image constitute a first predicted image pair, and the predicted clear image and the third clear image constitute a second predicted image pair.

[0159] Furthermore, a third loss value can be determined based on the first predicted image pair and the second predicted image pair, and the network parameters of the third neural network to be trained can be adjusted using the third loss value until the third neural network to be trained meets a third training cutoff condition. For example, the third training cutoff condition includes the third loss value being less than a set third loss threshold, the third neural network to be trained converges, and the number of training times for the third neural network to be trained being greater than a set third number threshold. The first training cutoff condition, the second training cutoff condition, and the third training cutoff condition can be the same or different.

[0160] In the above embodiment, the predicted blurred image and the third blurred image are used to form a first predicted image pair, and the predicted clear image and the third clear image are used to form a second predicted image pair. The third neural network to be trained is trained using an adversarial training method, so that the third neural network to be trained can better learn the image degradation representation, and help the encoder in the third neural network to be trained to extract the degradation representation that is irrelevant to the image content from the blurred image; and then after the third target neural network is trained, the degradation feature map generated by the encoder in the third target neural network can more accurately represent the spatial feature changes between the blurred image and the clear image corresponding to the blurred image.

[0161] At the same time, based on the first predicted image pair consisting of the predicted blurred image and the third blurred image, and the second predicted image pair consisting of the predicted clear image and the third clear image, the third neural network to be trained is trained, thereby realizing adversarial training of the third neural network to be trained. The third target neural network learns richer information, so that the trained third target neural network has higher accuracy and better performance.

[0162] The learning-based deblurring algorithms in the related art have little exploration and research on the degradation representation of images, which makes it impossible for the blur kernel estimated by the algorithm to guarantee the improvement of deblurring performance, resulting in unstable image quality output by the algorithm and poor deblurring effect. The present disclosure, by adding an encoder to a target neural network (such as a first target neural network, a second target neural network, etc.), uses the encoder to learn the degradation representation of the blurred image, and realizes the research on the degradation representation, so that the subsequent decoder (such as the first decoder) can use the learned degradation feature map to perform deblurring or blurring processing, thereby improving the deblurring effect or blurring effect, and making the deblurred image or blurring image have better image quality.

[0163] The following is an example of the process of the first decoder. Since the network structure of the first decoder and the second decoder is the same, the specific process of the second decoder can refer to the following description.

[0164] In S403, when the first decoder in the third neural network to be trained is used to perform feature decoding processing on the sample degradation feature map and the third blurred image to generate a third sample feature map corresponding to the third blurred image, the following steps are specifically included:

[0165] S4031: Utilize the first decoder in the third neural network to be trained to perform feature decoding processing on the third blurred image to generate a multi-level first feature map.

[0166] S4032: Determine a degradation feature map corresponding to the first feature map of each level based on the sample degradation feature map.

[0167] S4033 : Based on the multiple levels of first feature maps and the degraded feature map corresponding to each level of the first feature map, generate a second feature map corresponding to each level of the first feature map.

[0168] S4034: Perform feature fusion processing on the second feature maps corresponding to the multi-level first feature maps to generate a third sample feature map corresponding to the third blurred image.

[0169] During implementation, the third blurred image is input into the first decoder of the third neural network to be trained, and at least one feature extraction sub-network can be used to perform feature extraction processing on the third blurred image to obtain a multi-level first feature map; Figure 3As shown, if the multi-level feature extraction module includes a feature extraction subnetwork at the first level, a feature extraction subnetwork at the second level, a feature extraction subnetwork at the third level, and a feature extraction subnetwork at the fourth level, the process of obtaining the multi-level first feature map includes the following:

[0170] 1. Input the third blurred image into the first-level feature extraction subnetwork. The first residual subnetwork in the first-level feature extraction subnetwork performs feature extraction on the received third blurred image to obtain the first-level first feature map. Then, downsample the first-level first feature map and use the feature map obtained after downsampling. Figure 1 Input to the second-level feature extraction sub-network.

[0171] 2. The first residual sub-network in the second-level feature extraction sub-network performs the feature extraction on the feature. Figure 1 Perform feature extraction processing to obtain the first feature map of the second level; then perform downsampling processing on the first feature map of the second level, and input the feature map 2 obtained after the downsampling processing into the feature extraction subnetwork of the third level.

[0172] 3. The first residual subnetwork in the third-level feature extraction subnetwork performs feature extraction on the feature map 2 to obtain the third-level first feature map; then downsamples the third-level first feature map and converts the feature map obtained after downsampling into Figure 3 Input to the fourth-level feature extraction sub-network.

[0173] 4. The first residual sub-network in the fourth-level feature extraction sub-network performs the feature extraction on the feature. Figure 3 Perform feature extraction processing to obtain the fourth-level first feature map; then perform downsampling processing on the fourth-level first feature map to obtain the feature Figure 4 , and the feature Figure 4 The fifth-level first feature map is determined. Through the above process, a multi-level first feature map (ie, a five-level first feature map) is obtained.

[0174] The first residual sub-network may be a Resblock module, which may include at least one convolutional layer.

[0175] Here, the parameter information corresponding to the first feature maps of different levels is different, and the parameter information includes one or more of width, height, and number of channels. The larger the level, the smaller the width value or height value of the first feature map, and / or the more channels. For example, the width value or height value of the first-level first feature map (level 1) is greater than or equal to the second-level first feature map (level 2); and / or the number of channels of the first-level first feature map is less than or equal to the second-level first feature map.

[0176] In the embodiment of the present disclosure, the parameter information corresponding to the first feature map may include length value × width value × number of channels; the parameter information corresponding to the first feature maps of different levels is different, for example, the size of the first-level first feature map can be 1024×1024×3, and the size of the second-level first feature map can be 512×512×28, etc.

[0177] During implementation, for each level of the first feature map, the degradation feature map corresponding to each level of the first feature map can be determined based on the sample degradation feature map; for example, when the multi-level first feature map includes the first level first feature map to the fifth level first feature map, the degradation feature map 0 corresponding to the fifth level first feature map can be the sample degradation feature map; the degradation feature map corresponding to the fourth level first feature map can be the sample degradation feature map. Figure 1 The degradation feature map of the sample can be obtained by performing an upsampling process on the degradation feature map. The degradation feature map 2 corresponding to the first feature map of the third level can be obtained by performing an upsampling process on the degradation feature map. Figure 1 The degraded feature map corresponding to the first feature map of each level can be obtained by performing an upsampling process. The method for upsampling the sample degraded feature map can be determined according to actual needs. For example, the upsampling method can be an interpolation method, a transposed convolution method, an up-pooling method, etc.

[0178] Then, based on the multi-level first feature maps and the degraded feature maps corresponding to each level of the first feature map, a second feature map corresponding to each level of the first feature map is generated. Figure 3 The generation of the second feature map corresponding to each level of the first feature map is described. The multi-level first feature map includes the first feature map of the first level to the fifth level. The fifth level first feature map and the degradation feature map corresponding to the fifth level first feature map (i.e., the sample degradation feature map) are first input into the parameter modulation subnetwork at the fifth level to obtain the degradation feature map corresponding to the fourth level first feature map and the second feature map corresponding to the fifth level first feature map. The fourth level first feature map and the degradation feature map corresponding to the fourth level first feature map are then input into the parameter modulation subnetwork at the fourth level to obtain the degradation feature map corresponding to the third level first feature map and the second feature map corresponding to the fourth level first feature map. By analogy, the second feature map corresponding to each level of the first feature map can be obtained.

[0179] Then, the second feature maps corresponding to the multi-level first feature maps can be subjected to feature fusion processing to generate a third sample feature map corresponding to the third blurred image. Figure 3For example, if the multi-level second feature map includes the first-level second feature map (i.e., the first-level second feature map corresponding to the first-level first feature map) to the fifth-level second feature map, the fifth-level second feature map (i.e., the fifth-level second feature map output by the fifth-level parameter modulation subnetwork) can be first input into the feature fusion subnetwork at the fourth level, and the fifth-level second feature map is upsampled using the fourth-level upsampling feature layer to generate a fifth-level intermediate feature map. The fifth-level intermediate feature map and the fourth-level second feature map are input into the fourth-level connection layer for cascading, and the cascaded feature map is input into the fourth-level second residual subnetwork for feature extraction to generate a fourth-level fusion feature map.

[0180] The 4th-level fusion feature map is then input into the feature fusion subnetwork of the third level, and the 4th-level fusion feature map is upsampled by the upsampling feature layer of the third level to generate a 4th-level intermediate feature map; the 4th-level intermediate feature map and the 3rd-level second feature map are input into the connection layer for cascading, and the cascaded feature map is input into the second residual subnetwork of the third level for feature extraction to generate a 3rd-level fusion feature map. Based on the same processing flow, the 2nd-level fusion feature map and the 1st-level fusion feature map can be obtained; finally, the 1st-level fusion feature map is determined as the third sample feature map corresponding to the third blurred image.

[0181] In the above embodiment, degradation feature maps corresponding to the first feature maps of each level are determined based on the sample degradation feature map. For example, the sample degradation feature map can be upsampled to obtain degradation feature maps corresponding to the first feature maps of different levels, so that degradation features of different levels can represent different image degradation patterns; then, the first feature map of each level is fused with the degradation feature map corresponding to the first feature map of that level to generate a second feature map corresponding to the first feature map of each level, so that the generated second feature maps of different levels can learn the features of different image degradation patterns; and then, by performing feature fusion processing on the second feature maps of multiple levels to obtain a third sample feature map, the third sample feature map can learn different image degradation patterns, so that the third sample feature map can generate a target image with better image quality using the third sample feature map.

[0182] In one possible implementation, in S4033, generating a second feature map corresponding to each level of the first feature map based on the multiple levels of the first feature map and the degradation feature map corresponding to each level of the first feature map includes:

[0183] Step B1: For each level of the first feature map, perform a first convolution process on the degraded feature map corresponding to the first feature map of the level to obtain a first convolution feature map corresponding to the first feature map of the level; and perform a second convolution process on the degraded feature map corresponding to the first feature map of the level to obtain a second convolution feature map corresponding to the first feature map of the level.

[0184] Step B2: performing batch normalization on the first feature map of the level to obtain the processed first feature map of the level.

[0185] Step B3: Fusing the first convolution feature map, the second convolution feature map, and the first feature map after processing at this level to generate a second feature map corresponding to the first feature map of this level; wherein the first convolution feature map is used to adjust the variance of the processed first feature map; and the second convolution feature map is used to adjust the average value of the processed first feature map.

[0186] During implementation, for each level of the first feature map, at least one convolutional layer may be used to perform a first convolution process on the degraded feature map corresponding to the first feature map of that level to obtain a first convolutional feature map corresponding to the first feature map of that level; and at least one convolutional layer may be used to perform a second convolution process on the degraded feature map corresponding to the first feature map of that level to obtain a second convolutional feature map corresponding to the first feature map of that level. Simultaneously, batch normalization (BN) processing may be performed on the first feature map of that level to obtain a processed first feature map of that level.

[0187] The first convolution feature map and the second convolution feature map can then be used to adjust the first feature map after processing at that level to generate a second feature map corresponding to the first feature map of that level. The first convolution feature map can be used to adjust the variance of the processed first feature map; the second convolution feature map can be used to adjust the average value of the processed first feature map.

[0188] See also Figure 4 The structural diagram of the parameter modulation sub-network shown in the figure is combined with Figure 4 The process of generating the second feature map is exemplified. The specific process is as follows: First, the first feature map of the level is input into the batch normalization subnetwork, and the batch normalization subnetwork performs batch normalization on the first feature map of the level to obtain the first feature map after processing of the level. Second, the degraded feature map M corresponding to the first feature map of the level is obtained. i Input to the first convolution feature processing layer, the first convolution feature processing layer processes the degraded feature map M i Perform convolution processing to obtain the first convolution feature map corresponding to the first feature map of this level; and the degradation feature map M corresponding to the first feature map of this level i Input to the second convolution feature processing layer, the second convolution feature processing layer processes the degraded feature map M iPerform convolution to obtain a second convolution feature map corresponding to the first feature map of that level. Third, multiply the feature values ​​at the same feature position in the first convolution feature map and the first feature map after processing to obtain a multiplied feature map; add the feature values ​​at the same feature position in the second convolution feature map and the multiplied feature map to obtain a second feature map corresponding to the first feature map of that level. The first and second steps are not in any particular order and can be processed in parallel.

[0189] The parameter modulation sub-network can also perform upsampling on the degraded feature map to obtain the degraded feature map M corresponding to the first feature map of the previous level of the first feature map of this level. i+1 For example, if the first feature map of this level is the first feature map of the fourth level, the parameter modulation subnetwork can obtain the degraded feature map corresponding to the first feature map of the third level while obtaining the second feature map corresponding to the first feature map of the fourth level.

[0190] Among them, the structure of the parameter modulation sub-network is not unique. Figure 4 This is only an exemplary description and can be flexibly transformed as needed in actual use. For example, the parameter modulation subnetwork can also include a third convolutional feature processing layer. During implementation, the degraded feature map corresponding to the first feature map of the level can be first input into the third convolutional feature processing layer for convolution processing to obtain a convolutional feature map; then the convolutional feature map is input into the first convolutional feature processing layer and the second convolutional feature processing layer respectively to obtain the first convolutional feature map and the second convolutional feature map; finally, the first convolutional feature map, the second convolutional feature map and the first feature map after processing of the level are fused to generate the second feature map corresponding to the first feature map of the level.

[0191] Here, by performing batch normalization processing on the first feature map of this level, the features of the first feature map after processing at this level can be made more stable; and the degradation feature map corresponding to the first feature map of this level is subjected to the first convolution processing and the second convolution processing to obtain the first convolution feature map and the second convolution feature map; then the first convolution feature map, the second convolution feature map and the first feature map after processing at this level are fused, for example, the variance of the first feature map after processing at this level is modulated by the first convolution feature map, and the average value of the first feature map after processing at this level is modulated by the second convolution feature map to obtain the second feature map, thereby realizing the modulation of the first feature map by the degradation feature map, and enabling the second feature map to learn the image degradation pattern in the degradation feature map.

[0192] During implementation, the target neural network (such as the first target neural network and the third target neural network) can be integrated into an image acquisition device, such as a mobile phone, a tablet, a camera, etc., so that after the image acquisition device acquires the original image, the target neural network can be used to generate a clearer target image and display it; or after the server or terminal device obtains the video data, the target neural network can be used to deblur each video frame in the video data to generate deblurred video data.

[0193] Alternatively, after the image acquisition device captures the original image, the target neural network (such as the second target neural network or the third target neural network) can be used to blur the original image to generate a target image that can protect privacy information and display it; or after the server or terminal device obtains the video data, the target neural network can be used to blur any video frame in the video data to generate blurred video data.

[0194] In a possible implementation, the image to be processed is an original image acquired by an image acquisition device. After generating the target image, the method further includes: controlling the image acquisition device to display the target image.

[0195] During implementation, the target neural network can be integrated into an image acquisition device. After the image acquisition device captures an original image, the original image is used as the image to be processed, and the process from S101 to S103 is executed to generate a target image. The image acquisition device is then controlled to display the target image. If the original image is a blurred image, the generated target image is a deblurred image; if the original image is a clear image, the generated target image is a blurred image.

[0196] Among them, when controlling the image acquisition device to display the target image, only the target image can be displayed and the original image can be deleted; the original image and the target image can be retained and only the target image can be displayed; the original image and the target image can be retained and both can be displayed at the same time.

[0197] Here, when the image to be processed is the original image captured by the image acquisition device, the target image can be generated more quickly, and the image acquisition device can be controlled to display the target image. For example, the original image can be deblurred to generate a clearer target image, thereby improving the image display effect of the image acquisition device.

[0198] In one possible implementation, the image to be processed is any blurred video frame in the video data. After generating the target image, the method further includes: generating deblurred video data based on the target images corresponding to each video frame in the video data.

[0199] During implementation, after the image acquisition device acquires the video data, the process from S101 to S103 can be executed for each video frame in the video data to generate a target image corresponding to the video frame in real time; and then, based on the target images corresponding to each video frame, the deblurred video data can be obtained.

[0200] Here, after obtaining the video data, deblurring processing may be performed on the video data to obtain deblurred video data, thereby improving the quality of the video data.

[0201] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0202] Based on the same concept, the present disclosure also provides an image processing device, see Figure 6 FIG. 5 is a schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure, including an acquisition module 501, a determination module 502, and a first generation module 503. Specifically:

[0203] An acquisition module 501 is configured to acquire an image to be processed; the image to be processed includes a blurred image to be processed and / or a clear image to be processed;

[0204] A determination module 502 is configured to determine a degradation feature map corresponding to the image to be processed, wherein the degradation feature map is used to characterize an image degradation pattern of spatial feature changes between a blurred image and a clear image corresponding to the blurred image;

[0205] The first generation module 503 is used to generate a target image based on the image to be processed and the degradation feature map; wherein, when the image to be processed includes a blurred image to be processed, the target image includes a deblurred image corresponding to the blurred image to be processed; when the image to be processed includes a clear image to be processed, the target image includes a blurred image corresponding to the clear image to be processed.

[0206] In a possible implementation, the determining module 502, when determining the degradation feature map corresponding to the image to be processed, is configured to:

[0207] In a case where the image to be processed includes the blurred image to be processed but does not include the clear image to be processed, performing feature coding processing on the image to be processed to obtain a degradation feature map corresponding to the image to be processed;

[0208] In a case where the image to be processed includes the clear image to be processed and does not include the blurred image to be processed, determining a degradation feature map corresponding to the image to be processed from a plurality of degradation feature maps; or performing feature encoding processing on any acquired reference blurred image to obtain a degradation feature map, and determining the degradation feature map as the degradation feature map corresponding to the image to be processed;

[0209] In the case that the image to be processed includes the clear image to be processed and the blurred image to be processed, feature encoding processing is performed on the blurred image to be processed to obtain a degradation feature map, and the degradation feature map is determined as the degradation feature map corresponding to the clear image to be processed.

[0210] In a possible implementation, when generating a target image based on the image to be processed and the degradation feature map, the first generating module 503 is configured to:

[0211] Performing feature decoding processing on the image to be processed and the degradation feature map to generate a target feature map corresponding to the image to be processed; wherein the target feature map includes blur pattern information that matches a feature map of at least one resolution of the image to be processed;

[0212] The target feature map is used to process the image to be processed to generate the target image.

[0213] In one possible implementation, when the image to be processed includes the blurred image to be processed and does not include the clear image to be processed, the target image is generated by a trained first target neural network; the first target neural network includes: an encoder and a first decoder; the encoder is connected to the first decoder;

[0214] In a case where the image to be processed includes the clear image to be processed and does not include the blurred image to be processed, the target image is generated by a trained second target neural network; the second target neural network includes: the encoder and a second decoder; the encoder is connected to the second decoder;

[0215] In the case where the image to be processed includes the clear image to be processed and the blurred image to be processed, the target image is generated by a trained third target neural network; the third target neural network includes: the encoder, a first decoder and a second decoder; the encoder is connected to the first decoder and the second decoder respectively.

[0216] In one possible implementation, the first decoder includes m feature extraction subnetworks, m+1 parameter modulation subnetworks, and m feature fusion subnetworks; wherein m is a positive integer;

[0217] The i-th feature extraction subnetwork includes the i-th first residual subnetwork and the i-th downsampling feature layer; the i-th feature fusion subnetwork includes the i-th upsampling feature layer, the i-th connection layer, and the i-th second residual subnetwork;

[0218] The i-1th feature extraction subnetwork is connected to the i-2th feature extraction subnetwork and the i-th feature extraction subnetwork respectively; the i-1th feature fusion subnetwork is connected to the i-2th feature fusion subnetwork and the i-th feature fusion subnetwork respectively;

[0219] The i-th parameter modulation subnetwork is connected to the i+1-th parameter modulation subnetwork, the i-th first residual subnetwork in the i-th feature extraction subnetwork, and the i-th connection layer in the i-th feature fusion subnetwork respectively; where i is a positive integer greater than or equal to 1 and less than or equal to m;

[0220] The m+1th parameter modulation sub-network is connected to the mth downsampling feature layer in the mth feature extraction sub-network and the mth upsampling feature layer in the mth feature fusion sub-network respectively.

[0221] In a possible implementation, the apparatus further includes a training module 504 configured to train the first target neural network according to the following steps:

[0222] Acquire a first training sample, wherein the first training sample includes a plurality of first sample image pairs, and the first sample image pairs include a first blurred image and a first clear image corresponding to the first blurred image;

[0223] Using an encoder in a first neural network to be trained, performing feature encoding processing on the first blurred image to generate a sample degradation feature map corresponding to the first blurred image;

[0224] Using a first decoder in the first neural network to be trained, performing feature decoding processing on the sample degradation feature map and the first blurred image to generate a first sample feature map corresponding to the first blurred image;

[0225] Performing feature fusion processing on the first sample feature map and the first blurred image to generate a predicted clear image corresponding to the first blurred image;

[0226] Based on the predicted clear image and the first clear image, the first neural network to be trained is trained until the first neural network to be trained meets a first training cutoff condition, thereby obtaining the first target neural network.

[0227] In one possible implementation, the training module 504 is further configured to train the second target neural network according to the following steps:

[0228] Acquire a second training sample, wherein the second training sample includes a plurality of second sample image pairs, and the second sample image pairs include a second blurred image and a second clear image corresponding to the second blurred image;

[0229] Using an encoder in a second neural network to be trained, performing feature encoding processing on the second blurred image to generate a sample degradation feature map corresponding to the second blurred image;

[0230] Using a second decoder in the second neural network to be trained, performing feature decoding processing on the sample degradation feature map and the second clear image to generate a second sample feature map corresponding to the second clear image;

[0231] Performing feature fusion processing on the second sample feature map and the second clear image to generate a predicted blurred image corresponding to the second clear image;

[0232] Based on the predicted blurred image and the second blurred image, the second neural network to be trained is trained until the second neural network to be trained meets a second training cutoff condition, thereby obtaining the second target neural network.

[0233] In one possible implementation, the training module 504 is further configured to train the third target neural network according to the following steps:

[0234] Acquire a third training sample, wherein the third training sample includes a plurality of third sample image pairs, and the third sample image pairs include a third blurred image and a third clear image corresponding to the third blurred image;

[0235] Using an encoder in a third neural network to be trained, performing feature encoding processing on the third blurred image to generate a sample degradation feature map corresponding to the third blurred image;

[0236] performing feature decoding processing on the sample degradation feature map and the third blurred image using the first decoder in the third neural network to be trained to generate a third sample feature map corresponding to the third blurred image; and performing feature fusion processing on the third sample feature map and the third blurred image to generate a predicted clear image corresponding to the third blurred image;

[0237] and performing feature decoding processing on the sample degradation feature map and the third clear image using the second decoder in the third neural network to be trained to generate a fourth sample feature map corresponding to the third clear image; and performing feature fusion processing on the fourth sample feature map and the third clear image to generate a predicted blurred image corresponding to the third clear image;

[0238] Based on a first predicted image pair consisting of the predicted blurred image and the third blurred image, and a second predicted image pair consisting of the predicted clear image and the third clear image, the third neural network to be trained is trained until the third neural network to be trained meets a third training cutoff condition, thereby obtaining the third target neural network.

[0239] In one possible implementation, the training module 504, when performing feature decoding processing on the sample degradation feature map and the third blurred image using the first decoder in the third neural network to be trained to generate a third sample feature map corresponding to the third blurred image, is configured to:

[0240] Using the first decoder in the third neural network to be trained, perform feature decoding processing on the third blurred image to generate a multi-level first feature map;

[0241] Determining, based on the sample degradation feature map, a degradation feature map corresponding to the first feature map at each level;

[0242] generating a second feature map corresponding to each level of the first feature map based on the multi-level first feature map and the degradation feature map corresponding to each level of the first feature map;

[0243] The second feature maps corresponding to the multi-level first feature maps are subjected to feature fusion processing to generate a third sample feature map corresponding to the third blurred image.

[0244] In one possible implementation, the training module 504, when generating the second feature map corresponding to each level of the first feature map based on the multi-level first feature map and the degraded feature map corresponding to each level of the first feature map, is configured to:

[0245] For each level of the first feature map, performing a first convolution process on the degraded feature map corresponding to the first feature map of the level to obtain a first convolution feature map corresponding to the first feature map of the level; and performing a second convolution process on the degraded feature map corresponding to the first feature map of the level to obtain a second convolution feature map corresponding to the first feature map of the level;

[0246] Performing batch normalization on the first feature map of the level to obtain the processed first feature map of the level;

[0247] The first convolution feature map, the second convolution feature map and the first feature map after processing at this level are fused to generate a second feature map corresponding to the first feature map of this level; wherein the first convolution feature map is used to adjust the variance of the processed first feature map; and the second convolution feature map is used to adjust the average value of the processed first feature map.

[0248] In a possible implementation, the apparatus further includes: a control module 505, wherein, when the image to be processed is an original image acquired by an image acquisition device, the control module 505 is configured to:

[0249] Control the image acquisition device to display the target image.

[0250] In one possible implementation, the apparatus further includes: a second generating module 506, wherein, when the image to be processed is any blurred video frame in the video data, the second generating module 506 is configured to:

[0251] Deblurred video data is generated based on the target images respectively corresponding to the video frames in the video data.

[0252] In some embodiments, the functions or templates contained in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0253] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 6 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present disclosure, including a processor 601, a memory 602, and a bus 603. Among them, the memory 602 is used to store execution instructions, including a memory 6021 and an external memory 6022; the memory 6021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 601 and the data exchanged with the external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the memory 6021. When the electronic device 600 is running, the processor 601 communicates with the memory 602 via the bus 603, so that the processor 601 executes the following instructions:

[0254] Acquire an image to be processed; the image to be processed includes a blurred image to be processed and / or a clear image to be processed;

[0255] Determining a degradation feature map corresponding to the image to be processed, wherein the degradation feature map is used to characterize an image degradation pattern of spatial feature changes between a blurred image and a clear image corresponding to the blurred image;

[0256] A target image is generated based on the image to be processed and the degradation feature map; wherein, when the image to be processed includes a blurred image to be processed, the target image includes a deblurred image corresponding to the blurred image to be processed; and when the image to be processed includes a clear image to be processed, the target image includes a blurred image corresponding to the clear image to be processed.

[0257] The specific processing flow of the processor 601 can refer to the description of the above method embodiment and will not be repeated here.

[0258] In addition, embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the image processing method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0259] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0260] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0261] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0262] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0263] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0264] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0265] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; The image to be processed includes a blurred image to be processed and / or a clear image to be processed; Determining a degradation feature map corresponding to the image to be processed, wherein the degradation feature map is used to characterize an image degradation pattern of spatial feature changes between a blurred image and a clear image corresponding to the blurred image; generating a target image based on the image to be processed and the degradation feature map; wherein, if the image to be processed includes a blurred image to be processed, the target image includes a deblurred image corresponding to the blurred image to be processed; and if the image to be processed includes a clear image to be processed, the target image includes a blurred image corresponding to the clear image to be processed; In a case where the image to be processed includes the blurred image to be processed and does not include the clear image to be processed, the target image is generated by a trained first target neural network, the first target neural network including: an encoder and a first decoder connected to the encoder; The first decoder includes m feature extraction subnetworks, m+1 parameter modulation subnetworks, and m feature fusion subnetworks; wherein m is a positive integer; The i-th feature extraction subnetwork includes the i-th first residual subnetwork and the i-th downsampling feature layer; the i-th feature fusion subnetwork includes the i-th upsampling feature layer, the i-th connection layer, and the i-th second residual subnetwork; The i-1th feature extraction subnetwork is connected to the i-2th feature extraction subnetwork and the i-th feature extraction subnetwork respectively; the i-1th feature fusion subnetwork is connected to the i-2th feature fusion subnetwork and the i-th feature fusion subnetwork respectively; The i-th parameter modulation subnetwork is connected to the i+1-th parameter modulation subnetwork, the i-th first residual subnetwork in the i-th feature extraction subnetwork, and the i-th connection layer in the i-th feature fusion subnetwork respectively; where i is a positive integer greater than or equal to 1 and less than or equal to m; The m+1th parameter modulation sub-network is connected to the mth downsampling feature layer in the mth feature extraction sub-network and the mth upsampling feature layer in the mth feature fusion sub-network respectively.

2. The method according to claim 1, characterized in that The determining of the degradation feature map corresponding to the image to be processed includes: In a case where the image to be processed includes the blurred image to be processed but does not include the clear image to be processed, performing feature coding processing on the image to be processed to obtain a degradation feature map corresponding to the image to be processed; In a case where the image to be processed includes the clear image to be processed and does not include the blurred image to be processed, determining a degradation feature map corresponding to the image to be processed from a plurality of degradation feature maps; or performing feature encoding processing on any acquired reference blurred image to obtain a degradation feature map, and determining the degradation feature map as the degradation feature map corresponding to the image to be processed; In the case that the image to be processed includes the clear image to be processed and the blurred image to be processed, feature encoding processing is performed on the blurred image to be processed to obtain a degradation feature map, and the degradation feature map is determined as the degradation feature map corresponding to the clear image to be processed.

3. The method according to claim 1, characterized in that The generating a target image based on the image to be processed and the degradation feature map includes: Performing feature decoding processing on the image to be processed and the degradation feature map to generate a target feature map corresponding to the image to be processed; wherein the target feature map includes blur pattern information that matches a feature map of at least one resolution of the image to be processed; The target feature map is used to process the image to be processed to generate the target image.

4. The method according to claim 1, wherein In a case where the image to be processed includes the clear image to be processed and does not include the blurred image to be processed, the target image is generated by a trained second target neural network; The second target neural network includes: the encoder and the second decoder; the encoder is connected to the second decoder; In the case where the image to be processed includes the clear image to be processed and the blurred image to be processed, the target image is generated by a trained third target neural network; the third target neural network includes: the encoder, a first decoder and a second decoder; the encoder is connected to the first decoder and the second decoder respectively.

5. The method according to claim 4, characterized in that The first target neural network is obtained by training according to the following steps: Acquire a first training sample, wherein the first training sample includes a plurality of first sample image pairs, and the first sample image pairs include a first blurred image and a first clear image corresponding to the first blurred image; Using an encoder in a first neural network to be trained, performing feature encoding processing on the first blurred image to generate a sample degradation feature map corresponding to the first blurred image; Using a first decoder in the first neural network to be trained, performing feature decoding processing on the sample degradation feature map and the first blurred image to generate a first sample feature map corresponding to the first blurred image; Performing feature fusion processing on the first sample feature map and the first blurred image to generate a predicted clear image corresponding to the first blurred image; Based on the predicted clear image and the first clear image, the first neural network to be trained is trained until the first neural network to be trained meets a first training cutoff condition, thereby obtaining the first target neural network.

6. The method according to claim 4, characterized in that The second target neural network is obtained by training according to the following steps: Acquire a second training sample, wherein the second training sample includes a plurality of second sample image pairs, and the second sample image pairs include a second blurred image and a second clear image corresponding to the second blurred image; Using an encoder in a second neural network to be trained, performing feature encoding processing on the second blurred image to generate a sample degradation feature map corresponding to the second blurred image; Using a second decoder in the second neural network to be trained, performing feature decoding processing on the sample degradation feature map and the second clear image to generate a second sample feature map corresponding to the second clear image; Performing feature fusion processing on the second sample feature map and the second clear image to generate a predicted blurred image corresponding to the second clear image; Based on the predicted blurred image and the second blurred image, the second neural network to be trained is trained until the second neural network to be trained meets a second training cutoff condition, thereby obtaining the second target neural network.

7. The method according to claim 4, characterized in that The third target neural network is obtained by training according to the following steps: Acquire a third training sample, wherein the third training sample includes a plurality of third sample image pairs, and the third sample image pairs include a third blurred image and a third clear image corresponding to the third blurred image; Using an encoder in a third neural network to be trained, performing feature encoding processing on the third blurred image to generate a sample degradation feature map corresponding to the third blurred image; performing feature decoding processing on the sample degradation feature map and the third blurred image using the first decoder in the third neural network to be trained to generate a third sample feature map corresponding to the third blurred image; and performing feature fusion processing on the third sample feature map and the third blurred image to generate a predicted clear image corresponding to the third blurred image; and performing feature decoding processing on the sample degradation feature map and the third clear image using the second decoder in the third neural network to be trained to generate a fourth sample feature map corresponding to the third clear image; and performing feature fusion processing on the fourth sample feature map and the third clear image to generate a predicted blurred image corresponding to the third clear image; Based on a first predicted image pair consisting of the predicted blurred image and the third blurred image, and a second predicted image pair consisting of the predicted clear image and the third clear image, the third neural network to be trained is trained until the third neural network to be trained meets a third training cutoff condition, thereby obtaining the third target neural network.

8. The method according to claim 7, characterized in that The step of utilizing the first decoder in the third neural network to be trained to perform feature decoding processing on the sample degradation feature map and the third blurred image to generate a third sample feature map corresponding to the third blurred image includes: Using the first decoder in the third neural network to be trained, perform feature decoding processing on the third blurred image to generate a multi-level first feature map; Determining, based on the sample degradation feature map, a degradation feature map corresponding to the first feature map at each level; generating a second feature map corresponding to each level of the first feature map based on the multi-level first feature map and the degradation feature map corresponding to each level of the first feature map; The second feature maps corresponding to the multi-level first feature maps are subjected to feature fusion processing to generate a third sample feature map corresponding to the third blurred image.

9. The method according to claim 8, characterized in that The generating, based on the multi-level first feature maps and the degradation feature map corresponding to each level of the first feature map, a second feature map corresponding to each level of the first feature map, includes: For each level of the first feature map, performing a first convolution process on the degraded feature map corresponding to the first feature map of the level to obtain a first convolution feature map corresponding to the first feature map of the level; and performing a second convolution process on the degraded feature map corresponding to the first feature map of the level to obtain a second convolution feature map corresponding to the first feature map of the level; Performing batch normalization on the first feature map of the level to obtain the processed first feature map of the level; The first convolution feature map, the second convolution feature map and the first feature map after processing at this level are fused to generate a second feature map corresponding to the first feature map of this level; wherein the first convolution feature map is used to adjust the variance of the processed first feature map; and the second convolution feature map is used to adjust the average value of the processed first feature map.

10. The method according to any one of claims 1 to 9, characterized in that: The image to be processed is an original image acquired by an image acquisition device. After generating the target image, the method further includes: Control the image acquisition device to display the target image.

11. The method according to any one of claims 1 to 9, characterized in that: The image to be processed is any video frame in a blurred state in the video data. After generating the target image, the method further includes: Deblurred video data is generated based on the target images respectively corresponding to the video frames in the video data.

12. An image processing device, characterized in that: include: An acquisition module, configured to acquire an image to be processed; wherein the image to be processed includes a blurred image to be processed and / or a clear image to be processed; A determination module, configured to determine a degradation feature map corresponding to the image to be processed; wherein the degradation feature map is used to characterize an image degradation pattern of spatial feature changes between a blurred image and a corresponding clear image; a first generating module, configured to generate a target image based on the image to be processed and the degradation feature map; wherein, if the image to be processed includes a blurred image to be processed, the target image includes a deblurred image corresponding to the blurred image to be processed; and if the image to be processed includes a clear image to be processed, the target image includes a blurred image corresponding to the clear image to be processed; In a case where the image to be processed includes the blurred image to be processed and does not include the clear image to be processed, the target image is generated by a trained first target neural network, the first target neural network including: an encoder and a first decoder connected to the encoder; The first decoder includes m feature extraction subnetworks, m+1 parameter modulation subnetworks, and m feature fusion subnetworks; wherein m is a positive integer; The i-th feature extraction subnetwork includes the i-th first residual subnetwork and the i-th downsampling feature layer; the i-th feature fusion subnetwork includes the i-th upsampling feature layer, the i-th connection layer, and the i-th second residual subnetwork; The i-1th feature extraction subnetwork is connected to the i-2th feature extraction subnetwork and the i-th feature extraction subnetwork respectively; the i-1th feature fusion subnetwork is connected to the i-2th feature fusion subnetwork and the i-th feature fusion subnetwork respectively; The i-th parameter modulation subnetwork is connected to the i+1-th parameter modulation subnetwork, the i-th first residual subnetwork in the i-th feature extraction subnetwork, and the i-th connection layer in the i-th feature fusion subnetwork respectively; where i is a positive integer greater than or equal to 1 and less than or equal to m; The m+1th parameter modulation sub-network is connected to the mth downsampling feature layer in the mth feature extraction sub-network and the mth upsampling feature layer in the mth feature fusion sub-network respectively.

13. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 11 are performed.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the image processing method according to any one of claims 1 to 11.

Images