Image processing method and device

Abstract

The present disclosure relates to an image processing method and device. The image processing method comprises: extracting shallow features of an image; extracting detail features of the image at a local scale based on the shallow features to obtain local dynamic features of the image, and extracting regional features of the image at a global scale based on the shallow features to obtain deformable features of the image; fusing the local dynamic features and deformable features of the image to obtain fused features; and reconstructing the fused features to obtain a reconstructed image. According to the image processing method and device disclosed in the present disclosure, the defocus blurring effect and efficiency of image processing can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and more specifically, to an image processing method and device. Background Art

[0002] In the related art, defocus blur is a common phenomenon in photography. Sometimes it is a visual effect actively created by the photographer, and sometimes it brings unnecessary image quality degradation. When shooting with a camera equipped with a large aperture lens, objects outside the depth of field often show different degrees of blur and distortion. The degree of defocus blur is related to many factors. On the one hand, between different defocus images, the difference in imaging effect and degree of defocus is directly caused by the different apertures, depths of field, optical aberrations, etc. used by the shooting cameras. On the other hand, within a single image, the difference in scene depth in different areas will also lead to different degrees of defocus. The farther the object is from the depth of field, the more serious the defocus blur. At present, the algorithm for generating fully focused images with defocused images as input has broad application prospects, such as in the fields of image refocusing and photo retouching. At the same time, fully focused images also provide good input for other computer vision tasks, such as scene parsing, depth estimation, object detection and other tasks based on semantic segmentation.

[0003] Defocused image deblurring is a classic ill-posed problem, whose goal is to optimize the input defocused image to be as close as possible to the corresponding fully focused image. In the past 20 years, a large number of researchers have conducted research on this. One type of traditional method relies on the defocus estimation result, which generally adopts a two-stage architecture. The first stage is to estimate the defocus, and the second stage is to remove the blur under the guidance of the defocus map. Figure 1 Generally, pixel-by-pixel scalars are used for representation, and deblurring algorithms generally use non-blind deconvolution methods. This type of method has two obvious defects. First, considering that the point spread function is not necessarily a pie-shaped function in reality, the defocus estimation map represented by pixel-by-pixel scalars has limited representation capabilities and cannot accurately provide sufficient information for defocusing complex scenes. Secondly, non-blind deconvolution has large space and time overheads and is not suitable for deployment in practical applications. This type of algorithm generally has weak generalization capabilities and has a certain effect on slight defocus blur, but has little effect in moderate and severe blurred scenes. Summary of the invention

[0004] An exemplary embodiment of the present disclosure is to provide an image processing method and apparatus to at least solve the image processing problems in the related art, but may not solve any of the above problems.

[0005] According to an exemplary embodiment of the present disclosure, there is provided an image processing method, comprising: extracting shallow features of an image; extracting detail features of the image at a local scale based on the shallow features to obtain local dynamic features of the image, and extracting regional features of the image at a global scale based on the shallow features to obtain deformable features of the image; fusing the local dynamic features and the deformable features of the image to obtain fused features; and reconstructing the fused features to obtain a reconstructed image.

[0006] Optionally, before extracting detail features on a local scale from an image, the image processing method further includes: predicting convolution properties of each pixel of the image based on shallow features, wherein the convolution properties of each pixel include local convolution kernel parameters of each pixel, and a deformable position offset and a sampling weight of each pixel, wherein the step of extracting detail features on a local scale from the image includes: extracting detail features on a local scale from the image based on the convolution properties of each pixel of the image, wherein the step of extracting regional features on a global scale from the image includes: extracting regional features on a global scale from the image based on the convolution properties of each pixel of the image.

[0007] Optionally, the step of predicting the convolution properties of each pixel of the image based on shallow features may include: performing multi-level decomposition of the image using a two-dimensional discrete wavelet transform to obtain multiple detail subbands of different scales of the image; performing feature extraction on the detail subband at each scale to obtain subband features of different scales; encoding the shallow features at each scale to obtain encoded features of different scales; combining the subband features of the same scale with the encoder features to obtain combined features of different scales; decoding the combined features of different scales to obtain local convolution kernel parameters, deformable position offsets and sampling weights for each pixel of the image.

[0008] Optionally, the detail subband may include structural information.

[0009] Optionally, the step of combining the sub-band features of the same scale with the encoder features may include: combining the sub-band features of the same scale with the encoder features in an iterative manner from a shallow layer to a deep layer.

[0010] Optionally, the step of extracting detail features of the image on a local scale based on the convolution properties of each pixel of the image may include: determining a local dynamic convolution kernel based on local convolution kernel parameters for each pixel of the image to obtain a local dynamic convolution kernel for each pixel of the image; and extracting detail features of the image on a local scale based on the local dynamic convolution kernel for each pixel of the image.

[0011] Optionally, the step of extracting detail features of the image at a local scale based on the dynamic convolution kernel of each pixel of the image may include: for each pixel of the image, performing a convolution operation on the dynamic convolution kernel of the pixel and the pixel.

[0012] Optionally, the step of extracting regional features of the image on a global scale based on the convolution property of each pixel of the image may include: extracting regional features of the image on a global scale based on a deformable position offset and a sampling weight of each pixel of the image.

[0013] Optionally, the step of extracting regional features of the image on a global scale based on the deformable position offset and sampling weight of each pixel of the image may include: for each pixel of the image, offsetting the position of the pixel based on the deformable position offset of the pixel, and performing a convolution operation on the pixel with the sampling weight of the offset pixel.

[0014] Optionally, before extracting detail features on a local scale from an image, the image processing method may further include: acquiring convolution properties of the image from shallow features, wherein the convolution properties include convolution kernel parameters, position offsets, and sampling weights, wherein the step of extracting detail features on a local scale from the image may include: extracting detail features on a local scale from the image based on the convolution properties of the image, wherein the step of extracting regional features on a global scale from the image may include: extracting regional features on a global scale from the image based on the convolution properties of the image.

[0015] Optionally, the step of extracting detail features of the image at a local scale based on the convolution properties of the image may include: determining a convolution kernel of the image based on convolution kernel parameters of the image; and performing a convolution operation on the convolution kernel of the image with each pixel of the image.

[0016] Optionally, the step of extracting regional features of the image on a global scale based on the convolution properties of the image may include: offsetting the position of each pixel of the image based on the position offset of the image; and performing a convolution operation on the sampling weight of the image with each pixel after the offset.

[0017] According to an exemplary embodiment of the present disclosure, there is provided an image processing apparatus, comprising: a feature extraction unit, configured to extract shallow features of an image; a feature processing unit, configured to extract detail features of the image at a local scale based on the shallow features to obtain local dynamic features of the image, and to extract regional features of the image at a global scale based on the shallow features to obtain deformable features of the image, and to fuse the local dynamic features and the deformable features of the image to obtain fused features; and an image reconstruction unit, configured to reconstruct the fused features to obtain a reconstructed image.

[0018] Optionally, the feature processing unit may include: a pixel-by-pixel prediction unit, configured to predict the convolution properties of each pixel of the image based on shallow features, wherein the convolution properties of each pixel include local convolution kernel parameters of each pixel, and a deformable position offset and sampling weight of each pixel; and an adaptive feature fusion unit, configured to extract detail features of the image on a local scale based on the convolution properties of each pixel of the image, and to extract regional features of the image on a global scale based on the convolution properties of each pixel of the image, and to fuse the local dynamic features and deformable features of the image to obtain fused features.

[0019] Optionally, the pixel-by-pixel prediction unit can be configured as: using two-dimensional discrete wavelet transform to perform multi-level decomposition of the image to obtain multiple detail subbands of different scales of the image; extracting features of the detail subband at each scale to obtain subband features of different scales; encoding shallow features at each scale to obtain encoding features of different scales; combining the subband features of the same scale with the encoder features to obtain combined features of different scales; decoding the combined features of different scales to obtain local convolution kernel parameters, deformable position offsets and sampling weights for each pixel of the image.

[0020] Optionally, the detail subband may include structural information.

[0021] Optionally, the pixel-by-pixel prediction unit may be configured to iteratively combine sub-band features of the same scale with encoder features from shallow layers to deep layers.

[0022] Optionally, the adaptive feature fusion unit may include a dynamic branch unit, which is configured to: determine a local dynamic convolution kernel based on local convolution kernel parameters for each pixel of the image, to obtain a local dynamic convolution kernel for each pixel of the image; and extract detail features of the image at a local scale based on the local dynamic convolution kernel for each pixel of the image.

[0023] Optionally, the dynamic branching unit may be configured to: for each pixel of the image, perform a convolution operation on the dynamic convolution kernel of the pixel and the pixel.

[0024] Optionally, the adaptive feature fusion unit may include a deformable branch unit configured to extract regional features of the image on a global scale based on a deformable position offset and a sampling weight of each pixel of the image.

[0025] Optionally, the deformable branch unit may be configured to: for each pixel of the image, offset the pixel position based on the deformable position offset of the pixel, and perform a convolution operation on the sampling weight of the pixel and the offset pixel.

[0026] Optionally, the feature processing unit can be configured to: obtain convolution properties of the image from shallow features, wherein the convolution properties include convolution kernel parameters, position offsets, and sampling weights; extract detail features of the image at a local scale based on the convolution properties of the image, and extract regional features of the image at a global scale based on the convolution properties of the image.

[0027] Optionally, the feature processing unit may be configured to: determine a convolution kernel of the image based on a convolution kernel parameter of the image; and perform a convolution operation on the convolution kernel of the image with each pixel of the image.

[0028] Optionally, the feature processing unit may be configured to: shift the position of each pixel of the image based on the position offset of the image; and perform a convolution operation on the sampling weight of the image with each pixel after the offset.

[0029] According to an exemplary embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement an image processing method according to an exemplary embodiment of the present disclosure.

[0030] According to an exemplary embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, enables the electronic device to perform an image processing method according to an exemplary embodiment of the present disclosure.

[0031] According to an exemplary embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction. When the computer program / instruction is executed by a processor, the image processing method according to the exemplary embodiment of the present disclosure is implemented.

[0032] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0033] Prevent loss of details in out-of-focus areas;

[0034] Improve the defocus blur effect and efficiency of image processing.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0037] Figure 1 An exemplary system architecture is shown in which exemplary embodiments of the present disclosure may be applied.

[0038] Figure 2 A flowchart of an image processing method according to an exemplary embodiment of the present disclosure is shown.

[0039] Figure 3 A flowchart showing an image processing method according to another exemplary embodiment of the present disclosure is shown.

[0040] Figure 4 The exemplary embodiment according to the present disclosure is shown Figure 3 Schematic diagram of an example structure of an image processing network for an image processing method in FIG.

[0041] Figure 5 A schematic diagram showing a two-dimensional Haar transform according to an exemplary embodiment of the present disclosure.

[0042] Figure 6 The prediction process of the pixel-by-pixel attribute prediction module according to an exemplary embodiment of the present disclosure is shown.

[0043] Figure 7 A schematic diagram showing how to obtain local dynamic features through local dynamic branches according to an exemplary embodiment of the present disclosure.

[0044] Figure 8 A schematic diagram showing a deformable feature obtained by a deformable branch according to an exemplary embodiment of the present disclosure.

[0045] Fig. 9 A block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure is shown.

[0046] Fig.10 A block diagram illustrating an image processing apparatus according to another exemplary embodiment of the present disclosure.

[0047] Fig.11 A block diagram of the adaptive feature fusion unit 103 according to an exemplary embodiment of the present disclosure is shown.

[0048] Fig.12 is a block diagram of an electronic device 1200 according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0051] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0052] In the related art, the principle of defocused blurred imaging based on a thin convex lens is as follows. The light emitted by the object on the focal plane is refracted by the lens and converges to a single point on the imaging plane, and a clear image is formed; the light emitted by the object on the defocused plane will diverge on the circular area of ​​the imaging plane to form a diffuse circle. The degree of defocus of a pixel can be measured by the diameter of the diffuse circle, and it only changes with the depth of the scene in a single image. Ideally, the diameter of the diffuse circle is 0 when and only when the object is on the focal plane, and as the distance of the defocused plane from the focal plane increases, the corresponding diameter of the diffuse circle increases, and the blurring becomes more severe. In most shooting environments, even a single scene contains objects of various depths, so the degree of defocus in a single defocused image varies in the spatial domain. In areas with a lighter degree of defocus, the detailed texture information is well maintained, and the network needs to effectively extract differentiated depth features; in areas with severe defocus, objects diffuse and overlap with each other, and the network needs to have a larger receptive field to better perceive the overall structure of the object. In the frequency domain, defocus blur generally shows the phenomenon of detail information loss, which means that there is a significant loss of high-frequency detail information. If no additional improvements are taken, some convolutional neural networks can also restore some high-frequency detail information, but the disadvantage is that the effect is not good and it is not robust enough. In order to further enhance the ability to extract, process and reconstruct high-frequency detail information, the present disclosure takes the strategy of allowing the network to explicitly use the information of the wavelet domain detail subband as an additional input as the starting point when designing the pixel-by-pixel attribute prediction module.

[0053] In the present disclosure, when removing the defocus blur phenomenon caused by the photographed object being outside the depth of field,

[0054] First, in view of the change of the point spread function scale in the spatial domain, an adaptive feature fusion strategy can be adopted to combine the dual features from the local dynamic branch and the deformable branch;

[0055] Second, to address the detail loss phenomenon in the defocused area, a pixel-by-pixel attribute prediction module based on wavelet features can be used to assign different convolution attributes to each pixel.

[0056] Next, we will refer to Figures 1 to 12 An image processing method and apparatus according to exemplary embodiments of the present disclosure are described in detail.

[0057] Figure 1 An exemplary system architecture 100 is shown in which exemplary embodiments of the present disclosure may be applied.

[0058] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. A user may use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (e.g., image processing requests), etc. Various image applications may be installed on the terminal devices 101, 102, 103, such as camera software, video recording software, etc. The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may include but are not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, and can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0059] The terminal devices 101, 102, and 103 may be equipped with image acquisition devices (e.g., cameras) to acquire image (or video) data. In practice, the smallest visual unit that constitutes a video is a frame. Each frame is a static image. Synthesizing a temporally continuous sequence of frames together forms a dynamic video.

[0060] The server 105 may be a server that provides various services, such as a background server that provides support for image applications installed on the terminal devices 101, 102, and 103. The background server may process the received image to achieve a defocusing effect.

[0061] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0062] It should be noted that the image processing method provided in the embodiments of the present disclosure is usually executed by a terminal device, but can also be executed by a server, or can also be executed by a terminal device and a server in collaboration. Accordingly, the image processing apparatus can be set in the terminal device, in the server, or in both the terminal device and the server.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers, and the present disclosure has no limitation on this.

[0064] Figure 2 A flowchart of an image processing method according to an exemplary embodiment of the present disclosure is shown.

[0065] Reference Figure 2 In step S201, shallow features of an image are extracted. Here, the image is a single defocused image, and the shallow features include texture features.

[0066] In step S202, detail features of the image are extracted on a local scale based on the shallow features to obtain local dynamic features of the image, and regional features of the image are extracted on a global scale based on the shallow features to obtain deformable features of the image.

[0067] In an exemplary embodiment of the present disclosure, before extracting detail features on a local scale from an image, the convolution properties of the image may be first obtained from shallow features. Then, when extracting detail features on a local scale from an image, the detail features on a local scale are extracted from the image based on the convolution properties of the image, and when extracting regional features on a global scale from the image, the regional features on a global scale are extracted from the image based on the convolution properties of the image. Here, the convolution properties may include convolution kernel parameters, position offsets, and sampling weights.

[0068] In an exemplary embodiment of the present disclosure, when extracting detail features of an image at a local scale based on the convolution properties of the image, the convolution kernel of the image can be first determined based on the convolution kernel parameters of the image, and then the convolution kernel of the image can be convolved with each pixel of the image.

[0069] In an exemplary embodiment of the present disclosure, when extracting regional features of an image on a global scale based on the convolution properties of the image, the position of each pixel of the image can be first offset based on the position offset of the image, and then the sampling weight of the image can be convolved with each pixel after the offset.

[0070] In step S203, the local dynamic features and deformable features of the image are fused to obtain fused features.

[0071] In an exemplary embodiment of the present disclosure, the two types of features (local dynamic features and deformable features) may be fused by splicing in the feature dimension.

[0072] In step S204, the fused features are reconstructed to obtain a reconstructed image.

[0073] In an exemplary embodiment of the present disclosure, any image reconstruction method may be used to reconstruct the fused features.

[0074] Figure 3 A flowchart showing an image processing method according to another exemplary embodiment of the present disclosure is shown. Figure 4 The exemplary embodiment according to the present disclosure is shown Figure 3 Schematic diagram of an example structure of an image processing network for an image processing method in . Figure 4 In the image processing network, there are feature extraction modules, pixel-by-pixel attribute prediction modules, adaptive feature fusion modules and reconstruction modules. Figure 4 As shown in FIG, these four modules are connected in series, and the output of the previous module is used as the input of the next module until the deblurred result is reconstructed.

[0075] Reference Figure 3 In step S301, shallow features of an image are extracted. Here, the image is a single defocused image. Specifically, the single defocused image can be input into an image processing network, and a feature extraction module extracts shallow features from the input single defocused image.

[0076] In step S302, the convolution properties of each pixel of the image are predicted based on the shallow features. Here, the convolution properties of each pixel include local convolution kernel parameters, deformable position offset and sampling weight of each pixel.

[0077] In an exemplary embodiment of the present disclosure, when predicting the convolution properties of each pixel of an image based on shallow features, the image can first be decomposed into multiple levels using a two-dimensional discrete wavelet transform to obtain multiple detail subbands of different scales of the image, and features are extracted from the detail subbands at each scale to obtain subband features of different scales (also called wavelet subband features). The shallow features are encoded at each scale to obtain encoded features of different scales, and then the subband features of the same scale are respectively combined with the encoder features to obtain combined features of different scales. Finally, the combined features of different scales are decoded to obtain local convolution kernel parameters, deformable position offsets and sampling weights for each pixel of the image.

[0078] In an exemplary embodiment of the present disclosure, the detail subband may include structural information.

[0079] In an exemplary embodiment of the present disclosure, when combining sub-band features of the same scale with encoder features, the sub-band features of the same scale may be combined with the encoder features in an iterative manner from a shallow layer to a deep layer.

[0080] exist Figure 4 In

[15] , the pixel-by-pixel attribute prediction module is based on wavelet features. The pixel-by-pixel attribute prediction module can adopt an encoder-decoder structure similar to U-Net, and its goal is to predict the convolution attributes of each pixel of the image for the adaptive feature fusion module (including the convolution kernel parameters of the local dynamic branch, and the position offset and sampling weight of the deformable branch). The pixel-by-pixel attribute prediction module uses the detail subband features of the wavelet transform (e.g., 2D Haar transform) as additional information and flexibly incorporates it into the encoder-decoder network structure.

[0081] Specifically, the pixel-by-pixel attribute prediction module uses two-dimensional Haar transform to perform multi-level decomposition on the input single defocused image to obtain detail sub-bands of different scales, which are then combined with the deep features of the encoder-decoder network. The entire combination process is carried out iteratively from shallow to deep layers.

[0082] Figure 5 FIG. 2 is a schematic diagram showing a two-dimensional Haar transform according to an exemplary embodiment of the present disclosure. Figure 5As shown in the figure, the input single defocused image is decomposed into four sub-bands after a one-scale two-dimensional Haar transform, and the brightness of the last three sub-bands is scaled to achieve a better visual effect. Each sub-band is one-fourth the size of the input single defocused image (i.e., the original image) and contains different information: an approximate sub-band represents the approximation of the original image at a low resolution scale and contains most of the brightness information; the three detail sub-bands represent the details of the original image in the horizontal, vertical, and diagonal directions, respectively, and contain sparse structural information. The decomposition process of the two-dimensional discrete wavelet transform is lossless, and the corresponding inverse transform can completely reconstruct the original image with the four sub-bands as input.

[0083] In the present disclosure, in order to avoid the phenomenon of high-frequency information loss in the frequency domain of the input defocused image, it is hoped to enhance the detail subband information in the encoder-decoder network structure to guide the network to better detect and reconstruct high-frequency textures.

[0084] Figure 6 FIG. 2 shows a prediction process of a pixel-by-pixel attribute prediction module according to an exemplary embodiment of the present disclosure. Figure 6 As shown in Figure 1, first, the pixel-by-pixel attribute prediction module decomposes the input defocused image into multi-scale sub-bands through a multi-level two-dimensional Haar wavelet transform. Then, within each scale, a shallow wavelet feature extraction network is used to convert the detail sub-bands into wavelet features, as shown in Figure 1. Figure 6 As shown in the dashed box area. Then, the wavelet features are combined with the encoder features of the corresponding scale in a splicing manner. Finally, the entire combination process is performed iteratively from the shallow layer to the deep layer.

[0085] In terms of specific implementation, the pixel-by-pixel attribute prediction module can be obtained by making the following modifications to the standard U-Net structure: a convolution layer with a step size of 2 is used instead of the pooling operation to retain more feature information; a residual module is used instead of the ordinary convolution layer in the encoder.

[0086] In step S303, detail features of the image are extracted on a local scale based on the convolution property of each pixel of the image to obtain local dynamic features of the image, and regional features of the image are extracted on a global scale based on the convolution property of each pixel of the image to obtain deformable features of the image.

[0087] In an exemplary embodiment of the present disclosure, when performing detail feature extraction on a local scale and regional feature extraction on a global scale on an image based on the convolution property of each pixel of the image, a local dynamic convolution kernel can be first determined based on local convolution kernel parameters for each pixel of the image to obtain a local dynamic convolution kernel for each pixel of the image, and then detail feature extraction on a local scale can be performed on the image based on the local dynamic convolution kernel for each pixel of the image.

[0088] In an exemplary embodiment of the present disclosure, when extracting detail features of an image at a local scale based on a dynamic convolution kernel of each pixel of the image, the dynamic convolution kernel of the pixel may be convolved with the pixel for each pixel of the image.

[0089] In an exemplary embodiment of the present disclosure, when performing global scale regional feature extraction on an image based on the convolution property of each pixel of the image, global scale regional feature extraction on the image may be performed based on the deformable position offset and sampling weight of each pixel of the image.

[0090] In an exemplary embodiment of the present disclosure, when performing regional feature extraction on a global scale for an image based on a deformable position offset and a sampling weight of each pixel of the image, the position of the pixel can be offset based on the deformable position offset of the pixel for each pixel of the image, and a convolution operation can be performed on the offset pixel and the sampling weight of the pixel.

[0091] like Figure 4 As shown in the figure, the adaptive feature fusion module includes a local dynamic branch and a deformable branch. These two branches extract features from the defocused image in parallel and output them to the next module after fusion. The two branches in the feature fusion module use local dynamic convolution and deformable convolution respectively, which increase the degree of freedom in three dimensions compared to the standard convolution layer, including convolution kernel parameters, position offset, and sampling weight, and enhance the modeling ability of defocus blur of different scales.

[0092] Figure 7 FIG. 2 is a schematic diagram showing a method of obtaining local dynamic features through local dynamic branches according to an exemplary embodiment of the present disclosure. Figure 7 As shown in the figure, the local dynamic branch uses local dynamic convolution to replace the standard convolution layer. The principle of the local dynamic branch is to extract features using adaptively generated convolution kernel parameters for each pixel (or pixel position). In the defocus blurring task of a single image, the local dynamic branch breaks through the parameter sharing limitation of the standard convolution and has two advantages. On the one hand, the local dynamic branch can adapt to the overall difference in the degree of defocus between samples. On the other hand, the local dynamic branch can adapt to the local difference in the degree of blur within a single image.

[0093] Figure 8 FIG. 2 is a schematic diagram showing a deformable feature obtained by a deformable branch according to an exemplary embodiment of the present disclosure. Figure 8As shown in the figure, when the deformable branch performs convolution on the pixel position, it does not use the fixed neighborhood grid points as sampling points, but generates the offset and weight of the sampling points relative to the current position in a position-dependent manner, which has the following three advantages. First, the receptive field of the deformable branch is larger. Second, the sampling point position of the deformable branch is more flexible. Finally, the deformable branch equipped with sampling weights can play an important role in feature screening.

[0094] The local dynamic branch and the deformable branch are complementary in function. The former provides detailed features with distinguishing properties at the local scale, while the latter provides regional features with adaptive defocus degree at the global scale. By combining the two, the image processing network can extract relatively complete texture information and defocus information from the blurred image for optimized reconstruction.

[0095] In step S304, the local dynamic features and deformable features of the image are fused to obtain fused features.

[0096] In an exemplary embodiment of the present disclosure, the two types of features (local dynamic features and deformable features) may be fused by splicing in the feature dimension.

[0097] In step S305, the fused features are reconstructed to obtain a reconstructed image.

[0098] In an exemplary embodiment of the present disclosure, any image reconstruction method may be used to reconstruct the fused features. Figure 4 The reconstruction module in adopts the same encoder-decoder network structure as the pixel-by-pixel attribute prediction module to better restore the multi-scale object information of the defocused image, which is added to the input defocused image to obtain the final deblurred result.

[0099] In addition, in an exemplary embodiment of the present disclosure, the Charbonnier loss function and the SSIM loss function can be combined to perform the following Figure 4 The image processing network shown in the figure is trained. The Charbonnier loss function is a differentiable variant of the L1 loss function, which is used to measure the pixel-by-pixel difference between the deblurring result and the full focus true value in the image space. The SSIM loss function is based on the structural similarity index and combines the measurements of multiple dimensions (brightness, contrast, and structure). Compared with the pixel-by-pixel L1 loss function, L2 loss function, Charbonnier loss function, etc., it is more in line with the subjective perception of the human eye. The overall loss function is the weighted sum of the two or includes the weighted sum of the two.

[0100] The above has been combined Figures 1 to 3 The image processing method according to the exemplary embodiment of the present disclosure is described. Figures 9 to 11An image processing apparatus and its units according to an exemplary embodiment of the present disclosure are described.

[0101] Fig. 9 A block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure is shown.

[0102] Reference Fig. 9 The image processing device 900 includes a feature extraction unit 91, a feature processing unit 92 and an image reconstruction unit 93.

[0103] The feature extraction unit 91 is configured to extract shallow features of the image.

[0104] The feature processing unit 92 is configured to extract detail features of the image at a local scale based on the shallow features to obtain local dynamic features of the image, and to extract regional features of the image at a global scale based on the shallow features to obtain deformable features of the image, and to fuse the local dynamic features and deformable features of the image to obtain fused features.

[0105] In an exemplary embodiment of the present disclosure, the feature processing unit 92 may be configured to: obtain convolution properties of an image from shallow features, wherein the convolution properties include convolution kernel parameters, position offsets, and sampling weights; extract detail features of the image at a local scale based on the convolution properties of the image, and extract regional features of the image at a global scale based on the convolution properties of the image.

[0106] In an exemplary embodiment of the present disclosure, the feature processing unit 92 may be configured to: determine a convolution kernel of the image based on a convolution kernel parameter of the image; and perform a convolution operation on the convolution kernel of the image with each pixel of the image.

[0107] In an exemplary embodiment of the present disclosure, the feature processing unit 92 may be configured to: shift the position of each pixel of the image based on the position offset of the image; and perform a convolution operation on the sampling weight of the image with each pixel after the offset.

[0108] The image reconstruction unit 93 is configured to reconstruct the fused features to obtain a reconstructed image.

[0109] Fig.10 A block diagram illustrating an image processing apparatus according to another exemplary embodiment of the present disclosure.

[0110] Reference Fig.10 The image processing device 1000 includes a feature extraction unit 101, a pixel-by-pixel prediction unit 102, an adaptive feature fusion unit 103 and an image reconstruction unit 104.

[0111] The feature extraction unit 101 is configured to extract shallow features of an image.

[0112] The pixel-by-pixel prediction unit 102 is configured to predict the convolution properties of each pixel of the image based on the shallow features, wherein the convolution properties of each pixel include local convolution kernel parameters, deformable position offsets and sampling weights of each pixel.

[0113] In an exemplary embodiment of the present disclosure, the pixel-by-pixel prediction unit is configured to: perform multi-level decomposition of an image using a two-dimensional discrete wavelet transform to obtain multiple detail subbands of different scales of the image; perform feature extraction on the detail subband at each scale to obtain subband features of different scales; encode shallow features at each scale to obtain encoding features of different scales; respectively combine the subband features of the same scale with the encoder features to obtain combined features of different scales; and decode the combined features of different scales to obtain local convolution kernel parameters, deformable position offsets, and sampling weights for each pixel of the image.

[0114] In an exemplary embodiment of the present disclosure, the detail subband includes structural information.

[0115] In an exemplary embodiment of the present disclosure, the pixel-by-pixel prediction unit is configured to iteratively combine sub-band features of the same scale with encoder features from shallow layers to deep layers.

[0116] The adaptive feature fusion unit 103 is configured to extract detail features of the image on a local scale based on the convolution property of each pixel of the image to obtain local dynamic features of the image, and to extract regional features of the image on a global scale based on the convolution property of each pixel of the image to obtain deformable features of the image, and to fuse the local dynamic features and deformable features of the image to obtain fused features.

[0117] In an exemplary embodiment of the present disclosure, the adaptive feature fusion unit 103 may include a dynamic branch unit 1031, which is configured to: determine a local dynamic convolution kernel based on local convolution kernel parameters for each pixel of the image, to obtain a local dynamic convolution kernel for each pixel of the image; and extract detail features of the image at a local scale based on the local dynamic convolution kernel for each pixel of the image.

[0118] In an exemplary embodiment of the present disclosure, the dynamic branching unit 1031 may be configured to: for each pixel of the image, perform a convolution operation on the dynamic convolution kernel of the pixel and the pixel.

[0119] In an exemplary embodiment of the present disclosure, the adaptive feature fusion unit 103 may include a deformable branch unit 1032 configured to extract regional features of the image on a global scale based on a deformable position offset and a sampling weight of each pixel of the image.

[0120] In an exemplary embodiment of the present disclosure, the deformable branch unit 1032 may be configured to: for each pixel of the image, offset the position of the pixel based on the deformable position offset of the pixel, and perform a convolution operation on the sampling weight of the pixel and the offset pixel.

[0121] The image reconstruction unit 104 is configured to reconstruct the fused features to obtain a reconstructed image.

[0122] Fig.11 FIG. 1 is a block diagram of an adaptive feature fusion unit 103 according to an exemplary embodiment of the present disclosure. Fig.11 As shown, the adaptive feature fusion unit 103 includes a dynamic branch unit 1031 and a deformable branch unit 11032.

[0123] In an exemplary embodiment of the present disclosure, the feature extraction unit 101, the pixel-by-pixel prediction unit 102, the adaptive feature fusion unit 103 and the image reconstruction unit 104 may be respectively Figure 4 The feature extraction module, pixel-by-pixel attribute prediction module, adaptive feature fusion module and reconstruction module in the embodiment are similar, or the feature extraction unit 101, the pixel-by-pixel prediction unit 102, the adaptive feature fusion unit 103 and the image reconstruction unit 104 can be respectively connected to Figure 4 The feature extraction module, pixel-by-pixel attribute prediction module, adaptive feature fusion module and reconstruction module in the embodiment perform similar functions. The dynamic branch unit 1031 and the deformable branch unit 11032 can be respectively connected to Figure 4 The local dynamic branch and the deformable branch in are similar, or the dynamic branch unit 1031 and the deformable branch unit 11032 can be respectively Figure 4 The local dynamic branch and deformable branch in perform similar functions.

[0124] In an exemplary embodiment of the present disclosure, the image processing apparatus may include Figure 4 Image processing network in .

[0125] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0126] The above has been combined Figures 9 to 11 The image processing apparatus according to the exemplary embodiment of the present disclosure is described. Fig.12 An electronic device according to an exemplary embodiment of the present disclosure is described.

[0127] Fig.12 is a block diagram of an electronic device 1200 according to an exemplary embodiment of the present disclosure.

[0128] Reference Fig.12 The electronic device 1200 includes at least one memory 1201 and at least one processor 1202, wherein the at least one memory 1201 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 1202, the image processing method according to the exemplary embodiment of the present disclosure is performed.

[0129] In an exemplary embodiment of the present disclosure, the electronic device 1200 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1200 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1200 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0130] In the electronic device 1200, the processor 1202 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0131] The processor 1202 may execute instructions or codes stored in the memory 1201, wherein the memory 1201 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0132] The memory 1201 may be integrated with the processor 1202, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1201 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1201 and the processor 1202 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1202 can read files stored in the memory.

[0133] In addition, the electronic device 1200 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1200 may be connected to each other via a bus and / or a network.

[0134] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium including instructions is also provided, such as a memory 1201 including instructions, and the instructions can be executed by a processor 1202 of the device 1200 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0135] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided. The computer program product includes a computer program / instruction. When the computer program / instruction is executed by a processor, the image processing method according to the exemplary embodiment of the present disclosure is implemented.

[0136] The above has been referred to Figures 1 to 12 The image processing method and apparatus according to the exemplary embodiments of the present disclosure are described. However, it should be understood that: Figures 9 to 11 The image processing apparatus and its units shown in the figure may be respectively configured as software, hardware, firmware or any combination of the above items to perform specific functions. Fig.12 The electronic device shown in is not limited to including the components shown above, but some components may be added or deleted as needed, and the above components may also be combined.

[0137] According to the image processing method and device disclosed in the present invention, shallow features of an image are extracted, and based on the shallow features, detail features of the image are extracted on a local scale to obtain local dynamic features of the image, and regional features of the image are extracted on a global scale to obtain deformable features of the image, and local dynamic features and deformable features of the image are fused to obtain fused features, and the fused features are reconstructed to obtain a reconstructed image, thereby improving the defocus blur effect and efficiency of image processing.

[0138] In addition, according to the image processing method and device disclosed in the present invention, an adaptive feature fusion strategy can be adopted to target the change of the point spread function scale in the spatial domain, combining the dual features from the local dynamic branch and the deformable branch, thereby focusing on the feature information of the local scale through the local dynamic branch and taking into account the feature information of the global scale through the deformable branch.

[0139] In addition, according to the image processing method and device disclosed in the present invention, to address the phenomenon of detail loss in defocused areas, a pixel-by-pixel attribute prediction module based on wavelet features can be used to assign different convolution attributes to each pixel, thereby enhancing texture restoration at different scales.

[0140] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0141] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Extract shallow features of images; Predicting a convolution property of each pixel of the image based on shallow features, wherein the convolution property of each pixel includes at least a sampling weight of each pixel; Based on the convolution property of each pixel of the image, the detailed features of the image are extracted on a local scale to obtain the local dynamic features of the image, and based on the convolution property of each pixel of the image, the regional features of the image are extracted on a global scale to obtain the deformable features of the image; The local dynamic features and deformable features of the image are fused to obtain fused features; Reconstruct the fused features to obtain the reconstructed image. The step of predicting the convolution attribute of each pixel of the image based on the shallow features includes: The image is decomposed into multiple levels using two-dimensional discrete wavelet transform to obtain multiple detail sub-bands of different scales of the image; Perform feature extraction on detail sub-bands at each scale to obtain sub-band features of different scales; Encode shallow features at each scale to obtain encoded features of different scales; Combine the sub-band features of the same scale with the encoder features to obtain combined features of different scales; The combined features of different scales are decoded to obtain the convolution properties of each pixel of the image.

2. The image processing method according to claim 1, characterized in that: The convolution properties of each pixel also include the local convolution kernel parameters of each pixel and the deformable position offset of each pixel.

3. The image processing method according to claim 1, characterized in that: The detail subband includes structural information.

4. The image processing method according to claim 1, characterized in that: The steps of combining sub-band features of the same scale with encoder features include: The sub-band features of the same scale are combined with the encoder features in an iterative manner from shallow to deep layers.

5. The image processing method according to claim 2, characterized in that: The steps of extracting detail features of an image at a local scale based on the convolution properties of each pixel of the image include: For each pixel of the image, a local dynamic convolution kernel is determined based on the local convolution kernel parameters to obtain a local dynamic convolution kernel for each pixel of the image; Based on the local dynamic convolution kernel of each pixel of the image, the detailed features of the image are extracted at the local scale.

6. The image processing method according to claim 5, characterized in that: The steps of extracting detail features of an image at a local scale based on a dynamic convolution kernel for each pixel of the image include: For each pixel of the image, a dynamic convolution kernel of the pixel is convolved with the pixel.

7. The image processing method according to claim 2, characterized in that: The steps of extracting regional features of an image on a global scale based on the convolution properties of each pixel of the image include: Regional features of the image are extracted at a global scale based on the deformable position offset and sampling weight of each pixel in the image.

8. The image processing method according to claim 7, characterized in that: The steps of extracting regional features of the image on a global scale based on the deformable position offset and sampling weight of each pixel of the image include: For each pixel of the image, the position of the pixel is offset based on the deformable position offset of the pixel, and a convolution operation is performed on the sampling weight of the pixel and the offset pixel.

9. An image processing device, characterized in that: include: A feature extraction unit configured to extract shallow features of the image; A pixel-by-pixel prediction unit is configured to predict a convolution property of each pixel of the image based on the shallow features, wherein the convolution property of each pixel includes at least a sampling weight of each pixel; an adaptive feature fusion unit configured to extract detail features of the image at a local scale based on a convolution property of each pixel of the image to obtain a local dynamic feature of the image, extract regional features of the image at a global scale based on the convolution property of each pixel of the image to obtain a deformable feature of the image, and fuse the local dynamic feature and the deformable feature of the image to obtain a fused feature; and The image reconstruction unit is configured to reconstruct the fused features to obtain a reconstructed image. Among them, the pixel-by-pixel prediction unit is configured as: The image is decomposed into multiple levels using two-dimensional discrete wavelet transform to obtain multiple detail sub-bands of different scales of the image; Perform feature extraction on detail sub-bands at each scale to obtain sub-band features of different scales; Encode shallow features at each scale to obtain encoded features of different scales; Combine the sub-band features of the same scale with the encoder features to obtain combined features of different scales; The combined features of different scales are decoded to obtain the convolution properties of each pixel of the image.

10. The image processing device according to claim 9, characterized in that: The convolution properties of each pixel also include the local convolution kernel parameters of each pixel and the deformable position offset of each pixel.

11. The image processing device according to claim 9, characterized in that: The detail subband includes structural information.

12. The image processing device according to claim 9, characterized in that: The pixel-by-pixel prediction unit is configured as: The sub-band features of the same scale are combined with the encoder features in an iterative manner from shallow to deep layers.

13. The image processing device according to claim 10, characterized in that: The adaptive feature fusion unit includes a dynamic branch unit configured as follows: For each pixel of the image, a local dynamic convolution kernel is determined based on the local convolution kernel parameters to obtain a local dynamic convolution kernel for each pixel of the image; Based on the local dynamic convolution kernel of each pixel of the image, the detailed features of the image are extracted at the local scale.

14. The image processing device according to claim 13, characterized in that: The dynamic branching unit is configured as: For each pixel of the image, a dynamic convolution kernel of the pixel is convolved with the pixel.

15. The image processing device according to claim 10, characterized in that: The adaptive feature fusion unit includes a deformable branch unit configured as follows: Regional features of the image are extracted at a global scale based on the deformable position offset and sampling weight of each pixel in the image.

16. The image processing device according to claim 15, characterized in that: The deformable branch unit is configured as: For each pixel of the image, the position of the pixel is offset based on the deformable position offset of the pixel, and a convolution operation is performed on the sampling weight of the pixel and the offset pixel.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 8.

18. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor of an electronic device, the electronic device is caused to execute the image processing method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

Images