Image deblurring method and device, electronic equipment and storage medium
By extracting local features of blurred images and edge images and performing feature fusion, the problem that the debuffering method in the prior art fails to effectively pay attention to the detailed texture and edges, and achieves a better image debuffering effect.
Patent Information
- Application Number
- CN202411996183.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-06
AI Technical Summary
The existing blind defuzzing methods based on deep learning fail to effectively focus on features such as detailed textures and edges, resulting in poor image defuzzing.
By obtaining the blurred image and the corresponding edge image, input it into the encoder to extract local features, and perform spatial dimension stitching, combining the adaptive attention fusion module for feature fusion, and finally jump connection in the decoder to achieve defuzzing.
By paying attention to the detailed texture and edge features, the image deblurring effect is significantly improved and the image detail recovery quality is improved.
Smart Images
Figure CN119941565A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an image deblurring method, device, electronic device and storage medium. Background Art
[0002] Blurring is a degradation process that will lead to the loss of image texture and edge details. Deblurring technology is divided into non-blind and blind deblurring according to whether it relies on blur kernel. Compared with non-blind deblurring methods, blind deblurring methods based on deep learning have greatly improved the deblurring effect by directly predicting the nonlinear relationship between blurred and sharpened image pairs.
[0003] However, the deblurring task refers to restoring detailed textures from degraded blurred images, while current deblurring methods based on deep learning do not focus on effective features such as detailed textures and edges, which reduces the effect of image deblurring. Summary of the invention
[0004] In view of this, the purpose of the present application is to provide an image deblurring method, device, electronic device and storage medium, which can focus on effective features such as detailed texture and edges based on a blurred image and a corresponding edge image, thereby improving the image deblurring effect.
[0005] In a first aspect, an embodiment of the present application provides a method for deblurring an image, the method comprising:
[0006] Obtain a blurred image and a corresponding edge image;
[0007] Inputting the blurred image and the edge image into corresponding encoders respectively, obtaining first blurred image local features, second blurred image local features, initial blurred image local features and initial edge image local features corresponding to each initial image region;
[0008] Splicing the initial blurred image local features and initial edge image local features corresponding to a preset number of adjacent initial image regions in spatial dimension respectively, to obtain target blurred image local features and target edge image local features corresponding to the target image region; wherein each initial blurred image local feature is spliced only once; each initial edge image local feature is spliced only once; the preset number is an integral multiple of the number of the initial image regions;
[0009] Inputting the target blurred image local features and the target edge image local features corresponding to the same target image area into an adaptive attention fusion module for feature fusion to obtain image fusion features corresponding to the target image area;
[0010] The image fusion features corresponding to the same target image area, the first blurred image local features, the second blurred image local features and the blurred local image corresponding to the target image area in the blurred image are input into the decoder for jump connection to obtain a deblurred image.
[0011] In a possible implementation, the encoder includes a first residual network unit, a first downsampling module, a second residual network unit, a second downsampling module and a third residual network unit.
[0012] In a possible implementation, the blurred image or the edge image is input into a corresponding encoder to obtain first blurred image local features, second blurred image local features, and initial blurred image local features corresponding to each initial image region, or initial edge image local features corresponding to each initial image region, including:
[0013] Inputting the blurred image or the edge image into the corresponding first residual network unit to obtain first blurred image local features or first edge image local features corresponding to each initial image region;
[0014] Inputting the first blurred image local features or the first edge image local features corresponding to each initial image region into the corresponding first downsampling module for first downsampling;
[0015] Inputting the first blurred image local features or the first edge image local features after the first downsampling corresponding to each initial image region into the corresponding second residual network unit respectively, to obtain the second blurred image local features or the second edge image local features corresponding to each initial image region;
[0016] Inputting the second blurred image local features or the second edge image local features corresponding to each initial image region into the corresponding second down-sampling module for a second down-sampling;
[0017] The second blurred image local features or the second edge image local features after the second downsampling corresponding to each initial image area are respectively input into the corresponding third residual network unit to obtain the initial blurred image local features or the initial edge image local features.
[0018] In a possible implementation, the initial blurred image local features or initial edge image local features corresponding to a preset number of adjacent initial image regions are spliced in a spatial dimension to obtain a target blurred image local feature or a target edge image local feature corresponding to a target image region, including:
[0019] The initial blurred image local features or initial edge image local features corresponding to a preset number of adjacent initial image regions are spliced in spatial dimension based on the following formula:
[0020]
[0021] Among them, K i is the target blurred image local feature or target edge image local feature corresponding to the i-th target image area, S is the initial blurred image local feature or initial edge image local feature corresponding to the initial image area, n is a preset number, m is the number of initial image areas, S ni-() is the ni-(n-1)th local feature of the initial blurred image or the local feature of the initial edge image, S ni-n is the ni-nth local feature of the initial blurred image or the local feature of the initial edge image, S ni is the ni-th local feature of the initial blurred image or the local feature of the initial edge image, For splicing operation.
[0022] In a possible implementation, the target blurred image local features and the target edge image local features corresponding to the same target image area are input into an adaptive attention fusion module for feature fusion to obtain image fusion features corresponding to the target image area, including:
[0023] For each target image area, the local feature of the target blurred image corresponding to the target image area is used as the main feature, and the local feature of the target edge image corresponding to the target image area is used as the secondary feature and substituted into the following fusion formula to obtain the image fusion feature corresponding to the target image area;
[0024] f mid =sigmoid(conv(f main ));
[0025]
[0026] Among them, f mid is the intermediate feature, sigmoid is the activation operation, conv is the convolution operation, f main The main feature, F final is the spatial attention fusion feature, f aux is a secondary feature, F SAF is the image fusion feature, FC is three fully connected operations, is the dot product operation.
[0027] In one possible implementation, the decoder includes a fourth residual network unit, a first upsampling unit, a first adaptive attention fusion unit, a fifth residual network unit, a second upsampling unit, a second adaptive attention fusion unit, a sixth residual network unit, and a third adaptive attention fusion unit.
[0028] In a possible implementation, the step of inputting the image fusion features corresponding to the same target image area, the first blurred image local features, the second blurred image local features, and the blurred local image corresponding to the target image area in the blurred image into a decoder for skip connection to obtain a deblurred image includes:
[0029] For each target image region, inputting the image fusion feature corresponding to the target image region into the fourth residual network unit and the first upsampling unit in sequence to obtain a first image feature corresponding to the target image region;
[0030] Inputting the first image feature corresponding to the target image area and the second fuzzy image local feature into the first adaptive attention fusion unit to obtain the second image feature corresponding to the target image area;
[0031] Inputting the second image feature corresponding to the target image area into the fifth residual network unit and the second upsampling unit in sequence to obtain the third image feature corresponding to the target image area;
[0032] Inputting the third image feature corresponding to the target image area and the first fuzzy image local feature into the second adaptive attention fusion unit to obtain a fourth image feature corresponding to the target image area;
[0033] Inputting the fourth image feature corresponding to the target image area into the sixth residual network unit to obtain the fifth image feature corresponding to the target image area;
[0034] The fifth image feature corresponding to the target image area and the blurred local image corresponding to the target image area in the blurred image are input into the third adaptive attention fusion unit to obtain a deblurred image corresponding to the target image area.
[0035] In a second aspect, an embodiment of the present application further provides an image deblurring device, the device comprising:
[0036] An acquisition module, used for acquiring a blurred image and a corresponding edge image;
[0037] An input module, used to input the blurred image and the edge image into corresponding encoders respectively, to obtain first blurred image local features, second blurred image local features, initial blurred image local features and initial edge image local features corresponding to each initial image region;
[0038] A splicing module, used for splicing the initial blurred image local features and initial edge image local features corresponding to a preset number of adjacent initial image regions in spatial dimensions, respectively, to obtain target blurred image local features and target edge image local features corresponding to the target image region; wherein each initial blurred image local feature is spliced only once; each initial edge image local feature is spliced only once; the preset number is an integral multiple of the number of the initial image regions;
[0039] The input module is further used to input the target blurred image local features and the target edge image local features corresponding to the same target image area into the adaptive attention fusion module for feature fusion to obtain the image fusion features corresponding to the target image area;
[0040] The input module is also used to input the image fusion features corresponding to the same target image area, the first blurred image local features, the second blurred image local features and the blurred local image corresponding to the target image area in the blurred image into the decoder for jump connection to obtain a deblurred image.
[0041] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the image deblurring method as described in any one of the first aspects.
[0042] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image deblurring method as described in any one of the first aspects are executed.
[0043] The embodiment of the present application provides an image deblurring method, device, electronic device and storage medium, the method comprising: inputting a blurred image and an edge image into a corresponding encoder respectively, obtaining a first blurred image local feature, a second blurred image local feature, an initial blurred image local feature and an initial edge image local feature corresponding to each initial image region; splicing the initial blurred image local features and initial edge image local features corresponding to a preset number of adjacent initial image regions in spatial dimension to obtain the corresponding target blurred image local features and target edge image local features; inputting the target blurred image local features and target edge image local features corresponding to the same target image region into an adaptive attention fusion module for feature fusion to obtain the corresponding image fusion features; inputting the image fusion features, the first blurred image local features, the second blurred image local features and the blurred local image corresponding to the same target image region into a decoder for jump connection to obtain a deblurred image. Through the present application, it is possible to focus on effective features such as detail texture and edge based on the blurred image and the corresponding edge image, thereby improving the effect of image deblurring. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 A flow chart of an image deblurring method provided by an embodiment of the present application is shown;
[0046] Figure 2 A flowchart of another image deblurring method provided by an embodiment of the present application is shown;
[0047] Figure 3 A schematic diagram of the structure of an image deblurring device provided in an embodiment of the present application is shown;
[0048] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0049] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0050] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0051] In order to enable those skilled in the art to use the contents of this application, the following implementation is provided in conjunction with a specific application scenario, "the field of image processing technology". For those skilled in the art, the general principles defined herein may be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the "field of image processing technology", it should be understood that this is only an exemplary embodiment.
[0052] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0053] The following is a detailed description of an image deblurring method provided in an embodiment of the present application.
[0054] Reference Figure 1 As shown, it is a flowchart of an image deblurring method provided in an embodiment of the present application. The exemplary steps of the embodiment of the present application are described below:
[0055] S101, obtaining a blurred image and a corresponding edge image.
[0056] In the implementation manner of the present application, the blurred image may be any image that needs to be deblurred; and the edge image corresponding to the blurred image is extracted using a phase consistency algorithm.
[0057] The phase consistency formula is as follows:
[0058]
[0059] Where PC(,θ) is the phase consistency value at x in direction θ, θ represents the edge direction, and θ ranges from [0, π]. W(x,θ) is the frequency propagation weighted value of the corresponding scale of the nth Fourier component at x on the blurred image (used to adjust the weights of different frequency components, usually used to suppress high-frequency noise). A n (,θ) is the magnitude of the scale corresponding to the nth Fourier component at x in the direction θ on the blurred image, is the phase value of the scale corresponding to the nth Fourier component at x in the direction θ on the blurred image; is the phase difference (indicating the difference between the phase of the nth Fourier component at x in the direction θ on the blurred image and the local average phase), T is the noise threshold (used to filter noise, only the part with energy exceeding the threshold will be counted in the result), ε is a constant (to avoid the denominator being 0), The weighted average of the local phase angles of the Fourier components when the eigenvalue at x in the direction θ on the edge image reaches the maximum value (indicating the average value of the local phase, used to calculate the phase difference).
[0060] Here, the embodiment of the present application pre-extracts an edge image of the blurred image as an edge prior to increase the focus on edge information.
[0061] S102, respectively inputting the blurred image and the edge image into corresponding encoders to obtain first blurred image local features, second blurred image local features, initial blurred image local features and initial edge image local features corresponding to each initial image region.
[0062] In the implementation mode of the present application, the blurred image is input into the corresponding encoder to obtain the first blurred image local features, the second blurred image local features, and the initial blurred image local features corresponding to each initial image region; the edge image is input into the corresponding encoder to obtain the initial edge image local features. The encoder includes a first residual network unit, a first downsampling module, a second residual network unit, a second downsampling module, and a third residual network unit.
[0063] The initial image region is a local image region in the blurred image and the edge image. The blurred local images corresponding to each initial image region in the blurred image do not overlap; the edge local images corresponding to each initial image region in the edge image do not overlap. The blurred local images corresponding to all the initial image regions in the blurred image are spliced to form a blurred image; the blurred local images corresponding to all the initial image regions in the edge image are spliced to form an edge image.
[0064] Specifically, the blurred image or edge image is input into the corresponding encoder to obtain the first blurred image local features, the second blurred image local features and the initial blurred image local features corresponding to each initial image region, or the initial edge image local features corresponding to each initial image region, including:
[0065] Step 1: Input the blurred image or the edge image into the corresponding first residual network unit to obtain the first blurred image local features or the first edge image local features corresponding to each initial image area.
[0066] In an embodiment of the present application, the blurred local images corresponding to each initial image area in the blurred image are input into the corresponding first residual network unit to obtain the first blurred image local features corresponding to each initial image area; or the edge local images corresponding to each initial image area in the edge image are input into the corresponding first residual network unit to obtain the first edge image local features corresponding to each initial image area.
[0067] Step 2: Input the first blurred image local features or the first edge image local features corresponding to each initial image region into the corresponding first down-sampling module for the first down-sampling.
[0068] In an embodiment of the present application, the first blurred image local features corresponding to each initial image area are respectively input into the corresponding first downsampling module for the first downsampling; or the edge image local features corresponding to each initial image area are respectively input into the corresponding first downsampling module for the first downsampling.
[0069] Step three: input the first blurred image local features or the first edge image local features after the first downsampling corresponding to each initial image area into the corresponding second residual network unit to obtain the second blurred image local features or the second edge image local features corresponding to each initial image area.
[0070] In an embodiment of the present application, the first blurred image local features after the first downsampling corresponding to each initial image area are respectively input into the corresponding second residual network unit to obtain the second blurred image local features corresponding to each initial image area; or the first edge image local features after the first downsampling corresponding to each initial image area are respectively input into the corresponding second residual network unit to obtain the second edge image local features corresponding to each initial image area.
[0071] Step 4: Input the second blurred image local features or the second edge image local features corresponding to each initial image region into the corresponding second down-sampling module for a second down-sampling.
[0072] In an embodiment of the present application, the second blurred image local features corresponding to each initial image area are respectively input into the corresponding second downsampling module for a second downsampling; or the second edge image local features corresponding to each initial image area are respectively input into the corresponding second downsampling module for a second downsampling.
[0073] Step 5: Input the second blurred image local features or the second edge image local features after the second downsampling corresponding to each initial image area into the corresponding third residual network unit to obtain the initial blurred image local features or the initial edge image local features.
[0074] In an embodiment of the present application, the second blurred image local features after the second downsampling corresponding to each initial image area are respectively input into the corresponding third residual network unit to obtain the initial blurred image local features corresponding to each initial image area; or the second edge image local features after the second downsampling corresponding to each initial image area are respectively input into the corresponding third residual network unit to obtain the initial edge image local features corresponding to each initial image area.
[0075] Here, the embodiment of the present application inputs the blurred image into the corresponding encoder to increase the focus on texture information, and the embodiment of the present application inputs the edge image into the corresponding encoder to increase the focus on edge information. The encoder does not deepen the network horizontally, but adopts a vertical deepening method, so the receptive field is unchanged, but richer features can be obtained.
[0076] S103, splicing the initial blurred image local features and initial edge image local features corresponding to a preset number of adjacent initial image regions in spatial dimension to obtain target blurred image local features and target edge image local features corresponding to the target image region.
[0077] In the implementation manner of the present application, each local feature of the initial blurred image is spliced only once; each local feature of the initial edge image is spliced only once; and the preset number is an integer multiple of the number of initial image areas.
[0078] Specifically, the initial blurred image local features or initial edge image local features corresponding to a preset number of adjacent initial image regions are spliced in spatial dimension based on the following formula:
[0079]
[0080] Among them, K i is the target blurred image local feature or target edge image local feature corresponding to the i-th target image region, S is the initial blurred image local feature or initial edge image local feature corresponding to the initial image region, n is the preset number, m is the number of initial image regions, S ni-() is the ni-(n-1)th initial blurred image local feature or initial edge image local feature, S ni-n is the ni-nth initial blurred image local feature or initial edge image local feature, S ni is the ni-th initial blurred image local feature or initial edge image local feature, For splicing operation.
[0081] S104, inputting the target blurred image local features and the target edge image local features corresponding to the same target image area into the adaptive attention fusion module for feature fusion, and obtaining the image fusion features corresponding to the target image area.
[0082] In the implementation of the present application, since the phase consistency edge extraction algorithm is sensitive to noise, the extracted edge image will have some noise, so it is necessary to suppress the noise. The essence of the attention mechanism is the process of weighting a certain target. The weight of important information is larger, and the weight of useless information is smaller, thereby suppressing useless information. Therefore, the present application adopts an adaptive attention fusion module to fuse the local features of the target blurred image and the local features of the target edge image.
[0083] Specifically, for each target image area, the local feature of the target blurred image corresponding to the target image area is used as the main feature, and the local feature of the target edge image corresponding to the target image area is used as the secondary feature and substituted into the following fusion formula to obtain the image fusion feature corresponding to the target image area;
[0084] f mid =sigmoid(conv(f main ));
[0085]
[0086] Among them, f mid is the intermediate feature, sigmoid is the activation operation, conv is the convolution operation, f main The main feature, F final is the spatial attention fusion feature, f aux is a secondary feature, F SAF is the image fusion feature, FC is three fully connected operations, is the dot product operation.
[0087] S105, inputting the image fusion features corresponding to the same target image area, the first blurred image local features, the second blurred image local features and the blurred local image corresponding to the target image area in the blurred image into the decoder for jump connection to obtain a deblurred image.
[0088] In an embodiment of the present application, the decoder includes a fourth residual network unit, a first upsampling unit, a first adaptive attention fusion unit, a fifth residual network unit, a second upsampling unit, a second adaptive attention fusion unit, a sixth residual network unit, and a third adaptive attention fusion unit.
[0089] Specifically, refer to Figure 2 As shown, it is a schematic diagram of the process of jump connection provided in the embodiment of the present application. The exemplary steps of the embodiment of the present application are described below:
[0090] S201. For each target image area, the image fusion feature corresponding to the target image area is sequentially input into the fourth residual network unit and the first upsampling unit to obtain the first image feature corresponding to the target image area.
[0091] In an embodiment of the present application, the image fusion features corresponding to the target image area are input into the fourth residual network unit, and the image features output by the fourth residual network unit are input into the first upsampling unit to obtain the first image features corresponding to the target image area.
[0092] S202. Input the first image features corresponding to the target image area and the second fuzzy image local features into the first adaptive attention fusion unit to obtain the second image features corresponding to the target image area.
[0093] In an embodiment of the present application, the image feature obtained by splicing all the second blurred image local features corresponding to the target image area is used as the main feature, and the first image feature corresponding to the target image area is substituted into the aforementioned fusion formula as the secondary feature to obtain the second image feature corresponding to the target image area.
[0094] For example, if the target image area is the area obtained by splicing the first initial image area and the second initial image area, the image features obtained by splicing the second blurred image local features corresponding to the first initial image area and the second blurred image local features corresponding to the second initial image area are taken as the main features.
[0095] S203. Input the second image feature corresponding to the target image area into the fifth residual network unit and the second upsampling unit in sequence to obtain the third image feature corresponding to the target image area.
[0096] In an embodiment of the present application, the second image feature corresponding to the target image area is input into the fifth residual network unit, and the image feature output by the fifth residual network unit is input into the second upsampling unit to obtain the third image feature corresponding to the target image area.
[0097] S204. Input the third image feature corresponding to the target image area and the first fuzzy image local feature into the second adaptive attention fusion unit to obtain the fourth image feature corresponding to the target image area.
[0098] In an embodiment of the present application, the image feature obtained by splicing all the first blurred image local features corresponding to the target image area is used as the main feature, and the third image feature corresponding to the target image area is substituted into the aforementioned fusion formula as the secondary feature to obtain the fourth image feature corresponding to the target image area.
[0099] S205. Input the fourth image feature corresponding to the target image area into the sixth residual network unit to obtain the fifth image feature corresponding to the target image area.
[0100] S206. Input the fifth image feature corresponding to the target image area and the blurred local image corresponding to the target image area in the blurred image into the third adaptive attention fusion unit to obtain a deblurred image corresponding to the target image area.
[0101] In an implementation manner of the present application, the blurred local image corresponding to the target image area in the blurred image corresponding to the target image area is taken as the main feature, and the fifth image feature corresponding to the target image area is substituted into the aforementioned fusion formula as the secondary feature to obtain the deblurred image corresponding to the target image area.
[0102] Here, in the embodiment of the present application, a jump connection method is used to integrate the shallow feature information in each layer of the encoder (the first blurred image local feature, the second blurred image local feature, and the blurred local image corresponding to the target image area in the blurred image) into the deep information in the decoder (the first image feature, the third image feature, and the fifth image feature), so that the restored sharp features contain not only high-frequency information, but also low-frequency information, and the sharpened image quality is higher (that is, the deblurred image quality is higher).
[0103] The embodiment of the present application provides a method for deblurring an image, the method comprising: inputting a blurred image and an edge image into a corresponding encoder respectively, obtaining a first blurred image local feature, a second blurred image local feature, an initial blurred image local feature and an initial edge image local feature corresponding to each initial image region; splicing the initial blurred image local features and initial edge image local features corresponding to a preset number of adjacent initial image regions in spatial dimension to obtain the corresponding target blurred image local features and target edge image local features; inputting the target blurred image local features and target edge image local features corresponding to the same target image region into an adaptive attention fusion module for feature fusion to obtain the corresponding image fusion features; inputting the image fusion features, the first blurred image local features, the second blurred image local features and the blurred local image corresponding to the same target image region into a decoder for jump connection to obtain a deblurred image. Through the present application, it is possible to focus on effective features such as detail texture and edge based on the blurred image and the corresponding edge image, thereby improving the effect of image deblurring.
[0104] Based on the same inventive concept, an image deblurring device corresponding to the image deblurring method is also provided in an embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned image deblurring method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0105] Reference Figure 3 FIG. 1 is a schematic diagram of a structure of an image deblurring device provided in an embodiment of the present application, wherein the image deblurring device comprises:
[0106] An acquisition module 301 is used to acquire a blurred image and a corresponding edge image;
[0107] An input module 302 is used to input the blurred image and the edge image into corresponding encoders respectively, to obtain first blurred image local features, second blurred image local features, initial blurred image local features and initial edge image local features corresponding to each initial image region;
[0108] The splicing module 303 is used to splice the initial blurred image local features and initial edge image local features corresponding to a preset number of adjacent initial image regions in the spatial dimension to obtain the target blurred image local features and target edge image local features corresponding to the target image region; wherein each initial blurred image local feature is spliced only once; each initial edge image local feature is spliced only once; the preset number is an integer multiple of the number of the initial image regions;
[0109] The input module 302 is further used to input the target blurred image local features and the target edge image local features corresponding to the same target image area into the adaptive attention fusion module for feature fusion to obtain the image fusion features corresponding to the target image area;
[0110] The input module 302 is also used to input the image fusion features corresponding to the same target image area, the first blurred image local features, the second blurred image local features and the blurred local image corresponding to the target image area in the blurred image into the decoder for jump connection to obtain a deblurred image.
[0111] In a possible implementation, the encoder includes a first residual network unit, a first downsampling module, a second residual network unit, a second downsampling module and a third residual network unit.
[0112] In a possible implementation, the input module 302 is specifically used to input the blurred image or the edge image into the corresponding first residual network unit to obtain the first blurred image local features or the first edge image local features corresponding to each initial image area; input the first blurred image local features or the first edge image local features corresponding to each initial image area into the corresponding first downsampling module for the first downsampling; input the first blurred image local features or the first edge image local features corresponding to each initial image area after the first downsampling into the corresponding second residual network unit to obtain the second blurred image local features or the second edge image local features corresponding to each initial image area; input the second blurred image local features or the second edge image local features corresponding to each initial image area into the corresponding second downsampling module for the second downsampling; input the second blurred image local features or the second edge image local features corresponding to each initial image area after the second downsampling into the corresponding third residual network unit to obtain the initial blurred image local features or the initial edge image local features.
[0113] In a possible implementation, the stitching module 303 is specifically configured to stitch the initial blurred image local features or initial edge image local features corresponding to a preset number of adjacent initial image regions in a spatial dimension based on the following formula:
[0114]
[0115] Among them, K i is the target blurred image local feature or target edge image local feature corresponding to the i-th target image area, S is the initial blurred image local feature or initial edge image local feature corresponding to the initial image area, n is a preset number, m is the number of initial image areas, S ni-() is the ni-(n-1)th local feature of the initial blurred image or the local feature of the initial edge image, S ni-n is the ni-nth local feature of the initial blurred image or the local feature of the initial edge image, S ni is the ni-th local feature of the initial blurred image or the local feature of the initial edge image, For splicing operation.
[0116] In a possible implementation, the input module 302 is specifically used to substitute, for each target image region, a local feature of a target blurred image corresponding to the target image region as a primary feature and a local feature of a target edge image corresponding to the target image region as a secondary feature into the following fusion formula to obtain an image fusion feature corresponding to the target image region;
[0117] f mid =sigmoid(conv(f main ));
[0118]
[0119] Among them, f mid is the intermediate feature, sigmoid is the activation operation, conv is the convolution operation, f main The main feature, F final is the spatial attention fusion feature, f aux is a secondary feature, F SAF is the image fusion feature, FC is three fully connected operations, is the dot product operation.
[0120] In one possible implementation, the decoder includes a fourth residual network unit, a first upsampling unit, a first adaptive attention fusion unit, a fifth residual network unit, a second upsampling unit, a second adaptive attention fusion unit, a sixth residual network unit, and a third adaptive attention fusion unit.
[0121] In a possible implementation, the input module 302 is specifically used to, for each target image area, input the image fusion feature corresponding to the target image area into the fourth residual network unit and the first upsampling unit in sequence to obtain the first image feature corresponding to the target image area; input the first image feature corresponding to the target image area and the second blurred image local feature into the first adaptive attention fusion unit to obtain the second image feature corresponding to the target image area; input the second image feature corresponding to the target image area into the fifth residual network unit and the second upsampling unit in sequence to obtain the third image feature corresponding to the target image area; input the third image feature corresponding to the target image area and the first blurred image local feature into the second adaptive attention fusion unit to obtain the fourth image feature corresponding to the target image area; input the fourth image feature corresponding to the target image area into the sixth residual network unit to obtain the fifth image feature corresponding to the target image area; input the fifth image feature corresponding to the target image area and the blurred local image corresponding to the target image area in the blurred image into the third adaptive attention fusion unit to obtain the deblurred image corresponding to the target image area.
[0122] The embodiment of the present application provides an image deblurring device, through which effective features such as detail texture and edge can be focused on based on a blurred image and a corresponding edge image, thereby improving the image deblurring effect.
[0123] like Figure 4 As shown, an electronic device 400 provided in an embodiment of the present application includes: a processor 401, a memory 402 and a bus, wherein the memory 402 stores machine-readable instructions executable by the processor 401. When the electronic device is running, the processor 401 communicates with the memory 402 through the bus, and the processor 401 executes the machine-readable instructions to perform the steps of the image deblurring method as described above.
[0124] Specifically, the memory 402 and the processor 401 can be general-purpose memories and processors, which are not specifically limited here. When the processor 401 runs the computer program stored in the memory 402, the image deblurring method can be executed.
[0125] Corresponding to the above-mentioned image deblurring method, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned image deblurring method are executed.
[0126] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0127] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0128] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0129] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the information processing method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0130] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for deblurring an image, characterized in that: The method comprises: Obtain a blurred image and a corresponding edge image; Inputting the blurred image and the edge image into corresponding encoders respectively, obtaining first blurred image local features, second blurred image local features, initial blurred image local features and initial edge image local features corresponding to each initial image region; Splicing the initial blurred image local features and initial edge image local features corresponding to a preset number of adjacent initial image regions in spatial dimension respectively, to obtain target blurred image local features and target edge image local features corresponding to the target image region; wherein each initial blurred image local feature is spliced only once; each initial edge image local feature is spliced only once; the preset number is an integral multiple of the number of the initial image regions; Inputting the target blurred image local features and the target edge image local features corresponding to the same target image area into an adaptive attention fusion module for feature fusion to obtain image fusion features corresponding to the target image area; The image fusion features corresponding to the same target image area, the first blurred image local features, the second blurred image local features and the blurred local image corresponding to the target image area in the blurred image are input into the decoder for jump connection to obtain a deblurred image.
2. The image deblurring method according to claim 1, characterized in that: The encoder includes a first residual network unit, a first down-sampling module, a second residual network unit, a second down-sampling module and a third residual network unit.
3. The image deblurring method according to claim 2, characterized in that: Inputting the blurred image or the edge image into a corresponding encoder to obtain first blurred image local features, second blurred image local features and initial blurred image local features corresponding to each initial image region, or initial edge image local features corresponding to each initial image region, including: Inputting the blurred image or the edge image into the corresponding first residual network unit to obtain first blurred image local features or first edge image local features corresponding to each initial image region; Inputting the first blurred image local features or the first edge image local features corresponding to each initial image region into the corresponding first downsampling module for first downsampling; Inputting the first blurred image local features or the first edge image local features after the first downsampling corresponding to each initial image region into the corresponding second residual network unit respectively, to obtain the second blurred image local features or the second edge image local features corresponding to each initial image region; Inputting the second blurred image local features or the second edge image local features corresponding to each initial image region into the corresponding second down-sampling module for a second down-sampling; The second blurred image local features or the second edge image local features after the second downsampling corresponding to each initial image area are respectively input into the corresponding third residual network unit to obtain the initial blurred image local features or the initial edge image local features.
4. The image deblurring method according to claim 1, characterized in that: The initial blurred image local features or initial edge image local features corresponding to a preset number of adjacent initial image regions are spliced in spatial dimension to obtain a target blurred image local feature or a target edge image local feature corresponding to a target image region, including: The initial blurred image local features or initial edge image local features corresponding to a preset number of adjacent initial image regions are spliced in spatial dimension based on the following formula: Among them, K i is the target blurred image local feature or the target edge image local feature corresponding to the i-th target image area, S is the initial blurred image local feature or the initial edge image local feature corresponding to the initial image area, n is a preset number, m is the number of initial image areas, S ni-(n-1) is the ni-(n-1)th local feature of the initial blurred image or the local feature of the initial edge image, S ni-n is the ni-nth local feature of the initial blurred image or the local feature of the initial edge image, S ni is the ni-th local feature of the initial blurred image or the local feature of the initial edge image, For splicing operation.
5. The image deblurring method according to claim 1, characterized in that: Inputting the target blurred image local features and the target edge image local features corresponding to the same target image area into the adaptive attention fusion module for feature fusion to obtain the image fusion features corresponding to the target image area, including: For each target image area, the local feature of the target blurred image corresponding to the target image area is used as the main feature, and the local feature of the target edge image corresponding to the target image area is used as the secondary feature and substituted into the following fusion formula to obtain the image fusion feature corresponding to the target image area; f mid =sigmoid(conv(f main )); Among them, f mid is the intermediate feature, sigmoid is the activation operation, conv is the convolution operation, f main The main feature, F final is the spatial attention fusion feature, f aux is a secondary feature, F SAF is the image fusion feature, FC is three fully connected operations, is the dot product operation.
6. The image deblurring method according to claim 1, characterized in that: The decoder includes a fourth residual network unit, a first upsampling unit, a first adaptive attention fusion unit, a fifth residual network unit, a second upsampling unit, a second adaptive attention fusion unit, a sixth residual network unit, and a third adaptive attention fusion unit.
7. The image deblurring method according to claim 6, characterized in that: The step of inputting the image fusion features corresponding to the same target image area, the first blurred image local features, the second blurred image local features, and the blurred local image corresponding to the target image area in the blurred image into a decoder for skip connection to obtain a deblurred image, comprising: For each target image region, inputting the image fusion feature corresponding to the target image region into the fourth residual network unit and the first upsampling unit in sequence to obtain a first image feature corresponding to the target image region; Inputting the first image feature corresponding to the target image area and the second fuzzy image local feature into the first adaptive attention fusion unit to obtain the second image feature corresponding to the target image area; Inputting the second image feature corresponding to the target image area into the fifth residual network unit and the second upsampling unit in sequence to obtain the third image feature corresponding to the target image area; Inputting the third image feature corresponding to the target image area and the first fuzzy image local feature into the second adaptive attention fusion unit to obtain a fourth image feature corresponding to the target image area; Inputting the fourth image feature corresponding to the target image area into the sixth residual network unit to obtain the fifth image feature corresponding to the target image area; The fifth image feature corresponding to the target image area and the blurred local image corresponding to the target image area in the blurred image are input into the third adaptive attention fusion unit to obtain a deblurred image corresponding to the target image area.
8. An image deblurring device, characterized in that: The device comprises: An acquisition module, used for acquiring a blurred image and a corresponding edge image; An input module, used to input the blurred image and the edge image into corresponding encoders respectively, to obtain first blurred image local features, second blurred image local features, initial blurred image local features and initial edge image local features corresponding to each initial image region; A splicing module, used for splicing the initial blurred image local features and initial edge image local features corresponding to a preset number of adjacent initial image regions in spatial dimensions, respectively, to obtain target blurred image local features and target edge image local features corresponding to the target image region; wherein each initial blurred image local feature is spliced only once; each initial edge image local feature is spliced only once; the preset number is an integral multiple of the number of the initial image regions; The input module is further used to input the target blurred image local features and the target edge image local features corresponding to the same target image area into the adaptive attention fusion module for feature fusion to obtain the image fusion features corresponding to the target image area; The input module is also used to input the image fusion features corresponding to the same target image area, the first blurred image local features, the second blurred image local features and the blurred local image corresponding to the target image area in the blurred image into the decoder for jump connection to obtain a deblurred image.
9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the image deblurring method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image deblurring method according to any one of claims 1 to 7 are executed.