Fusion enhancement method for enhancing robustness of blurred image
Through the DeBlurGAN-v2 algorithm, the FPN algorithm is fused with Inception-Resnet-v2 and MobileNet networks to form a dual convolution processing branch, solving the problem of increased computing resource occupation and processing time under a single deep convolution method, and achieving efficient and reliable fuzzy images Rodability and accurate judgment.
Patent Information
- Application Number
- CN202411264188.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-26
AI Technical Summary
When processing blurred images, the single depth convolution method occupies a large amount of computing resources, increases processing time, and makes it difficult to achieve efficient and robust processing in a specific environment.
The DeBlurGAN-v2 algorithm is used to fuse the FPN algorithm, combine the Inception-Resnet-v2 backbone network and MobileNet lightweight network to form a dual convolution processing branch, and high-quality or high-speed convolution processing is selected to generate discriminant losses through the global and local discrimination parts to perform images. Determination of authenticity.
It achieves balancing performance and efficiency according to demand in different environments, shortens processing time, and improves the accuracy and robustness of fuzzy images.
Smart Images

Figure CN119027344A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of image processing, and in particular to a fusion enhancement method for enhancing the robustness of a blurred image. Background Art
[0002] For the processing of blurred images, in order to better ensure the robustness of blurred images and enable better discrimination, the existing processing method is to restore the entire blurred image to the required clarity through a single deep convolution method, and then judge the authenticity of the cleaned image generated by the convolution. This processing method, due to the single deep convolution method, will occupy more computing resources in some specific environments, and will also increase the overall processing time.
[0003] Therefore, it is necessary to design a fusion enhancement method to enhance the robustness of blurred images. Summary of the invention
[0004] The object of the present invention is to provide a fusion enhancement method for enhancing the robustness of a blurred image, so as to solve the problems raised in the above background technology.
[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0006] A fusion enhancement method for enhancing the robustness of blurred images includes a generator module and a discriminator module. The generator module is composed of a DeBlurGAN-v2 algorithm fused with an FPN algorithm. The discriminator module is composed of a dual discriminant scale based on a global discriminant part and a local discriminant part of the DeBlurGAN-v2 algorithm. The method includes the following steps:
[0007] S100, the generator module uses the built-in FPN algorithm to extend and connect the Inception-Resnet-v2 backbone network and the MobileNet lightweight network;
[0008] S200, after importing the blurred image into the generator module, select the Inception-Resnet-v2 backbone network for convolution processing or the MobileNet lightweight network for convolution processing based on the needs;
[0009] S300, the image processed by the convolution in S200 is discriminated by the whole discriminant part and the local discriminant part in the DeBlurGAN-v2 algorithm, and two scales of discriminant loss are generated;
[0010] S400, combining the two-scale discrimination loss generated by S300, and performing authenticity judgment on the image generated after the convolution processing of the generator module.
[0011] According to the above technical solution, the specific method of S100 is as follows:
[0012] The main framework is composed of the DeBlurGAN-v2 algorithm, which is embedded with the FPN algorithm. The Inception-Resnet-v2 backbone network and the MobileNet lightweight network are then fused through the FPN algorithm to form two convolution processing branches, so that the blurred image processing can be selected according to needs, that is, the Inception-Resnet-v2 backbone network is selected as the high-quality convolution to generate high-quality images, and the MobileNet lightweight network is selected as the high-speed convolution to shorten the convolution processing time.
[0013] According to the above technical solution, the two convolution processing routes of S200 are as follows:
[0014] Inception-Resnet-v2 backbone network: It consists of stem, Inception-Resnet-A, Inception-Resnet-B and Inception-Resnet-C:
[0015] The stem is composed of an input image of size 299×299×3, and then three convolutions → maximum pooling → two convolutions → maximum pooling → four branches → stacking;
[0016] Inception-Resnet-A is a residual network structure, which contains four branches. One is directly output without processing, the second is processed by a 1×1 32-channel convolution, the third is processed by a 1×1 32-channel convolution and a 3×3 32-channel convolution, the fourth is processed by a 1×1 32-channel convolution, a 3×3 48-channel convolution and a 3×3 64-channel convolution, and then the processing results of the second, third and fourth are stacked and convolved once and the results of the first are added;
[0017] Inception-Resnet-B is a residual network structure, which contains three branches. One is directly output without processing, the second is processed by a 1×1 192-channel convolution, the third is processed by a 1×1 128-channel convolution, a 1×7 160-channel convolution and a 7×1 192-channel convolution, and then the results of the second and third are stacked, convolved and added to the result of the first;
[0018] Inception-Resnet-C is a residual network structure, which contains three branches. One is directly output without processing, the second is processed by a 1×1 192-channel convolution, the third is processed by a 1×1 192-channel convolution, a 1×3 224-channel convolution, and a 3×1 256-channel convolution. The results of the second and third are then stacked, convolved, and added to the results of the first.
[0019] MobileNet lightweight network: It is a separable convolutional network structure, which includes depth convolution and point convolution. Assuming that the input is composed of N×H×W×C, where N is the number of samples, H is the height, W is the width, and C is the number of channels, the following processing is performed respectively:
[0020] Deep convolution: The input is divided into C groups, and then each group is convolved with 3×3 to collect the spatial features of each channel;
[0021] Point convolution: Perform k ordinary 1×1 convolutions on the input to collect the features of each point;
[0022] The result of the depthwise convolution is combined with the result of the pointwise convolution to obtain a final result of N×H×W×k.
[0023] According to the above technical solution, the specific processing method of S300 is:
[0024] The convolutional image is judged by the whole judgment unit and the local judgment unit, resulting in two losses.
[0025] According to the above technical solution, the specific processing method of S400 is:
[0026] After normalizing the two losses generated by S300, they are both between 0 and 1:
[0027] For the generator, the closer it is to 1, the smaller it is, which means the generator has fooled the discriminator;
[0028] For the discriminator, when it is closer to 0 and smaller, the discriminator has distinguished the fake image; when it is closer to 1 and smaller, the discriminator has distinguished the real image.
[0029] According to the above technical solution, the main framework of the DeBlurGAN-v2 algorithm has a dual data set, one of which is composed of blurred images and clear images obtained in consecutive frames of a video shot by a high-frame-rate camera, and the other is composed of images generated by blurring clear images with a known or randomly generated motion blur kernel.
[0030] According to the above technical solution, the method for combining the result of the depth convolution with the result of the point convolution is as follows:
[0031] The width and height of the depth-wise convolution are padded arrays for the point-wise convolution, and the thickness of the depth-wise convolution is determined based on the number of point-wise convolutions, so as to effectively combine the two.
[0032] According to the above technical solution, the overall discrimination unit and the local discrimination unit specifically work as follows:
[0033] The input channel of all discriminant parts is the full picture feature input, which generates loss1 after discriminating the picture features;
[0034] The input channel of the local discrimination unit is the local image feature input. After discriminating the image features, it generates loss2.
[0035] According to the above technical solution, the specific method of the normalization process is:
[0036] The loss1 and loss2 obtained by S300 are imported, and then the maximum and minimum of the two are determined, and then calculated by the normalization formula, which is as follows:
[0037]
[0038] Where x is the value of the original data point, X norm It is a normalized value between 0 and 1, min is the minimum value between the two, and max is the maximum value between the two.
[0039] According to the above technical solution, the two data sets are cross-selected, that is, the selection process is effectively counted and accumulated by a counter, so that the data sets in the continuous image processing process have different sources, avoiding interference from the same source.
[0040] Compared with the prior art, the beneficial effects achieved by the present invention are:
[0041] Through the original DeBlurGAN-v2 algorithm based on the FPN algorithm, the Inception-Resnet-v2 backbone network and the MobileNet lightweight network are integrated, so that the blurred image convolution processing can select different convolution processing networks according to needs, so that performance and efficiency can be freely balanced, and then the real-time work requirements can be achieved. At the same time, the accuracy of judging the large and heterogeneous real-world blur of the subsequent generated images is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION
[0043] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0044] like Figure 1 As shown, the present invention provides a technical solution: a fusion enhancement method for enhancing the robustness of blurred images, a generator module and a discriminator module, the generator module is composed of a DeBlurGAN-v2 algorithm fused with an FPN algorithm, and the discriminator module is composed of a dual discriminant scale based on the global discriminant part and the local discriminant part of the DeBlurGAN-v2 algorithm, which contains the following steps:
[0045] S100, the generator module uses the built-in FPN algorithm to extend and connect the Inception-Resnet-v2 backbone network and the MobileNet lightweight network;
[0046] S200, after importing the blurred image into the generator module, select
[0047] The Inception-Resnet-v2 backbone network is used for convolution processing, or the MobileNet lightweight network is used for convolution processing;
[0048] S300, the image processed by the convolution in S200 is discriminated by the whole discriminant part and the local discriminant part in the DeBlurGAN-v2 algorithm, and two scales of discriminant loss are generated;
[0049] S400, combining the two-scale discrimination loss generated by S300, and performing authenticity judgment on the image generated after the convolution processing of the generator module.
[0050] Specifically, the specific methods of S100 are as follows:
[0051] The main framework is composed of the DeBlurGAN-v2 algorithm, which is embedded with the FPN algorithm. The Inception-Resnet-v2 backbone network and the MobileNet lightweight network are then fused through the FPN algorithm to form two convolution processing branches, so that the blurred image processing can be selected according to needs, that is, the Inception-Resnet-v2 backbone network is selected as the high-quality convolution to generate high-quality images, and the MobileNet lightweight network is selected as the high-speed convolution to shorten the convolution processing time.
[0052] Specifically, the two convolution processing routes of S200 are as follows:
[0053] Inception-Resnet-v2 backbone network: It consists of stem, Inception-Resnet-A, Inception-Resnet-B and Inception-Resnet-C:
[0054] The stem is composed of an input image of size 299×299×3, and then three convolutions → maximum pooling → two convolutions → maximum pooling → four branches → stacking;
[0055] Inception-Resnet-A is a residual network structure, which contains four branches. One is directly output without processing, the second is processed by a 1×1 32-channel convolution, the third is processed by a 1×1 32-channel convolution and a 3×3 32-channel convolution, the fourth is processed by a 1×1 32-channel convolution, a 3×3 48-channel convolution and a 3×3 64-channel convolution, and then the processing results of the second, third and fourth are stacked and convolved once and the results of the first are added;
[0056] Inception-Resnet-B is a residual network structure, which contains three branches. One is directly output without processing, the second is processed by a 1×1 192-channel convolution, the third is processed by a 1×1 128-channel convolution, a 1×7 160-channel convolution and a 7×1 192-channel convolution, and then the results of the second and third are stacked, convolved and added to the result of the first;
[0057] Inception-Resnet-C is a residual network structure, which contains three branches. One is directly output without processing, the second is processed by a 1×1 192-channel convolution, the third is processed by a 1×1 192-channel convolution, a 1×3 224-channel convolution, and a 3×1 256-channel convolution. The results of the second and third are then stacked, convolved, and added to the results of the first.
[0058] MobileNet lightweight network: It is a separable convolutional network structure, which includes depth convolution and point convolution. Assuming that the input is composed of N×H×W×C, where N is the number of samples, H is the height, W is the width, and C is the number of channels, the following processing is performed respectively:
[0059] Deep convolution: The input is divided into C groups, and then each group is convolved with 3×3 to collect the spatial features of each channel;
[0060] Point convolution: Perform k ordinary 1×1 convolutions on the input to collect the features of each point;
[0061] The result of the depthwise convolution is combined with the result of the pointwise convolution to obtain a final result of N×H×W×k.
[0062] Specifically, the specific processing method of S300 is:
[0063] The convolutional image is judged by the whole judgment unit and the local judgment unit, resulting in two losses.
[0064] Specifically, the specific processing method of S400 is:
[0065] After normalizing the two losses generated by S300, they are both between 0 and 1:
[0066] For the generator, the closer it is to 1, the smaller it is, which means the generator has fooled the discriminator;
[0067] For the discriminator, when it is closer to 0 and smaller, the discriminator has distinguished the fake image; when it is closer to 1 and smaller, the discriminator has distinguished the real image.
[0068] Specifically, the main framework of the DeBlurGAN-v2 algorithm has two data sets, one of which is composed of blurred images and clear images obtained in consecutive frames of a video shot by a high frame rate camera, and the other is composed of images generated by blurring clear images with a known or randomly generated motion blur kernel.
[0069] Specifically, the combination method of the result of the depth convolution and the result of the point convolution is as follows:
[0070] The width and height of the depth-wise convolution are padded arrays for the point-wise convolution, and the thickness of the depth-wise convolution is determined based on the number of point-wise convolutions, so as to effectively combine the two.
[0071] Specifically, the overall discrimination unit and the local discrimination unit work as follows:
[0072] The input channel of all discriminant parts is the full picture feature input, which generates loss1 after discriminating the picture features;
[0073] The input channel of the local discrimination unit is the local image feature input. After discriminating the image features, it generates loss2.
[0074] Specifically, the specific method of the normalization process is:
[0075] The loss1 and loss2 obtained by S300 are imported, and then the maximum and minimum of the two are determined, and then calculated by the normalization formula, which is as follows:
[0076]
[0077] Where x is the value of the original data point, X norm It is a normalized value between 0 and 1, min is the minimum value between the two, and max is the maximum value between the two.
[0078] Specifically, the two data sets are cross-selected, that is, the selection process is effectively counted and accumulated through a counter, so that the data sets in the continuous image processing process have different sources, avoiding interference from the same source.
[0079] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0080] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A fusion enhancement method for enhancing the robustness of blurred images, a generator module and a discriminator module, characterized in that: The generator module is composed of the DeBlurGAN-v2 algorithm and the FPN algorithm, and the discriminator module is composed of a dual discriminant scale based on the global discriminant part and the local discriminant part of the DeBlurGAN-v2 algorithm, which includes the following steps: S100, the generator module uses the built-in FPN algorithm to extend and connect the Inception-Resnet-v2 backbone network and the MobileNet lightweight network; S200, after importing the blurred image into the generator module, select the Inception-Resnet-v2 backbone network for convolution processing or the MobileNet lightweight network for convolution processing based on the needs; S300, the image processed by the convolution in S200 is discriminated by the whole discriminant part and the local discriminant part in the DeBlurGAN-v2 algorithm, and two scales of discriminant loss are generated; S400, combining the two-scale discrimination loss generated by S300, and performing authenticity judgment on the image generated after the convolution processing of the generator module.
2. The method for enhancing the robustness of a blurred image according to claim 1, characterized in that: The specific method of S100 is as follows: The main framework is composed of the DeBlurGAN-v2 algorithm, which is embedded with the FPN algorithm. The Inception-Resnet-v2 backbone network and the MobileNet lightweight network are then fused through the FPN algorithm to form two convolution processing branches, so that the blurred image processing can be selected according to needs, that is, the Inception-Resnet-v2 backbone network is selected as the high-quality convolution to generate high-quality images, and the MobileNet lightweight network is selected as the high-speed convolution to shorten the convolution processing time.
3. The method for enhancing the robustness of a blurred image according to claim 1, characterized in that: The two convolution processing routes of S200 are as follows: Inception-Resnet-v2 backbone network: It consists of stem, Inception-Resnet-A, Inception-Resnet-B and Inception-Resnet-C composition: The stem is composed of an input image of size 299×299×3, and then three convolutions → maximum pooling → two convolutions → maximum pooling → four branches → stacking; Inception-Resnet-A is a residual network structure, which contains four branches. One is directly output without processing, the second is processed by a 1×1 32-channel convolution, the third is processed by a 1×1 32-channel convolution and a 3×3 32-channel convolution, the fourth is processed by a 1×1 32-channel convolution, a 3×3 48-channel convolution and a 3×3 64-channel convolution, and then the processing results of the second, third and fourth are stacked and convolved once and the results of the first are added; Inception-Resnet-B is a residual network structure, which contains three branches. One is directly output without processing, the second is processed by a 1×1 192-channel convolution, the third is processed by a 1×1 128-channel convolution, a 1×7 160-channel convolution and a 7×1 192-channel convolution, and then the results of the second and third are stacked, convolved and added to the result of the first; Inception-Resnet-C is a residual network structure, which contains three branches. One is directly output without processing, the second is processed by a 1×1 192-channel convolution, the third is processed by a 1×1 192-channel convolution, a 1×3 224-channel convolution, and a 3×1 256-channel convolution. The results of the second and third are then stacked, convolved, and added to the results of the first. MobileNet lightweight network: It is a separable convolutional network structure, which includes depth convolution and point convolution. Assuming that the input is composed of N×H×W×C, where N is the number of samples, H is the height, W is the width, and C is the number of channels, the following processing is performed respectively: Deep convolution: The input is divided into C groups, and then each group is convolved with 3×3 to collect the spatial features of each channel; Point convolution: Perform k ordinary 1×1 convolutions on the input to collect the features of each point; The result of the depthwise convolution is combined with the result of the pointwise convolution to obtain a final result of N×H×W×k.
4. The method for enhancing the robustness of a blurred image according to claim 1, characterized in that: The specific processing method of S300 is: The convolutional image is judged by the whole judgment unit and the local judgment unit, resulting in two losses.
5. The method for enhancing the robustness of a blurred image according to claim 1, characterized in that: The specific processing method of S400 is: After normalizing the two losses generated by S300, they are both between 0 and 1: For the generator, the closer it is to 1, the smaller it is, which means the generator has fooled the discriminator; For the discriminator, when it is closer to 0 and smaller, the discriminator has distinguished the fake image; when it is closer to 1 and smaller, the discriminator has distinguished the real image.
6. The method for enhancing the robustness of a blurred image according to claim 2, characterized in that: The main framework of the DeBlurGAN-v2 algorithm has two data sets, one of which is composed of blurred images and clear images obtained in consecutive frames of a video shot by a high-frame-rate camera, and the other is composed of images generated by blurring clear images with a known or randomly generated motion blur kernel.
7. The method for enhancing the robustness of a blurred image according to claim 3, characterized in that: The combination method of the result of the depth convolution and the result of the point convolution is as follows: The width and height of the depth-wise convolution are padded arrays for the point-wise convolution, and the thickness of the depth-wise convolution is determined based on the number of point-wise convolutions, so as to effectively combine the two.
8. The method for enhancing the robustness of a blurred image according to claim 4, characterized in that: The overall discrimination unit and the local discrimination unit specifically work as follows: The input channel of all discriminant parts is the full picture feature input, which generates loss1 after discriminating the picture features; The input channel of the local discrimination unit is the local image feature input. After discriminating the image features, it generates loss2.
9. The method for enhancing the robustness of a blurred image according to claim 5, characterized in that: The specific method of the normalization process is: The loss1 and loss2 obtained by S300 are imported, and then the maximum and minimum of the two are determined, and then calculated by the normalization formula, which is as follows: Where x is the value of the original data point, X norm It is a normalized value between 0 and 1, min is the minimum value between the two, and max is the maximum value between the two.
10. The method for enhancing the robustness of a blurred image according to claim 6, characterized in that: The two data sets are cross-selected, that is, the selection process is effectively counted and accumulated by a counter, so that the data sets in the continuous image processing process have different sources, avoiding interference from the same source.
Citation Information
Patent Citations
Blurred image restoration method based on unsupervised generative adversarial network
CN112884671A
Blurred aerial image processing method based on WGAN
CN113538266A
Generative adversarial network image deblurring method based on nonlinear inactive network
CN117094910A
