Image deblurring model training method, image deblurring processing method, device, equipment and storage medium

Abstract

The invention discloses an image deblurring model training method and an image deblurring processing method, and the method comprises the steps: generating a degraded image corresponding to an original image through an image degradation method based on the original image; wherein the definition of the degraded image is lower than that of the original image; generating a training data set based on the original image and the degraded image; training the initial model based on the training data set to obtain an image deblurring model; wherein the initial model comprises a channel space attention module and a first re-parameterization module, and the image deblurring model comprises the channel space attention module and a second re-parameterization module corresponding to the first re-parameterization module. Acquiring a to-be-processed image; obtaining a processed image corresponding to the to-be-processed image based on the to-be-processed image through the image deblurring model; the image deblurring model is obtained based on training of a training data set, and the training data set is generated through an image degradation method; the definition of the processed image is higher than that of the to-be-processed image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a training method for an image deblurring model, an image deblurring processing method, a device, a equipment and a storage medium. Background Art

[0002] In scenarios such as QR code scanning and long-distance text scanning, applications in smartphones need to use images in the camera preview stream. When such functions are used at a long distance, the camera preview stream image will be blurred and deformed due to the long-distance zoom, which will cause the relevant functions of the application to be unable to correctly parse the content in the image.

[0003] However, traditional image deblurring technology cannot be well applied in smart terminals due to its high computational complexity. Summary of the invention

[0004] The embodiments of the present application provide a training method for an image deblurring model, a processing method, device, equipment and storage medium for image deblurring, which can solve the problem of image blur and improve the intelligence of terminal equipment.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for training an image deblurring model, the method comprising:

[0007] Based on the original image, a degraded image corresponding to the original image is generated by an image degradation method; wherein the definition of the degraded image is lower than that of the original image;

[0008] Generate a training dataset based on the original image and the degraded image;

[0009] An initial model is trained based on a training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first parameterization module, and the image deblurring model includes a channel space attention module and a second parameterization module corresponding to the first parameterization module.

[0010] In a second aspect, an embodiment of the present application provides a method for image deblurring, the method comprising:

[0011] Get the image to be processed;

[0012] Through the image deblurring model, based on the image to be processed, a processed image corresponding to the image to be processed is obtained; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training based on a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed.

[0013] In a third aspect, an embodiment of the present application provides a training device for an image deblurring model, and the training device for an image deblurring model includes:

[0014] A generating unit, configured to generate a degraded image corresponding to the original image by an image degradation method based on the original image; wherein the definition of the degraded image is lower than that of the original image; and to generate a training data set based on the original image and the degraded image;

[0015] A training unit is used to train an initial model based on a training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first parameterization module, and the image deblurring model includes a channel space attention module and a second parameterization module corresponding to the first parameterization module.

[0016] In a fourth aspect, an embodiment of the present application provides an image deblurring processing device, the image deblurring processing device comprising:

[0017] An acquisition unit is used to acquire an image to be processed; based on the image to be processed, a processed image corresponding to the image to be processed is obtained through an image deblurring model; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training based on a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed.

[0018] In a fifth aspect, an embodiment of the present application provides a computer device, which includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method of the first aspect or the second aspect is implemented.

[0019] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method of the first aspect or the second aspect described above is implemented.

[0020] The embodiment of the present application provides a training method for an image deblurring model, a processing method, device, equipment and storage medium for image deblurring. The present application discloses a training method for an image deblurring model and a processing method for image deblurring. Based on an original image, a degraded image corresponding to the original image is generated by an image degradation method; wherein the clarity of the degraded image is lower than that of the original image; a training data set is generated based on the original image and the degraded image; an initial model is trained based on the training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first parameterization module, and the image deblurring model includes a channel space attention module and a second parameterization module corresponding to the first parameterization module. Obtain an image to be processed; obtain a processed image corresponding to the image to be processed based on the image to be processed through the image deblurring model; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training the training data set, and the training data set is generated by the image degradation method; the clarity of the processed image is higher than that of the image to be processed. That is to say, in an embodiment of the present application, a training data set can be generated by a multi-degree image degradation method for model training, so that the obtained image deblurring model has excellent representation capabilities; the image deblurring model includes a channel space attention module and a reparameterization module, and the channel space attention mechanism and the reparameterization method can be used to further improve the representation capabilities of the model; by performing image deblurring processing through the image deblurring model, a clearer image can be obtained, the problem of image blur can be solved, and the intelligence of the terminal device can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the implementation flow of the training method for the image deblurring model proposed in the embodiment of the present application;

[0022] Figure 2 A schematic diagram of an initial model proposed for an embodiment of the present application;

[0023] Figure 3 A schematic diagram of an initial model proposed for an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the application of the channel space attention module proposed in the embodiment of the present application;

[0025] Figure 5 A schematic diagram of an initial model proposed for an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a training method for an initial module proposed in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of an image deblurring model proposed in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the process flow of implementing the image deblurring processing method proposed in the embodiment of the present application;

[0029] Fig. 9 A schematic diagram of an image deblurring model proposed in an embodiment of the present application;

[0030] Fig.10 This is a schematic diagram of the application of the image deblurring model proposed in the embodiment of the present application;

[0031] Fig.11 A schematic diagram of the structure of a training device for an image deblurring model proposed in an embodiment of the present application;

[0032] Fig.12 A schematic diagram of the composition structure of the image deblurring module processing device proposed in an embodiment of the present application;

[0033] Fig.13 A schematic diagram of the composition structure of the computer device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the difference application, rather than to limit the application. It should also be noted that, for the convenience of description, only the parts that are different from the related applications are shown in the drawings.

[0035] In daily life, we often need to scan and identify QR codes to perform corresponding processing. Among them, scanning QR codes usually requires using a smart device with a camera function, such as a smartphone or tablet, and then starting an application that can read QR codes and turning on the QR code scanning mode. After using the camera function to aim at the QR code, the application can read the QR code. If the QR code is valid and the application supports the type of the QR code, it will display relevant information or perform corresponding operations, such as opening a web page, adding a contact, downloading an application, etc. .

[0036] When using a mobile phone to perform long-distance scanning of codes, images, text, etc., the camera preview stream image will become blurred and deformed due to the long-distance zoom, which will cause the relevant functions of the application to be unable to correctly parse the content in the image.

[0037] For example, Figure 1 When using a mobile phone to scan a QR code from a distance, the image will be blurred due to zooming and deformed like a "tadpole", resulting in the failure of QR code decoding. Similar scenarios include long-distance text recognition and chart information extraction.

[0038] Scenarios that use camera preview streams to capture images usually require extremely fast response speeds and low power consumption, so some traditional image deblurring techniques become unusable due to high computational complexity. How to solve the blur caused by long-distance zooming through automated methods has become a key technical challenge in solving this problem.

[0039] In order to solve the above problems, the embodiment of the present application provides a training method for an image deblurring model, a processing method, device, equipment and storage medium for image deblurring. The present application discloses a training method for an image deblurring model and a processing method for image deblurring. Based on an original image, a degraded image corresponding to the original image is generated by an image degradation method; wherein the clarity of the degraded image is lower than that of the original image; a training data set is generated based on the original image and the degraded image; an initial model is trained based on the training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first parameterization module, and the image deblurring model includes a channel space attention module and a second parameterization module corresponding to the first parameterization module. An image to be processed is obtained; based on the image to be processed, a processed image corresponding to the image to be processed is obtained by the image deblurring model; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training based on the training data set, and the training data set is generated by the image degradation method; the clarity of the processed image is higher than that of the image to be processed. That is to say, in an embodiment of the present application, a training data set can be generated by a multi-degree image degradation method for model training, so that the obtained image deblurring model has excellent representation capabilities; the image deblurring model includes a channel space attention module and a reparameterization module, and the channel space attention mechanism and the reparameterization method can be used to further improve the representation capabilities of the model; by performing image deblurring processing through the image deblurring model, a clearer image can be obtained, the problem of image blur can be solved, and the intelligence of the terminal device can be improved.

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0041] An embodiment of the present application provides a method for training an image deblurring model. The method for training an image deblurring model can be applied to a training device or electronic device for an image deblurring model, and can also be applied to any terminal including a training device or electronic device for an image deblurring model.

[0042] Below, taking the training device of the image deblurring model as an example, the training method of the image deblurring model proposed in the embodiment of the present application is exemplarily described.

[0043] Furthermore, in the embodiments of the present application, Figure 1 The following is a flow chart of the training method for the image deblurring model proposed in the embodiment of the present application. Figure 1 As shown, the training method of the image deblurring model may include the following steps:

[0044] Step 101: Based on the original image, a degraded image corresponding to the original image is generated by an image degradation method; wherein the definition of the degraded image is lower than that of the original image.

[0045] In an embodiment of the present application, the first device may generate a degraded image corresponding to the original image by an image degradation method based on the original image.

[0046] In the embodiments of the present application, the original image may be one or more clear images acquired by any means.

[0047] Exemplarily, in some embodiments, the method of acquiring the original image includes but is not limited to one or more of the following: acquiring through a configured shooting module, receiving from other electronic devices, acquiring from a network, etc.

[0048] Furthermore, in the embodiments of the application, it is possible to choose not to restrict the image content of the original image, or to choose to restrict the image content of the original image based on the application scenario of the image deblurring model.

[0049] Exemplarily, in some embodiments, assuming that an image deblurring model including application scenarios such as QR code recognition needs to be trained, an image whose image content includes a QR code may be selected as the original image.

[0050] It is understood that in the embodiments of the present application, the number of original images is not specifically limited. The number of original images acquired based on different image acquisition methods, and the number of original images of each type of image content for different image contents are not specifically limited in the present application.

[0051] For example, in some embodiments, when acquiring the original image, you can choose to collect 10,000 clear, unzoomed images from the camera preview stream. Image types include QR code images, common chart images (flow charts, mind maps, pie charts, bar charts, architecture diagrams, line charts), document images, etc. Different models of mobile phones are used during the collection process to increase the diversity of images.

[0052] In the embodiments of the present application, the image degradation method can be understood as obtaining a blurred degraded image corresponding to a clear original image by adding noise, blurring, etc. to the original image. That is to say, in comparison, the clarity of the original image is higher than the clarity of the degraded image.

[0053] Furthermore, in an embodiment of the present application, when a degraded image corresponding to the original image is generated by an image degradation method based on the original image, the original image can be adjusted according to a preset adjustment method to obtain an adjusted image corresponding to the original image; then, the adjusted image can be degraded according to the image degradation method to obtain a degraded image corresponding to the original image.

[0054] It can be understood that in the embodiments of the present application, the preset adjustment methods include at least one or more of the following: horizontal flip adjustment, vertical flip adjustment, rotation adjustment, and skip adjustment. Among them, horizontal flip adjustment can be understood as an adjustment to flip the image based on the horizontal line; vertical flip adjustment can be understood as an adjustment to flip the image based on the vertical line; rotation adjustment can be understood as rotating the image according to a preset angle, wherein the preset angle can be any angle and is not specifically limited in the present application; skip adjustment can be understood as not performing any processing on the image, but directly using the unadjusted image as the adjusted image.

[0055] Of course, the preset adjustment method may also include any other methods, such as mirror adjustment, conversion adjustment, etc., which are not specifically limited in this application.

[0056] That is to say, in the embodiment of the present application, when adjusting the original image according to the preset adjustment method, one or more methods can be selected to adjust the original image, or the original image can be selected not to be adjusted, that is, the adjustment of the original image can be skipped.

[0057] Further, in the embodiment of the present application, for any preset adjustment method, it is possible to select whether to adjust the original image according to the preset adjustment method according to a certain probability. Different preset adjustment methods may correspond to different probabilities of whether to adjust.

[0058] For example, in some embodiments, the horizontal flip adjustment may be performed with a probability of 50%, that is, for all original images, the horizontal flip adjustment may be performed with a probability of 50%.

[0059] For example, in some embodiments, for rotation adjustment, assuming that the preset angle is 90 degrees, it can be selected to be performed with a probability of 55%, that is, for all original images, the original images can be rotated 90 degrees with a probability of 55%.

[0060] It should be noted that, in the embodiments of the present application, the execution order of one or more preset adjustment methods may not be limited, and the execution order of multiple preset adjustment methods may also be pre-set.

[0061] Exemplarily, in some embodiments, assuming that the preset adjustment methods include horizontal flip adjustment, vertical flip adjustment, and rotation adjustment with a preset angle of 90 degrees, and the execution probability is 50%, the execution order can be preset as: horizontal flip with a probability of 50%; vertical flip with a probability of 50%; 90-degree rotation with a probability of 50%.

[0062] Furthermore, in an embodiment of the present application, after adjusting the original image according to a preset adjustment method to obtain an adjusted image corresponding to the original image, the adjusted image can be further degraded according to an image degradation method to obtain a degraded image corresponding to the original image.

[0063] It can be understood that in the embodiments of the present application, the image degradation method includes at least one or more of the following: noise degradation, motion blur degradation, and tadpole-shaped degradation. Among them, noise degradation can be understood as randomly adding different degrees of noise to the image to simulate the noise problem in the real environment; motion blur degradation can be understood as randomly adding different degrees and different angles of motion blur to the image to simulate the lens shake problem existing in real mobile phone shooting; tadpole-shaped degradation can be understood as randomly adding average pooling and Gaussian blur to the image to simulate the tadpole-shaped deformation effect of the QR code at different distances in the real environment.

[0064] Of course, in the process of performing image degradation, you can also choose to skip degradation, where skipping degradation can be understood as not performing any processing on the image, but directly using the non-degraded image as the degraded image.

[0065] It is understandable that, in the embodiments of the present application, the image degradation method may also include any other manner, which is not specifically limited in the present application.

[0066] That is, in the embodiment of the present application, when performing degradation processing on the adjusted image, one or more image degradation methods may be selected to perform image degradation processing, or the image degradation processing may be not performed, that is, the image degradation may be skipped.

[0067] Furthermore, in the embodiments of the present application, for any image degradation method, it is possible to select whether to perform image degradation processing according to the image degradation method according to a certain probability. Different image degradation methods may have different corresponding probabilities of degradation.

[0068] Furthermore, in an embodiment of the present application, when the adjusted image is degraded according to the image degradation method to obtain a degraded image corresponding to the original image, when the first image is degraded according to noise degradation, the first noise is determined from the candidate noises; then, when the first noise is Gaussian noise, salt and pepper noise, or Poisson noise, the noise parameter of the first noise can be determined according to a preset noise parameter range corresponding to the first noise; based on the noise parameter, the first image is degraded according to the first noise to obtain a degraded image.

[0069] In the embodiment of the present application, the candidate noise includes at least one of the following: Gaussian noise, salt and pepper noise, Poisson noise, and no noise. That is, when degrading the image according to noise degradation, any type of noise can be selected to be added, or no noise can be selected, that is, no noise.

[0070] In an embodiment of the present application, the first image may be one of the following: an adjusted image, an image processed by motion blur degradation, and an image processed by tadpole-shaped degradation. That is, the first image may be an adjusted image that has been adjusted but not degraded, or an image processed by any degradation, for example, an image processed by motion blur degradation, or an image processed by tadpole-shaped degradation.

[0071] It can be seen that, in the embodiments of the present application, the timing for performing noise degradation can be selected after completing image adjustment, or after completing any type of degradation processing.

[0072] In an embodiment of the present application, assuming that the first noise is Gaussian noise, salt and pepper noise, or Poisson noise, when performing noise degradation, a preset noise parameter range corresponding to the first noise can be first obtained, and then a corresponding noise parameter can be selected from the preset noise parameter range.

[0073] It can be understood that, in the embodiments of the present application, for different types of noise, the corresponding preset noise parameter ranges may be different.

[0074] Exemplarily, in some embodiments, the preset noise parameter range corresponding to Gaussian noise may be [0, 32], the preset noise parameter range corresponding to Poisson noise may be [0, 32], and the preset noise parameter range corresponding to salt and pepper noise may be [0, 0.04].

[0075] It is understandable that in the embodiment of the present application, when determining the noise parameter, any parameter value in the preset noise parameter range corresponding to the first noise can be determined as the noise parameter corresponding to the first noise. In other words, the noise parameter can be randomly selected in the preset noise parameter range.

[0076] In the embodiment of the present application, for a plurality of candidate noises, the proportions of different candidate noises may be preset, and then when performing noise degradation, the candidate noise may be selected according to the preset proportions of the candidate noises.

[0077] Exemplarily, in some embodiments, it is assumed that the candidate noise includes four types: Gaussian noise, salt and pepper noise, Poisson noise, and no noise, and the corresponding proportions of these four types are 3:3:3:1, that is, when performing noise degradation, Gaussian noise can be added to 30% of the first image, salt and pepper noise can be added to 30% of the first image, Poisson noise can be added to 30% of the first image, and no noise can be added to 10% of the first image.

[0078] Accordingly, in the embodiment of the present application, assuming that the first noise is noise-free, when performing noise degradation, it is possible to choose to skip the increase of noise, that is, directly use the image without the increase of noise as the image after noise degradation.

[0079] Further, in an embodiment of the present application, when the adjusted image is degraded according to the image degradation method to obtain a degraded image corresponding to the original image, when the second image is degraded according to the motion blur degradation, a motion blur angle parameter is determined within a preset angle parameter range, and a motion blur degree parameter is determined within a preset degree parameter range. Then, the second image can be degraded based on the motion blur angle parameter and the motion blur degree parameter to obtain a degraded image.

[0080] In an embodiment of the present application, the second image may be one of the following: an adjusted image, an image processed by noise degradation, and an image processed by tadpole-shaped degradation. That is, the second image may be an adjusted image that has been adjusted but not degraded, or an image processed by any degradation, for example, an image processed by noise degradation, or an image processed by tadpole-shaped degradation.

[0081] It can be seen that, in the embodiments of the present application, the timing for executing motion blur degradation can be selected after completing image adjustment or after completing any type of degradation processing.

[0082] It can be understood that in an embodiment of the present application, when performing motion blur degradation, a preset angle parameter range and a preset degree parameter range can be obtained first, and then a motion blur angle parameter is selected within the preset angle parameter range, and a motion blur degree parameter is selected within the preset degree parameter range.

[0083] It is understood that in the embodiment of the present application, the preset angle parameter range can be used to determine the motion angle offset range of motion blur, and the preset degree parameter range can be used to determine the motion degree offset range of motion blur. The preset angle parameter range and the preset degree parameter range can be different.

[0084] For example, in some embodiments, the preset angle parameter range may be [1,180], and the preset degree parameter range may be [10,20].

[0085] It is understandable that in the embodiment of the present application, when determining the motion blur angle parameter, any parameter value in the preset angle parameter range can be determined as the motion blur angle parameter. In other words, the motion blur angle parameter can be obtained by randomly selecting in the preset angle parameter range.

[0086] It is understandable that in the embodiment of the present application, when determining the motion blur angle parameter, any parameter value in the preset angle parameter range can be determined as the motion blur angle parameter. In other words, the motion blur angle parameter can be obtained by randomly selecting in the preset angle parameter range.

[0087] It is understandable that in the embodiment of the present application, when determining the motion blur degree parameter, any parameter value in the preset degree parameter range can be determined as the motion blur degree parameter. In other words, the motion blur degree parameter can be obtained by randomly selecting in the preset degree parameter range.

[0088] In an embodiment of the present application, after determining the motion blur angle parameter and the motion blur degree parameter, the second image can be further degraded based on the motion blur angle parameter and the motion blur degree parameter to obtain a motion blur degraded image.

[0089] Furthermore, in an embodiment of the present application, motion blur degradation processing may be randomly selected. For example, the probability of whether to perform image degradation in a motion blur degradation manner may be pre-set, and then when performing motion blur degradation, the image to be subjected to motion blur degradation may be selected according to the preset probability.

[0090] Exemplarily, in some embodiments, assuming that the probability of performing motion blur degradation is preset to 20%, and the probability of not performing motion blur degradation is 80%, then when performing motion blur degradation, it can be selected to perform motion blur degradation on 20% of the second images according to randomly determined motion blur angle parameters and motion blur degree parameters, and it is selected not to perform motion blur degradation on 80% of the second images.

[0091] Accordingly, in the embodiment of the present application, assuming that motion blur degradation is not performed on the second image, the image that has not been subjected to motion blur degradation can be directly used as the image after motion blur degradation.

[0092] Further, in an embodiment of the present application, when the adjusted image is degraded according to the image degradation method to obtain a degraded image corresponding to the original image, when the third image is degraded according to the tadpole-shaped degradation, a first degradation mode is determined among the candidate degradation modes; when the first degradation mode is average pooling degradation or Gaussian blur degradation, a degradation degree parameter corresponding to the first degradation mode is determined; based on the degradation degree parameter, a degradation parameter corresponding to the first degradation mode is determined; based on the degradation parameter, the third image is degraded according to the first degradation mode to obtain a degraded image.

[0093] In an embodiment of the present application, the candidate degradation modes may include at least one of the following: average pooling degradation, Gaussian blur degradation, and no degradation. That is, when degrading an image according to tadpole-shaped degradation, any type of degradation mode may be selected to be added, or tadpole-shaped degradation may be skipped, i.e., no degradation.

[0094] In an embodiment of the present application, the third image may be one of the following: an adjusted image, an image processed by noise degradation, and an image processed by motion blur degradation. That is, the third image may be an adjusted image that has been adjusted but not degraded, or an image processed by any degradation, for example, an image processed by motion blur degradation, or an image processed by noise degradation.

[0095] It can be seen that, in the embodiments of the present application, the timing for performing tadpole-shaped degradation can be selected after completing image adjustment, or after completing any type of degradation processing.

[0096] In an embodiment of the present application, assuming that the first degradation mode is average pooling degradation or Gaussian blur degradation, when performing tadpole-shaped degradation, the degradation degree parameter corresponding to the first degradation mode can be determined first; and then the degradation parameter corresponding to the first degradation mode can be determined based on the degradation degree parameter.

[0097] It can be understood that in an embodiment of the present application, a degradation degree parameter can be used to determine the degree of degradation for performing tadpole-shaped degradation, wherein the degradation degree parameter may include a high degree of degradation and a low degree of degradation, wherein the proportion of the high degree of degradation and the low degree of degradation may be pre-set or random.

[0098] In the embodiment of the present application, for different types of candidate degradation modes, the corresponding proportions of high-degree degradation and low-degree degradation may be the same or different.

[0099] Exemplarily, in some embodiments, for average pooling degradation, the proportions of high-level degradation and low-level degradation may be 80% and 20%, respectively.

[0100] Exemplarily, in some embodiments, for Gaussian blur degradation, the proportions of high-degree degradation and low-degree degradation may be 80% and 20%, respectively.

[0101] It is understandable that in the embodiments of the present application, the degradation parameters corresponding to different candidate degradation modes may also be different. For average pooling degradation, the corresponding degradation parameter may be the pooling window size; for Gaussian blur degradation, the corresponding degradation parameter may be the Gaussian degradation parameter.

[0102] In an embodiment of the present application, the degradation parameter corresponding to the first degradation mode is determined based on the degradation degree parameter corresponding to the first degradation mode, that is, the degradation parameters used under different degradation degree parameters are different. For example, for average pooling degradation, the pooling window sizes corresponding to high degree degradation and low degree degradation are different; for Gaussian blur degradation, the Gaussian degradation parameters corresponding to high degree degradation and low degree degradation are different.

[0103] Exemplarily, in some embodiments, for average pooling degradation, when a low degree of degradation is selected, 1 / 5 of the original image size is used as the pooling window size; when a high degree of degradation is selected, 1 / 4 of the original image size is used as the pooling window size.

[0104] Exemplarily, in some embodiments, for Gaussian blur degradation, when a low degree of degradation is selected, Gaussian degradation parameters are randomly selected from [0, 16]; when a high degree of degradation is selected, Gaussian degradation parameters are randomly selected from [16, 32].

[0105] In the embodiment of the present application, after the degradation parameters corresponding to the first degradation mode are determined, the third image can be degraded according to the first degradation mode based on the degradation parameters to obtain a tadpole-shaped degraded image.

[0106] In an embodiment of the present application, for a plurality of candidate degradation modes, the proportions of different candidate degradation modes may be preset, and then when tadpole-shaped degradation is performed, a degradation mode may be selected according to the preset proportions of the candidate degradation modes.

[0107] Exemplarily, in some embodiments, it is assumed that the candidate degradation modes include three modes: average pooling degradation, Gaussian blur degradation, and no degradation, and the corresponding proportions of these three modes are 7:2:1, that is, when performing tadpole-shaped degradation, 70% of the third images can be subjected to average pooling degradation, 20% of the third images can be subjected to Gaussian blur degradation, and 10% of the third images can not be subjected to tadpole-shaped degradation.

[0108] Accordingly, in the embodiment of the present application, assuming that the first degradation mode is no degradation, when performing tadpole-shaped degradation, you can choose to skip the degradation process, that is, directly use the image that has not performed tadpole-shaped degradation as the image after tadpole-shaped degradation.

[0109] It can be seen that in the embodiments of the present application, degradation processing is performed according to the image degradation method, which may include but is not limited to one or more of random multiple degrees of noise degradation, random multiple degrees of motion blur degradation, and random multiple degrees of tadpole-shaped degradation.

[0110] It should be noted that, in the embodiments of the present application, the execution order of one or more image degradation methods may not be limited, and the execution order of multiple image degradation methods may be preset.

[0111] Exemplarily, in some embodiments, it is assumed that the multiple image degradation methods include three methods: random multi-degree noise degradation, random multi-degree motion blur degradation, and random multi-degree tadpole-shaped degradation. The execution order of the three methods is pre-set as follows: random multi-degree noise degradation, random multi-degree motion blur degradation, and random multi-degree tadpole-shaped degradation.

[0112] Furthermore, in the embodiments of the present application, by performing image degradation processing through an image degradation method, it is possible to simulate the noise in a real environment, simulate the lens shake that exists when taking pictures with a real mobile phone, and simulate the tadpole-like deformation of a QR code at different distances in a real environment, thereby obtaining a richer and more realistic degraded image.

[0113] Furthermore, in an embodiment of the present application, for an original image, it is possible to select to repeatedly process it through the preset adjustment method and / or image degradation method described above to obtain a plurality of corresponding degraded images. In this way, by combining the original image and the plurality of corresponding degraded images, a plurality of groups of corresponding training data can be obtained.

[0114] Step 102: Generate a training data set based on the original image and the degraded image.

[0115] In an embodiment of the present application, after a degraded image corresponding to the original image is generated by an image degradation method based on the original image, a training data set may be further generated based on the original image and the degraded image.

[0116] In an embodiment of the present application, the training data in the training data set may include original images and degraded images in one-to-one correspondence, wherein the definition of the original image is higher than that of the degraded image.

[0117] It is understandable that if an original image is repeatedly processed by the preset adjustment method and / or image degradation method described above, then an original image and the obtained multiple corresponding degraded images can constitute multiple sets of corresponding training data.

[0118] For example, in some embodiments, assuming that the number of original images is M, each original image is processed N times by the preset adjustment method and / or image degradation method described above to obtain N corresponding degraded images, and finally M×N groups of training data can be obtained, where M and N are both integers greater than 0.

[0119] Step 103: train an initial model based on the training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first parameterization module, and the image deblurring model includes a channel space attention module and a second parameterization module corresponding to the first parameterization module.

[0120] In an embodiment of the present application, after a training data set is generated based on the original image and the degraded image, an initial model may be further trained based on the training data set to obtain an image deblurring model.

[0121] In an embodiment of the present application, the initial model includes a channel-spatial attention module (Channel-SpatialAttention Block, CSAB) and a first parameterization module, and the image deblurring model obtained by training includes the channel-spatial attention module and a second parameterization module corresponding to the first parameterization module.

[0122] In an embodiment of the present application, the first reparameterization module and the second reparameterization module may be the same reparameterization module or different reparameterization modules. For example, the second reparameterization module may be obtained by optimizing and merging the first reparameterization module.

[0123] Furthermore, in the embodiments of the present application, Figure 2 A schematic diagram of the initial model proposed in the embodiment of the present application, such as Figure 2As shown in FIG, the initial model may include a downsampling module, an enhancement module, and an upsampling module. The downsampling module can be used to reduce the number of samples of a large number of categories in the data set, and can also be used to reduce the size of the feature map to reduce the amount of calculation. The enhancement module can be used to capture and utilize the feature differences between samples to improve the feature expression capability; the upsampling module can be used to increase the number of samples of a small number of categories, and can also be used to restore the spatial size of the feature map to the original input size.

[0124] Furthermore, in an embodiment of the present application, the downsampling module and the enhancement module both include a channel space attention module and a first parameterization module.

[0125] Furthermore, in the embodiments of the present application, Figure 3 A schematic diagram of the initial model proposed in the embodiment of the present application, such as Figure 3 As shown, the channel spatial attention module can include two parts: channel attention and spatial attention.

[0126] Among them, channel attention focuses on the feature channel dimension of the image, that is, a collection of different features. Its core is to assign a weight to each channel, which reflects the importance of the channel to the final task. By dynamically adjusting the weight of each channel, the network can focus on those features that are more critical to the task, thereby improving overall performance. Specifically, the channel attention mechanism usually aggregates spatial information through global pooling operations (such as global average pooling or global maximum pooling) and learns the importance of each channel. This information is then further processed through one or more convolutional layers or fully connected layers to ultimately generate attention weights for each channel. These weights are multiplied by the original feature map to achieve feature reweighting, so that important channel features are amplified and unimportant channel features are suppressed.

[0127] Spatial attention focuses on the spatial dimensions of the image, i.e. the width and height of the image. Its purpose is to identify important areas in the image (such as specific parts of an object) and assign higher weights to these areas. Through the spatial attention mechanism, the model is able to focus on the key visual elements of the image and ignore the background or other unimportant areas. Spatial attention is usually implemented by learning an attention mask that spatially weights each position in the feature map. This mask, which can be learned by a model such as a convolutional neural network, reflects the importance of each position to the final task. This mask is then multiplied with the original feature map to obtain a weighted feature map in which important areas are retained and enhanced, while unimportant areas are suppressed.

[0128] In the embodiments of the present application, channel attention and spatial attention can be combined to improve the performance of the model. For example, channel attention is used to extract features first, and then spatial attention is used to focus on key areas in the features. This combination can enable the model to achieve better results in both feature extraction and regional attention.

[0129] For example, in some embodiments, Figure 4 This is a schematic diagram of the application of the channel space attention module proposed in the embodiment of the present application, such as Figure 4 As shown in the figure, for the channel attention part, first use global maximum pooling and global average pooling to process the input X to obtain two new feature vectors, denoted as F1 and F2. Then use the same two-layer fully connected network to transform F1 and F2, and the results are denoted as A1 and A2. Finally, add A1 and A2 to get A3. Then use A3 to scale each channel of X to get the result X1. For spatial attention, first use global maximum pooling and global average pooling to process the input X1 to obtain two new feature vectors, denoted as D1 and D2. Then use 1x1 convolution on D1 and D2 to get the spatial attention map X2. Finally, use the spatial attention map X2 to multiply the input X1 to get the final output X3.

[0130] Furthermore, in the embodiments of the present application, Figure 5 A schematic diagram of the initial model proposed in the embodiment of the present application, such as Figure 5 As shown, the first parameterized module can be composed of two parts, the upper and lower parts. In the upper part, there are four branches, and their outputs are combined into one output by addition. In the lower part, there are two branches. The four branches in the upper part can include 1x1 depth separable convolution and batch normalization (Batch Normalization, BN), 3x3 depth separable convolution and BN, BN, jump connection. The two branches in the lower part can include 1x1 convolution and BN, BN.

[0131] Furthermore, in an embodiment of the present application, the upsampling part of the image deblurring model uses a bilinear interpolation method, referring to the formula:

[0132]

[0133] Among them, f(x,y) is the calculated interpolation, Q 11 =(x1,y1),Q 12 =(x1,y2),Q 21 =(x2,y1) and Q 22 =(x2,y2) are the values ​​of four known pixels. The calculation process of bilinear interpolation has no parameters that need to be learned, so the inference speed is extremely fast.

[0134] Furthermore, in an embodiment of the present application, when an initial model is trained based on a training data set to obtain an image deblurring model, a deblurred image corresponding to the degraded image can be obtained through the initial model based on the degraded image in the training data set; then, based on the original image and the deblurred image corresponding to the degraded image, a loss is determined; and based on the loss, the initial model is corrected to obtain the image deblurring model.

[0135] It can be understood that in the embodiments of the present application, when the initial model is trained based on the training data set, the degraded image in the training data set can be input into the initial model for image deblurring processing, and the corresponding prediction result is output to obtain the corresponding deblurred image, that is, the loss between the deblurred image and the corresponding original image is determined based on the loss function, and the model parameters of the initial model are further corrected using the calculated loss, and finally the image deblurring model is obtained.

[0136] Further, in an embodiment of the present application, when obtaining a deblurred image corresponding to the degraded image based on the degraded image in the training data set through the initial model, the first feature image corresponding to the degraded image can be obtained based on the degraded image through the channel space attention module in the upsampling module; then, based on the first feature image, a second feature image corresponding to the degraded image can be obtained through the first reparameterization module in the upsampling module; then, based on the second feature image, a third feature image corresponding to the degraded image can be obtained through the channel space attention module in the enhancement module; then, based on the third feature image, a fourth feature image corresponding to the degraded image can be obtained through the first reparameterization module in the enhancement module; finally, the deblurred image can be obtained based on the fourth feature image through the upsampling module.

[0137] For example, in some embodiments, Figure 6 A schematic diagram of the training method of the initial module proposed in the embodiment of the present application is shown as follows: Figure 6 As shown, after the degraded image is input into the initial model, feature extraction can be performed through four down-sampling modules, wherein each down-sampling module includes a channel space attention module and a first re-parameterization module connected in sequence, and then the feature map output by the down-sampling module can be input into the enhancement module, wherein each of the three enhancement modules includes a channel space attention module and a first re-parameterization module connected in sequence, and then the feature map output by the enhancement module can be input into the up-sampling module, and finally the deblurred image can be output through the four up-sampling modules.

[0138] Further, in an embodiment of the present application, when determining the loss based on the original image and the deblurred image corresponding to the degraded image, any type of loss function can be selected to calculate the loss between the original image and the deblurred image.

[0139] For example, in some embodiments, L1 Loss can be selected for use during model training. L1 Loss is also called Mean Absolute Error (MAE), which refers to the average value of the absolute difference between the model predicted value f(x) and the true value y, and the formula is as follows:

[0140]

[0141] Among them, f(x i ) and y i Respectively represent the predicted value and the corresponding true value of the i-th sample, that is, f(x i ) represents the deblurred image, y i represents the original image, n is the number of samples. The image format for input model training is unified as a three-channel grayscale JPG image of size 128×128.

[0142] Furthermore, in an embodiment of the present application, when obtaining an image deblurring model by correcting the initial model based on the loss, the initial model can be first corrected based on the loss to obtain a corrected model corresponding to the initial model; then the first parameterized module in the corrected model is merged and optimized into a second parameterized module to obtain the image deblurring model.

[0143] It can be understood that in the embodiments of the present application, when merging and optimizing the first re-parameterized module into the second re-parameterized module, the four branches in the upper half of the first re-parameterized module can be selected to be merged and optimized based on linear additivity, or the two branches in the lower half of the first re-parameterized module can be selected to be merged and optimized, and the present application does not make specific limitations.

[0144] For example, in some embodiments, Figure 7 This is a schematic diagram of the image deblurring model proposed in the embodiment of the present application, such as Figure 7 As shown, the four branches in the upper half of the first reparameterization module can be merged and optimized based on linear additivity to obtain two merged branches, such as the first branch and the second branch; the two branches in the lower half of the first reparameterization module can also be merged and optimized to obtain one merged branch, such as the third branch; this application does not make specific limitations.

[0145] That is, in the embodiment of the present application, the re-parameterization module can maintain the following in the training phase: Figure 5The structure shown in the figure can merge these modules after training based on the linear additive property, thereby improving the speed of reasoning.

[0146] In summary, the training method of the image deblurring model proposed in the embodiment of the present application designs a new lightweight image deblurring model. Among them, on the one hand, the training data set for training the image deblurring model can be generated by an image degradation method, which improves the representation ability of the image deblurring model; on the other hand, the channel space attention mechanism and the reparameterization method can be used to increase the representation ability of the model during the training process; on the other hand, in the application process of the image deblurring model, the combined optimized reparameterization module can be used to improve the model calculation speed in the inference stage to meet the requirements of low latency and low power consumption on the mobile phone side.

[0147] It can be seen that the training method of the image deblurring model proposed in the embodiment of the present application includes an end-to-end neural network structure, which improves the representation ability of the model by using reparameterization and channel space attention modules, while also maintaining a low overall computational complexity of the model. Therefore, this solution can be applied to actual mobile phone applications, with the characteristics of high precision and low power consumption.

[0148] The embodiment of the present application proposes a training method for an image deblurring model, which generates a degraded image corresponding to the original image through an image degradation method based on an original image; wherein the clarity of the degraded image is lower than that of the original image; a training data set is generated based on the original image and the degraded image; an initial model is trained based on the training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first re-parameterization module, and the image deblurring model includes a channel space attention module and a second re-parameterization module corresponding to the first re-parameterization module. That is to say, in the embodiment of the present application, a training data set generated by a multi-degree image degradation method can be used for model training, so that the obtained image deblurring model has excellent representation ability; the image deblurring model includes a channel space attention module and a re-parameterization module, and the channel space attention mechanism and the re-parameterization method can be used to further enhance the representation ability of the model; image deblurring processing is performed by the image deblurring model to obtain a clearer image, solve the problem of image blur, and enhance the intelligence of the terminal device.

[0149] Based on the above embodiments, another embodiment of the present application proposes an image deblurring processing method, which can be applied to an image deblurring processing device or electronic device, and can also be applied to any terminal including an image deblurring processing device or electronic device.

[0150] Below, taking an image deblurring processing device as an example, the image deblurring processing method proposed in the embodiment of the present application is exemplarily described.

[0151] Furthermore, in the embodiments of the present application, Figure 8 The following is a flowchart of the image deblurring method proposed in the embodiment of the present application. Figure 8 As shown, the image deblurring processing method may include the following steps:

[0152] Step 201: Obtain an image to be processed.

[0153] Step 202: obtaining a processed image corresponding to the image to be processed based on the image to be processed through an image deblurring model; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training based on a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed.

[0154] In an embodiment of the present application, an image to be processed may be first acquired, and then a processed image corresponding to the image to be processed may be acquired based on the image to be processed through an image deblurring model, wherein the processed image has a higher definition than the image to be processed.

[0155] That is to say, in the embodiments of the present application, the image deblurring model can be used to improve the clarity of the image. For a blurred image to be processed with low clarity, a processed image with high clarity can be obtained through the image deblurring model.

[0156] Furthermore, in an embodiment of the present application, the image deblurring model is obtained by training based on a training data set, and the training data set is generated by an image degradation method.

[0157] In the embodiment of the present application, the image degradation method can be understood as obtaining a blurred degraded image corresponding to a clear original image by adding noise, blurring, etc. to the original image. When generating a degraded image corresponding to the original image by the image degradation method, the original image can be adjusted according to a preset adjustment method to obtain an adjusted image corresponding to the original image; then, the adjusted image can be degraded according to the image degradation method to obtain a degraded image corresponding to the original image.

[0158] It can be understood that in the embodiments of the present application, the image degradation method includes at least one or more of the following: noise degradation, motion blur degradation, and tadpole-shaped degradation. Among them, noise degradation can be understood as randomly adding different degrees of noise to the image to simulate the noise problem in the real environment; motion blur degradation can be understood as randomly adding different degrees and different angles of motion blur to the image to simulate the lens shake problem existing in real mobile phone shooting; tadpole-shaped degradation can be understood as randomly adding average pooling and Gaussian blur to the image to simulate the tadpole-shaped deformation effect of the QR code at different distances in the real environment.

[0159] Of course, in the process of performing image degradation, you can also choose to skip degradation, where skipping degradation can be understood as not performing any processing on the image, but directly using the non-degraded image as the degraded image. That is, when performing degradation processing on the adjusted image, you can select one or more image degradation methods to perform image degradation processing, or you can choose not to perform image degradation processing, that is, skip the image degradation.

[0160] Further, in an embodiment of the present application, the image deblurring model is obtained by training the initial model with a training data set. The initial model includes a channel space attention module and a first re-parameterization module, and the image deblurring model obtained by training includes a channel space attention module and a second re-parameterization module. The first re-parameterization module and the second re-parameterization module may be the same re-parameterization module or different re-parameterization modules. For example, the second re-parameterization module may be obtained by optimizing and merging the first re-parameterization module.

[0161] That is, in the embodiment of the present application, the re-parameterization module can maintain the following in the training phase: Figure 5 The structure shown in the figure can merge these modules after training based on the linear additive property, thereby improving the speed of reasoning.

[0162] Furthermore, in the embodiments of the present application, Fig. 9 This is a schematic diagram of the image deblurring model proposed in the embodiment of the present application, such as Fig. 9 As shown in FIG, the image deblurring model may include a downsampling module, an enhancement module, and an upsampling module. The downsampling module may be used to reduce the number of samples of a larger number of categories in the data set, or may be used to reduce the size of the feature map to reduce the amount of calculation. The enhancement module may be used to capture and utilize the feature differences between samples to improve the feature expression capability; the upsampling module may be used to increase the number of samples of a smaller number of categories, or may be used to restore the spatial size of the feature map to the original input size.

[0163] Furthermore, in an embodiment of the present application, the downsampling module and the enhancement module both include a channel space attention module and a second parameterization module.

[0164] Further, in an embodiment of the present application, the channel space attention module may include two parts: channel attention and spatial attention. Among them, channel attention focuses on the characteristic channel dimension of the image, that is, a collection of different features. Its core is to assign a weight to each channel, which reflects the importance of the channel to the final task. By dynamically adjusting the weight of each channel, the network can focus on those features that are more critical to the task, thereby improving the overall performance. Specifically, the channel attention mechanism usually aggregates spatial information through global pooling operations (such as global average pooling or global maximum pooling) and learns the importance of each channel. Then, this information is further processed through one or more convolutional layers or fully connected layers to finally generate the attention weights for each channel. These weights are multiplied with the original feature map to achieve feature reweighting, so that important channel features are amplified and unimportant channel features are suppressed. Spatial attention focuses on the spatial dimensions of the image, that is, the width and height of the image. Its purpose is to identify important areas in the image (such as specific parts of an object) and assign higher weights to these areas. Through the spatial attention mechanism, the model can focus on the key visual elements of the image and ignore the background or other unimportant areas. Spatial attention is usually achieved by learning an attention mask that weights each position of the feature map spatially. This mask can be learned by models such as convolutional neural networks, which reflects the importance of each position for the final task. Then, this mask is multiplied with the original feature map to obtain a weighted feature map, in which important areas are retained and enhanced, while unimportant areas are suppressed.

[0165] In the embodiments of the present application, channel attention and spatial attention can be combined to improve the performance of the model. For example, channel attention is used to extract features first, and then spatial attention is used to focus on key areas in the features. This combination can enable the model to achieve better results in both feature extraction and regional attention.

[0166] Further, in an embodiment of the present application, when obtaining a processed image corresponding to the image to be processed based on the image to be processed through an image deblurring model, the first feature image corresponding to the image to be processed can be obtained based on the image to be processed through a channel space attention module in an upsampling module; then, based on the first feature image, a second feature image corresponding to the image to be processed can be obtained through a second re-parameterization module in the upsampling module; then, based on the second feature image, a third feature image corresponding to the image to be processed can be obtained through a channel space attention module in an enhancement module; then, based on the third feature image, a fourth feature image corresponding to the image to be processed can be obtained through the second re-parameterization module in the enhancement module; and finally, based on the fourth feature image, a processed image can be obtained through an upsampling module.

[0167] For example, in some embodiments, Fig.10 This is a schematic diagram of the application of the image deblurring model proposed in the embodiment of the present application. Fig.10 As shown, after the image to be processed is input into the image deblurring model, feature extraction can be performed through four down-sampling modules, wherein each down-sampling module includes a channel space attention module and a second re-parameterization module connected in sequence, and then, the feature map output by the down-sampling module can be input into the enhancement module, wherein each of the three enhancement modules includes a channel space attention module and a second re-parameterization module connected in sequence, and then, the feature map output by the enhancement module can be input into the up-sampling module, and finally the processed image can be output through the four up-sampling modules.

[0168] In summary, the image deblurring processing method proposed in the embodiment of the present application improves the clarity of the image through a new lightweight image deblurring model. Among them, on the one hand, the image deblurring model can be obtained by training a training data set generated based on an image degradation method, so that the image deblurring model has a strong representation ability; on the other hand, the image deblurring model can utilize the channel space attention mechanism and the reparameterization method to deblur the image, further improving the representation ability of the model; on the other hand, in the application process of the image deblurring model, the merged optimized reparameterization module can be used to improve the model calculation speed in the inference stage, so as to meet the requirements of low latency and low power consumption on the mobile phone side.

[0169] It can be seen that the image deblurring processing method proposed in the embodiment of the present application includes an end-to-end neural network structure, which improves the representation ability of the model by using reparameterization and channel space attention modules, while also maintaining a low overall computational complexity of the model. Therefore, this solution can be applied to actual mobile phone applications, with the characteristics of high precision and low power consumption.

[0170] The embodiment of the present application proposes a method for image deblurring, which obtains an image to be processed; through an image deblurring model, based on the image to be processed, a processed image corresponding to the image to be processed is obtained; wherein the image deblurring model includes a channel space attention module and a second re-parameterization module, and the image deblurring model is obtained based on training of a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed. That is to say, in the embodiment of the present application, a training data set generated by a multi-degree image degradation method can be used for model training, so that the obtained image deblurring model has excellent representation ability; the image deblurring model includes a channel space attention module and a re-parameterization module, and the channel space attention mechanism and the re-parameterization method can be used to further enhance the representation ability of the model; the image deblurring model is used to perform image deblurring processing, and a clearer image can be obtained, which solves the problem of image blur and enhances the intelligence of the terminal device.

[0171] Based on the above embodiments, another embodiment of the present application provides a method for training an image deblurring model and a method for processing image deblurring, wherein the image deblurring model obtained through training can be used to implement image deblurring processing. When generating a training data set for training the image deblurring model, an image degradation method is mainly used, and the image deblurring model can be used to implement image deblurring processing using an image feature extraction method, an image feature fusion method, and an attention mechanism.

[0172] It is understandable that there are many applications in smartphones that need to use images in the camera preview stream (such as QR code scanning, long-distance text scanning, etc.). When such functions are used at a long distance, the camera preview stream image will be blurred and deformed due to the long-distance zoom. The lightweight image deblurring solution proposed in the embodiment of the present application can solve the problem of low image quality when such functions that rely on camera preview stream images need to be used at a long distance.

[0173] Furthermore, in an embodiment of the present application, a clear camera preview stream image is first collected, and then a data set paired with the clear image is obtained through an image degradation method; then an end-to-end lightweight neural network is constructed, and the model is trained using the constructed training data, and finally the trained model is re-parameterized to obtain the final model, and deployed on the terminal device to achieve image deblurring processing through an image deblurring model.

[0174] Furthermore, in an embodiment of the present application, a lightweight image deblurring model training method may include the following detailed steps: image acquisition, image degradation, and image deblurring model construction and training.

[0175] Furthermore, in an embodiment of the present application, when performing image acquisition, 10,000 clear, unzoomed images are collected from the camera preview stream in the mobile phone. Image types include QR code images, common chart images (flow charts, mind maps, pie charts, bar charts, architecture diagrams, line charts), document images, etc. Different models of mobile phones are used during the acquisition process to increase the diversity of images.

[0176] Furthermore, in an embodiment of the present application, when image degradation is performed, the following steps are performed in sequence for each image in the data set:

[0177] 1) Flip horizontally with a probability of 50%;

[0178] 2) Perform vertical flip with a probability of 50%;

[0179] 3) Rotate 90 degrees with a probability of 50%;

[0180] 4) performing a random multi-degree noise degradation method;

[0181] 5) Execute random multi-degree motion blur degradation method;

[0182] 6) Perform a random multi-degree tadpole-like degradation method.

[0183] The above steps are repeated 10 times for each image, so each image can actually generate 10 degraded images. The degraded data set contains a total of 100,000 pairs of clear and degraded images, thus completing the construction of the training data set.

[0184] In order to obtain better image processing effects, you can choose to use the SLAM algorithm to obtain the position and posture (pose data) of the shooting module (such as a camera) and pre-map the image and pose. When performing image processing, for images for which accurate image processing parameters cannot be obtained, you can determine a reference image with a close but reliable position and posture based on the pose data, and use the image processing parameters of the reference image for image processing, thereby improving the image processing effect.

[0185] Among them, for the random multi-degree noise degradation method, different degrees of Gaussian noise, salt and pepper noise, and Poisson noise are randomly added to simulate the noise problem in the real environment. The proportions of Gaussian noise, salt and pepper noise, Poisson noise, and no noise are 3:3:3:1 respectively. The random selection range of the Gaussian noise degree parameter used in this scheme is [0,32], the random selection range of the Poisson noise degree parameter is [0,32], and the random selection range of the salt and pepper noise degree parameter is [0,0.04]. When noise degradation is performed on each image, a type of noise will be randomly selected according to the proportion of these four types of noise, and then the parameters will be randomly selected from the corresponding parameter range, and the final degradation operation will be performed. If no noise is selected, the original image will be directly retained.

[0186] Among them, for the random multi-degree motion blur degradation method, motion blur of different degrees and angles is added to simulate the lens shake problem existing in real mobile phone shooting to improve the robustness of the model. When each image is subjected to motion blur degradation, the image will be selected for motion blur degradation with a probability of 20%; and the original image will be retained with a probability of 80%. After the image is selected for motion blur degradation, the parameters of motion blur angle and motion blur degree will be randomly selected, and the parameter ranges are [10,20] and [1,180] respectively.

[0187] Among them, for the random multi-degree tadpole-shaped degradation method, the average pooling and Gaussian blurring methods are selected to simulate the tadpole-shaped deformation effect of the QR code at different long distances in the real environment. This solution uses three methods: average pooling degradation, Gaussian blur degradation, and non-degradation, with a ratio of 7:2:1.

[0188] When performing tadpole degradation on each image, we first select from three methods: average pooling degradation, Gaussian blur degradation, and no degradation with a probability of 0.7:0.2:0.1.

[0189] If average pooling degradation is selected, low degradation is selected with probability 0.8; high degradation is selected with probability 0.2. When low degradation is selected, 1 / 5 of the original image size is used as the pooling window size; when high degradation is selected, 1 / 4 of the original image size is used as the pooling window size. The pooling window step size is fixed to 1 / 4 of the original image size.

[0190] If Gaussian blur degradation is selected, low degradation is selected with probability 0.8; high degradation is selected with probability 0.2. When low degradation is selected, Gaussian degradation parameters are randomly selected from [0,16]; when high degradation is selected, Gaussian degradation parameters are randomly selected from [16,32].

[0191] If you choose not to perform image degradation processing, the original image will be directly retained.

[0192] Furthermore, in an embodiment of the present application, when constructing an image deblurring model, an encoder-decoder architecture is used in a lightweight deblurring model of a deep neural network. For example, the entire model first passes through a four-time downsampling module, then passes through a feature enhancement module of the network, and finally passes through a four-time upsampling module.

[0193] Each convolution module adopts the structure of Conv-ReLU-BN, but a channel spatial attention module is added afterwards. For the channel attention part, the input X is first processed using global maximum pooling and global average pooling to obtain two new feature vectors, denoted as F1 and F2. Then F1 and F2 are transformed using the same two-layer fully connected network, and the results are denoted as A1 and A2. Finally, A1 and A2 are added to obtain A3. Then A3 is used to scale each channel of X to obtain the result X1. For spatial attention, the input X1 is first processed using global maximum pooling and global average pooling to obtain two new feature vectors, denoted as D1 and D2. Then 1x1 convolution is performed on D1 and D2 to obtain a spatial attention map. Finally, the spatial attention map is multiplied with the input X1 to obtain the final output.

[0194] Each convolution module can adopt a reparameterization module, where the reparameterization module consists of two parts. In the upper part, there are four branches, and their outputs are combined into one output by adding. The four branches are: 1) 1x1 depthwise separable convolution and BN;

[0195] 2) 3x3 depth-wise separable convolution and BN;

[0196] 3) BN;

[0197] 4) Skip connections.

[0198] The lower part of the reparameterization module consists of two branches:

[0199] 1) 1x1 convolution and BN;

[0200] 2)BN.

[0201] The re-parameterized module maintains the structure of the first re-parameterized module during the training phase. After training, these modules can be merged and optimized into the second re-parameterized module according to the linear additive property, thereby improving the speed of inference.

[0202] Further, in the embodiment of the present application, the upsampling part of the image deblurring model uses bilinear interpolation, referring to formula (1). Where f(x, y) is the calculated interpolation, Q 11 =(x1,y1),Q 12=(x1,y2),Q 21 =(x2,y1) and Q 22 =(x2,y2) are the values ​​of four known pixels. The calculation process of bilinear interpolation has no parameters that need to be learned, so the inference speed is extremely fast.

[0203] Furthermore, in the embodiments of the present application, L1 Loss is used in model training. L1 Loss is also called Mean Absolute Error (MAE), which refers to the average value of the absolute difference between the model prediction value f(x) and the true value y, refer to formula (2). Where, f(x i ) and y i Respectively represent the predicted value and the corresponding true value of the i-th sample, that is, f(x i ) represents the deblurred image, y i represents the original image, n is the number of samples. The image format for input model training is unified as a three-channel grayscale JPG image of size 128×128.

[0204] It can be seen that the lightweight image deblurring solution proposed in the embodiment of the present application (including a method for constructing an image deblurring model and a method for processing an image deblurring model) can solve the problem of low image quality when such functions that rely on camera preview stream images need to be used at a long distance.

[0205] Furthermore, the lightweight image deblurring solution proposed in the embodiment of the present application includes an end-to-end neural network structure, which improves the representation ability of the model by using heavy parameterization and channel space attention modules, while also maintaining a low overall computational complexity of the model. Therefore, the lightweight image deblurring solution proposed in the embodiment of the present application can be applied to actual mobile phone applications, and has the characteristics of high precision and low power consumption.

[0206] In summary, the lightweight image deblurring solution proposed in the embodiment of the present application, on the one hand, designs a multi-degree image degradation method, which can simulate the problem of image blur in the preview stream caused by using a mobile phone camera at a long distance in a real environment. On the other hand, a new lightweight image deblurring model for mobile phones is designed. The model uses the channel space attention mechanism and the re-parameterization method to increase the representation ability of the model during the training process and improve the model calculation speed in the inference stage, thereby meeting the requirements of the mobile phone for low latency and low power consumption.

[0207] The embodiments of the present application provide a method for training an image deblurring model and a method for processing image deblurring. A training data set can be generated by using a multi-degree image degradation method for model training, so that the obtained image deblurring model has excellent representation capabilities. The image deblurring model includes a channel space attention module and a reparameterization module. The channel space attention mechanism and the reparameterization method can further improve the representation capabilities of the model. By performing image deblurring processing through the image deblurring model, a clearer image can be obtained, the problem of image blur can be solved, and the intelligence of the terminal device can be improved.

[0208] Based on the above embodiment, in another embodiment of the present application, Fig.11 A schematic diagram of the structure of the training device for the image deblurring model proposed in the embodiment of the present application is shown in FIG. Fig.11 As shown, the image deblurring model training device 110 proposed in the embodiment of the present application may include:

[0209] A generating unit 1101 is used to generate a degraded image corresponding to the original image by an image degradation method based on the original image; wherein the definition of the degraded image is lower than that of the original image; and to generate a training data set based on the original image and the degraded image;

[0210] The training unit 1102 is used to train the initial model based on the training data set to obtain the image deblurring model; wherein the initial model includes a channel space attention module and a first parameterization module, and the image deblurring model includes a channel space attention module and a second parameterization module corresponding to the first parameterization module.

[0211] In the embodiments of the present application, further, Fig.12 This is a schematic diagram of the composition structure of the image deblurring processing device proposed in the embodiment of the present application, as shown in FIG. Fig.12 As shown, the image deblurring processing device 120 proposed in the embodiment of the present application may include:

[0212] The acquisition unit 1201 is used to acquire the image to be processed; based on the image to be processed, a processed image corresponding to the image to be processed is obtained through an image deblurring model; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training based on a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed.

[0213] In the embodiments of the present application, further, Fig.13 This is a schematic diagram of the structure of the computer device proposed in the embodiment of the present application, such as Fig.13As shown, the computer device 130 proposed in the embodiment of the present application may include a processor 1301 , a memory 1302 , a communication interface 1303 , and a bus 1304 for connecting the processor 1301 , the memory 1302 , and the communication interface 1303 .

[0214] In the embodiment of the present application, the above-mentioned processor 1301 can be at least one of an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (ProgRAMmable Logic Device, PLD), a field programmable gate array (Field ProgRAMmable Gate Array, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and the embodiment of the present application is not specifically limited. The computer device 130 can also include a memory 1302, which can be connected to the processor 1301, wherein the memory 1302 is used to store executable program code, the program code includes computer operation instructions, and the memory 1302 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0215] In the embodiment of the present application, the bus 1304 is used to connect the communication interface 1303, the processor 1301 and the memory 1302, as well as the mutual communication between these devices.

[0216] In practical applications, the memory 1302 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 1301.

[0217] Further, in an embodiment of the present application, the processor 1301 is used to generate a degraded image corresponding to the original image through an image degradation method based on the original image; wherein the clarity of the degraded image is lower than that of the original image; generate a training data set based on the original image and the degraded image; train an initial model based on the training data set to obtain an image deblurring model; wherein the initial model includes a channel spatial attention module and a first re-parameterization module, and the image deblurring model includes a channel spatial attention module and a second re-parameterization module corresponding to the first re-parameterization module.

[0218] Furthermore, in an embodiment of the present application, the processor 1301 is used to obtain an image to be processed; based on the image to be processed, a processed image corresponding to the image to be processed is obtained through an image deblurring model; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained based on a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed.

[0219] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0220] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0221] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the training method of the image deblurring model as described above.

[0222] Specifically, the program instructions corresponding to the training method of an image deblurring model in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to the training method of an image deblurring model in the storage medium are read or executed by an electronic device, the following steps are included:

[0223] Based on the original image, a degraded image corresponding to the original image is generated by an image degradation method; wherein the definition of the degraded image is lower than that of the original image;

[0224] Generate a training dataset based on the original image and the degraded image;

[0225] An initial model is trained based on a training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first parameterization module, and the image deblurring model includes a channel space attention module and a second parameterization module corresponding to the first parameterization module.

[0226] Specifically, the program instructions corresponding to the image deblurring processing method in this embodiment may be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to the image deblurring processing method in the storage medium are read or executed by an electronic device, the following steps are included:

[0227] Get the image to be processed;

[0228] Through the image deblurring model, based on the image to be processed, a processed image corresponding to the image to be processed is obtained; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training based on a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed.

[0229] The embodiment of the present application also provides a computer program product.

[0230] In some embodiments, the computer program product may include a computer program or instructions.

[0231] In some embodiments, the computer program product can be applied to the computer device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the computer device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0232] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0233] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0234] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0235] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process described in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0236] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A training method for an image deblurring model, characterized in that: The method comprises: Based on the original image, generating a degraded image corresponding to the original image by an image degradation method; wherein the definition of the degraded image is lower than that of the original image; generating a training data set based on the original image and the degraded image; An initial model is trained based on the training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first re-parameterization module, and the image deblurring model includes the channel space attention module and a second re-parameterization module corresponding to the first re-parameterization module.

2. The method according to claim 1, wherein: The step of generating a degraded image corresponding to the original image by an image degradation method based on the original image includes: The original image is adjusted according to a preset adjustment method to obtain an adjusted image corresponding to the original image; wherein the preset adjustment method includes at least one or more of the following: horizontal flip adjustment, vertical flip adjustment, rotation adjustment, and skip adjustment; The adjusted image is degraded according to the image degradation method to obtain a degraded image corresponding to the original image; wherein the image degradation method includes at least one or more of the following: noise degradation, motion blur degradation, and tadpole-shaped degradation.

3. The method according to claim 2, wherein: The step of performing degradation processing on the adjusted image according to the image degradation method to obtain a degraded image corresponding to the original image includes: In the case where the first image is degraded according to the noise degradation, a first noise is determined in candidate noises; wherein the candidate noise includes at least one of the following: Gaussian noise, salt and pepper noise, Poisson noise, and no noise; the first image is one of the following: the adjusted image, the image after the motion blur degradation process, and the image after the tadpole-shaped degradation process; In a case where the first noise is Gaussian noise, salt and pepper noise, or Poisson noise, determining a noise parameter of the first noise according to a preset noise parameter range corresponding to the first noise; Based on the noise parameter, the first image is degraded according to the first noise to obtain the degraded image.

4. The method according to claim 2, wherein: The step of performing degradation processing on the adjusted image according to the image degradation method to obtain a degraded image corresponding to the original image includes: In the case where the second image is degraded according to the motion blur degradation, a motion blur angle parameter is determined in a preset angle parameter range, and a motion blur degree parameter is determined in a preset degree parameter range; wherein the second image is one of the following: the adjusted image, the image after the noise degradation process, and the image after the tadpole-shaped degradation process; The second image is degraded based on the motion blur angle parameter and the motion blur degree parameter to obtain the degraded image.

5. The method according to claim 2, wherein: The step of performing degradation processing on the adjusted image according to the image degradation method to obtain a degraded image corresponding to the original image includes: In the case where the third image is degraded according to the tadpole-shaped degradation, a first degradation mode is determined in candidate degradation modes; wherein the candidate degradation modes include at least one of the following: average pooling degradation, Gaussian blur degradation, and no degradation; the third image is one of the following: the adjusted image, the image processed by the noise degradation, and the image processed by the motion blur degradation; When the first degradation mode is average pooling degradation or Gaussian blur degradation, determining a degradation degree parameter corresponding to the first degradation mode; Determining a degradation parameter corresponding to the first degradation mode based on the degradation degree parameter; Based on the degradation parameter, the third image is degraded according to the first degradation mode to obtain the degraded image.

6. The method according to claim 1, wherein: The step of training an initial model based on the training data set to obtain an image deblurring model comprises: Obtaining a deblurred image corresponding to the degraded image based on the degraded image in the training data set by using the initial model; Determining a loss based on the original image and the deblurred image corresponding to the degraded image; The initial model is modified based on the loss to obtain the image deblurring model.

7. The method according to claim 6, characterized in that The step of correcting the initial model based on the loss to obtain the image deblurring model comprises: Modify the initial model based on the loss to obtain a modified model corresponding to the initial model; The first re-parameterized modules in the modified model are combined and optimized into the second re-parameterized modules to obtain the image deblurring model.

8. The method according to claim 6, characterized in that The initial model includes a downsampling module, an enhancement module and an upsampling module, wherein the downsampling module and the enhancement module both include the channel space attention module and the first re-parameterization module, and the deblurred image corresponding to the degraded image is obtained based on the degraded image in the training data set through the initial model, including: Obtaining, through the channel space attention module in the upsampling module, a first feature image corresponding to the degraded image based on the degraded image; Obtaining, by the first reparameterization module in the upsampling module, a second feature image corresponding to the degraded image based on the first feature image; Obtaining, by the channel space attention module in the enhancement module, a third feature image corresponding to the degraded image based on the second feature image; Obtaining, by the first re-parameterization module in the enhancement module, a fourth characteristic image corresponding to the degraded image based on the third characteristic image; The deblurred image is obtained based on the fourth feature image through the up-sampling module.

9. A method for image deblurring, characterized in that: The method comprises: Get the image to be processed; Through the image deblurring model, based on the image to be processed, a processed image corresponding to the image to be processed is obtained; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training based on a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed.

10. The method according to claim 9, characterized in that The image deblurring model includes a downsampling module, an enhancement module and an upsampling module, wherein the downsampling module and the enhancement module both include the channel space attention module and the second parameterization module, and the image deblurring model is used to obtain a processed image corresponding to the image to be processed based on the image to be processed, including: Obtaining, through the channel space attention module in the upsampling module, a first feature image corresponding to the image to be processed based on the image to be processed; Obtaining, through the second re-parameterization module in the up-sampling module, a second feature image corresponding to the image to be processed based on the first feature image; Obtaining, by the channel space attention module in the enhancement module, a third feature image corresponding to the image to be processed based on the second feature image; Obtaining, through the second parameterization module in the enhancement module, a fourth feature image corresponding to the image to be processed based on the third feature image; The processed image is obtained based on the fourth feature image through the up-sampling module.

11. A training device for an image deblurring model, characterized in that: The training device of the image deblurring model comprises: A generating unit, configured to generate a degraded image corresponding to the original image by an image degradation method based on the original image; wherein the definition of the degraded image is lower than that of the original image; and to generate a training data set based on the original image and the degraded image; A training unit is used to train an initial model based on the training data set to obtain an image deblurring model; wherein the initial model includes a channel space attention module and a first re-parameterization module, and the image deblurring model includes the channel space attention module and a second re-parameterization module corresponding to the first re-parameterization module.

12. An image deblurring processing device, characterized in that: The image deblurring processing device comprises: An acquisition unit is used to acquire an image to be processed; based on the image to be processed, a processed image corresponding to the image to be processed is obtained through an image deblurring model; wherein the image deblurring model includes a channel space attention module and a second parameterization module, and the image deblurring model is obtained by training based on a training data set, and the training data set is generated by an image degradation method; the clarity of the processed image is higher than that of the image to be processed.

13. A computer device, characterized in that: The computer device includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method according to any one of claims 1 to 8 or 9 to 10 is implemented.

14. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 8 or 9 to 10 is implemented.

Images