Deblurring method for blurred face images using latent coding library

Abstract

The present invention discloses a method for deblurring blurred facial images using a latent coding library, comprising: 1) downsampling the input blurred facial image, extracting a feature map of the corresponding resolution image and a latent code for each downsampling step, ultimately obtaining a latent coding matrix and feature maps of different resolutions; 2) layering the obtained latent coding matrix and feature maps of different resolutions into the latent coding library; 3) the latent coding library uses the feature map obtained by the final downsampling as the original input, continuously upsampling and convolving the latent coding style and the corresponding resolution feature map to achieve image reconstruction; 4) completing the image reconstruction of the latent coding library to form a high-resolution facial image that is highly similar to the input blurred facial image. The present invention can enhance the feature extraction of degraded images and their expressiveness in the reconstruction process, hierarchically decoupling the extraction of degraded features to facilitate the continuous decoupled application of features of corresponding scales in the hierarchical iterations of the reconstruction process.

Description

Technical Field

[0001] The present invention relates to the technical fields of image deblurring, image reconstruction and face deblurring, and in particular to a method for deblurring blurred face images using a latent coding library. Background Art

[0002] Image deblurring is the process of sharpening and restoring an image that has been degraded due to motion blur. It plays an important role in computer vision tasks. The most important part of the deblurring operation is to extract usable features from the blurred image and then reconstruct it.

[0003] In laboratory settings, image deblurring is typically accomplished by mathematically simulating the inverse of the convolution kernels of the blurred image. This method is theoretically supported by prior knowledge of the image gradient distribution. However, due to the randomness and uncertainty of motion blur, it requires excessively high iteration times and high precision in the number of convolution kernels. This results in significant errors and artifacts in practice. Subsequently, a new deblurring approach has been developed: extracting all features from the blurred image and reconstructing them based on these features. The reconstructed image theoretically possesses all the features of the blurred image while leveraging prior knowledge of the image to fill in the details of the remaining blurred image. However, the reconstruction performance of this method relies on the prior knowledge provided by the training set and the ability of the degraded image to express features in the restored image. As a result, the reconstructed image differs significantly from the original image due to feature expression and the coverage of the training set features, and restoration accuracy still needs to be improved. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose a deblurring method for blurring blurred facial images using a latent coding library. This method can enhance the feature extraction of degraded images and their expressiveness in the reconstruction process. At the same time, the extraction of degraded features is hierarchically decoupled to facilitate the continuous decoupled application of features of corresponding scales in the hierarchical iterations of the reconstruction process. This not only improves the feature expression of the original image, but also makes the expression of different features more independent during the reconstruction process rather than affecting the expression effect.

[0005] To achieve the above object, the present invention provides a technical solution: a method for deblurring a blurred face image using a latent coding library, comprising the following steps:

[0006] 1) Downsample the input blurred face image, extract the feature map and hidden code of the corresponding resolution image each time, and finally obtain the hidden code matrix and feature maps of different resolutions;

[0007] 2) The obtained latent coding matrix and feature maps of different resolutions are layered and passed into the latent coding library;

[0008] 3) The hidden coding library uses the final downsampled feature map as the original input, continuously upsampling and convolving the hidden coding style and the corresponding resolution feature map to achieve image reconstruction;

[0009] 4) Complete the image reconstruction of the latent coding library to form a high-resolution face image that is highly similar to the input blurred face image.

[0010] Furthermore, in step 1), the encoder performs N-level downsampling on the input blurred face image. First, the feature map of the blurred face image is extracted by RRDBNet, and then the resolution of the feature is gradually reduced by the following formula: i =E i (f i-1 ),i=1,2,3,...,N, where f i is the feature map extracted by RRDBNet at layer i; f i-1 is the feature map extracted by RRDBNet at layer i-1, with a resolution of f i 2*2 times; E i Represents a stack of convolution with a stride of 2 and a convolution with a stride of 1, which is a separate downsampling layer representation for RRDBNet. Finally, the hidden coding matrix is ​​generated through the convolution and fully connected layers. The hidden coding matrix is ​​expressed as: C = E N+1 (f N ), where f N is the feature map obtained by the Nth layer, E N+1 For the last layer, i.e. the N+1th downsampling layer, step 1) finally produces feature maps f of different resolutions i and the hidden encoding matrix C.

[0011] Furthermore, in step 2), the latent coding matrix is ​​C = (c0, c1, c2, ..., c p ,...,c N ),p=0,1,2,3,...,N, where c p represents the hidden code extracted by Np times downsampling of the pth column in C, which is the dimensionality reduction representation of the corresponding resolution image information, providing the hidden code library with high-dimensional information of features and style, helping it to restore the feature information of texture in the image reconstruction process; i f is the multi-scale convolution feature map from the original size of the input blurred face image to each resolution after i-layer downsampling; i Contains the feature information of the blurred face image itself, which is used to supervise and correct the feature expression of the image during reconstruction. The number of structural layers of the hidden coding library is the same as the number of downsampling times in step 1), and each layer requires a feature map of corresponding resolution f i And the latent code c extracted from the corresponding resolutionp As the feature information input of the image.

[0012] Further, in step 3), f N is the feature map obtained by the Nth layer, specifically the convolution feature map of the minimum resolution after the input blurred face image size is downsampled by N layers. It serves as the initial input of the latent coding library. The structural operation of the latent coding library is expressed as:

[0013]

[0014] Where S i Represents the hidden code library network structure from layer 1 to layer N, which is based on the style-based adversarial generative network; the hierarchical relationship of the network structure corresponds to the hierarchical structure of step 1); each layer contains the following four steps:

[0015] The first step is upsampling operation:

[0016] k(x,y)=k(i+u,j+v)

[0017] k(i+u,j+v)=(1-u)*(1-v)*k(i,j)+(1-u)*v*k(i,j+1)+u*(1-v)*k(i+1,j)+u*v*k(i+1,j+1)

[0018] Where k(x,y) represents the pixel value of the single channel at coordinate (x,y) after upsampling; (i,j) is the original coordinate before upsampling; u and v are the horizontal and vertical steps of the interpolation upsampling method; this step ultimately obtains features with a resolution twice that of the original face image;

[0019] In the second step, the style application module AdaIN, or instance normalization operation, is used to apply style and feature to the incoming image. The feature style information extracted by the latent code at the corresponding level is applied to the upsampled image, and finally a feature map containing the features contained in the latent code extracted at the corresponding level of resolution is obtained.

[0020] In the third step, a random noise adding module is used to add noise to the feature-added image, aiming to improve the fidelity of feature expression and reduce the artificial traces caused by feature expression when applied to the image;

[0021] The fourth step is to perform a convolution operation on the feature map passed in by the corresponding layer and the feature map generated after the style is applied, using the following formula:

[0022]

[0023] Where h[x,y] is the pixel value of the new image at (x,y) after convolution, gi-1 [x,y] is the pixel value at (x,y) of the feature map generated by the previous level, f N-i [x,y] is the pixel value at (x,y) of the feature map input at the current level, and n1 and n2 are the pixel values ​​of the single channel located around (x,y) for offset. By convolving the feature maps extracted from the input blurred face images of different resolutions, the aim is to supervise the style expression of the image and converge the basic pose generation of the image.

[0024] Back to the hidden code library network structure representation, the initial input S0 is represented as the network structure of layer 0. i The feature input generated by the previous block is missing; the 0th layer of the hidden coding library uses c0 as input, where c0 is the hidden coding matrix C = E N+1 (f N ) in the 0th column, perform adaptive instance normalization, and add random noise to randomize the image feature texture to enhance the realism of image reconstruction; finally, N Perform convolution operation to make the feature map supervise the style application operation, reduce the difference between the feature expression and the original image caused by the error, and obtain g0 as the input of the next layer, where g0 represents the feature generated by the first layer; g i represents the output feature of the i-th block, g i-1 The features generated by the previous block; the network structure of the i-th layer first generates g i-1 After upsampling to improve the feature resolution, the feature g generated by the previous network structure block is processed. i-1 and c p Perform convolution and instance normalization operations, where c p C = E N+1 (f N ) in the pth column, and the feature map f N-i Convolution of N-i is the feature map obtained by RRDBNet after Ni times of downsampling; until the final feature with the same resolution as the original face image is obtained, the instance normalization operation is the style-based generative adversarial network using hybrid regularization to perform explicit style control. This operation aims to generate the intersection of two latent codes in the trained latent code library, obtain the latent code and its feature expression after the two latent codes are regularized, and apply it to the feature map to be processed to generate the image after the feature style is applied.

[0025] Further, in step 4), g N As the feature map of the same resolution generated by the hidden coding library, after entering the decoder to perform channel conversion on the image, the leakyRelu activation function is used to delinearize the data. The formula is as follows:

[0026]

[0027] Where z is the value of each pixel under a single channel. After delinearization, the channel-transformed image is generated to obtain a high-resolution face image that is highly similar to the input blurred face image.

[0028] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0029] 1. The present invention adopts more than one original image feature information as input for image reconstruction for the first time, thereby improving the expression ability and expression accuracy of fuzzy image features in the image reconstruction process.

[0030] 2. Compared with traditional face deblurring methods, the present invention improves the feature accuracy and realism of reconstructed images.

[0031] 3. The method of the present invention has a wide range of applications in computer vision tasks, has low time complexity, and has good deblurring effect under the same number of training times. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a diagram of the architecture of the present invention.

[0033] Figure 2 This is a hierarchical structure diagram of the hidden code library of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0035] like Figure 1 and Figure 2 As shown, this embodiment provides a method for deblurring blurred face images using a latent coding library. The method uses an encoder-hidden coding library-decoder architecture and uses a style-based adversarial generative network as the network structure of the latent coding library to provide rich image prior knowledge and a traversable feature library for image reconstruction. The specific details are as follows:

[0036] 1) Downsample the input blurred face image, extract the feature map and hidden code of the corresponding resolution image each time, and finally obtain the hidden code matrix C and feature maps of different resolutions f i This step is reflected in Figure 1 In this step, the encoder performs N-level downsampling on the input blurred face image. First, RRDBNet extracts the feature map of the blurred face image input, and then gradually reduces the feature resolution by the following formula: i =E i (fi-1 ),i=1,2,3,...,N, where f i is the feature map extracted by RRDBNet at layer i; f i-1 is the feature map extracted by RRDBNet at layer i-1, with a resolution of f i 2*2 times; E i Represents a stack of convolution with a stride of 2 and a convolution with a stride of 1, which is a separate downsampling layer representation for RRDBNet. Finally, the hidden coding matrix is ​​generated through the convolution and fully connected layers. The hidden coding matrix is ​​expressed as: C = E N+1 (f N ), where f N is the feature map obtained by the Nth layer, E N+1 The last layer is the N+1th downsampling layer. This step ultimately produces two outputs, namely feature maps of different resolutions: f i And the hidden coding matrix C, these two outputs are Figure 1 The encoder part generates the output and sends it to Figure 1 The input passed by the hidden encoding library in . This step shows Figure 1 The role of the encoder is to encode the input image into low-dimensional information that describes the image feature information and then pass it to the post-structure.

[0037] 2) The obtained latent coding matrix and feature maps of different resolutions are layered and passed into the latent coding library. This step is mainly reflected in Figure 1 The encoder passes the latent coding matrix and feature map to the latent coding library according to the resolution. This step converts the latent coding matrix C obtained in step 1) into (c0, c1, c2, ..., c p ,...,c N ), p=0,1,2,3,...,N layers are passed into the hidden coding library, where c p represents the p-th column in C. The latent code extracted after Np times of downsampling in step 1) is a dimensionality-reduced representation of the image information of the corresponding resolution, providing the latent code library with high-dimensional information such as features and style, helping it to recover the feature information of texture prior during image reconstruction; i It is the multi-scale convolution feature map from the original size of the input blurred face image to each resolution after i-layer downsampling, f i Contains the characteristic information of the blurred face image itself, which is used to supervise and correct the characteristic expression of the image during the reconstruction process. The number of layers in the hidden coding library structure is the same as the number of downsampling in step 1), and each layer requires a feature map of corresponding resolution f i And the latent code c extracted from the corresponding resolution p As the feature information input of the image. The number of feature maps in this step is the same as the number of columns of the hidden code, which is the number of downsampling times in step 1), also Figure 1 The number of layers in the hidden coding library network structure is large, and the structure of such a network also satisfies the hierarchical correspondence when the encoder is transferred to the hidden coding library parameters.

[0038] 3) The hidden coding library takes the final down-sampled feature map as the original input, continuously up-sampling and convolution of the hidden coding style application and the corresponding resolution feature map to achieve image reconstruction. This step is reflected in Figure 1 In the structure of the latent coding library, the general single-layer operation process in the hierarchical structure of the latent coding library is as follows Figure 2 In this step, f N It is the feature map obtained by the Nth layer, specifically the convolution feature map of the input blurred face image size to the minimum resolution after N layers of downsampling, which serves as the initial input of the hidden coding library.

[0039] The structural operation of the latent coding library is expressed as:

[0040]

[0041] Where S i The hidden code library network structure from layer 1 to layer N is based on a style-based adversarial generative network. The hierarchical relationship of this network structure corresponds to the hierarchical structure of step 1). Each layer includes the following four steps:

[0042] The first step is upsampling operation:

[0043] k(x,y)=k(i+u,j+v)

[0044] k(i+u,j+v)=(1-u)*(1-v)*k(i,j)+(1-u)*v*k(i,j+1)+u*(1-v)*k(i+1,j)+u*v*k(i+1,j+1)

[0045] Where k(x,y) represents the pixel value of the single channel at coordinate (x,y) after upsampling; (i,j) is the original coordinate before upsampling; u and v are the horizontal and vertical steps of the interpolation upsampling method; this step ultimately obtains features with a resolution twice that of the original face image; this operation corresponds to Figure 2 Upsampling module;

[0046] In the second step, the style application module (AdaIN), also known as instance normalization, is used to apply style and feature information to the incoming image. The feature style information extracted by the latent code at the corresponding level is applied to the upsampled image, and the final result is a feature map containing the features contained in the latent code extracted at the corresponding level of resolution.

[0047] The third step is to add noise to the image with added features using the random noise adding module. This operation will only produce subtle random changes at the pixel level to the image features. This operation aims to improve the randomness of feature expression and reduce the artificial traces caused by the consistency of features when the feature expression is applied to the image, such as improving the randomness of the facial skin texture and hair posture stacking. The source of this random noise comes from Figure 2 The random noise module generates noise and appends it to the output feature map after each style application module.

[0048] The fourth step is to perform a convolution operation on the feature map passed in by the corresponding layer and the feature map generated after the style is applied, using the following formula:

[0049]

[0050] Where h[x,y] is the pixel value of the new image at (x,y) after convolution, g i-1 [x,y] is the pixel value at (x,y) of the feature map generated by the previous level, f N-i [x,y] is the pixel value at (x,y) of the feature map input at the current level, and n1 and n2 are the pixel values ​​of the single channel located around (x,y) for offset. By convolving the feature maps extracted from the input blurred face images of different resolutions, the aim is to supervise the style expression of the image and converge the basic pose generation of the image.

[0051] like Figure 2 As shown, the network structure of the i-th layer outputs the feature map g of the previous layer i-1 After upsampling and improving the feature resolution, the hidden code c is first p Perform style application operation, where c p C = E N+1 (f N ) in the pth column, that is, Figure 2 In the latent coding, random noise is added to the feature map after the style is applied to improve the fidelity of the feature expression, and then the feature map f is processed. N-i Convolution of N-i It is the feature map obtained by RRDBNet after Ni times of downsampling, that is, Figure 2 The feature map in the final output of this hierarchical structure is the feature map g after improving the resolution and feature application i , g i represents the output feature of the i-th block, g i-1 The features generated by the previous block; the network structure of the i-th layer first generates g i-1 After upsampling to improve the feature resolution, the feature g generated by the previous network structure block is processed.i-1 and c p Perform convolution and instance normalization operations, where c p C = E N+1 (f N ) in the pth column, and the feature map f N-i Convolution of N-i The feature map obtained by RRDBNet after Ni times of downsampling is iterated N times until the final output feature map g N It has the same resolution as the original image. The core of the style application module is the instance normalization operation, that is, the style-based generative adversarial network uses hybrid regularization to perform explicit style control; that is, by generating the intersection of two latent codes in the trained latent code library, the latent code and its feature expression after the regularization of the two latent codes are obtained.

[0052] More special is Figure 2 Cannot express correctly Figure 1 The structure of the first layer S0 of the latent coding library when receiving the parameters of the first layer of the encoder, S0 represents the network structure of the 0th layer. i Lack of feature map input generated by the previous block; layer 0 directly uses f N As the initial input, the upsampling step is skipped and the style is applied directly to c0, where c0 is the latent encoding matrix C = E N+1 (f N ), in the 0th column, random noise is added to randomize the image feature texture to enhance the realism of the image reconstruction. Finally, the feature map after random noise interference is output, and the output is Figure 2 The input of i=2.

[0053] 4) Complete the image reconstruction of the hidden code library to form a high-resolution face image with high similarity. This step is mainly reflected in Figure 1 The structure of the decoder in which g N The feature map of the same resolution generated by the latent coding library is the final output of the latent coding library. After entering the decoder part to perform channel conversion on the image, the leakyRelu activation function is used to delinearize the data and output the final image.

[0054]

[0055] Where z is the value of each pixel under a single channel. After delinearization, the channel-transformed image is generated to obtain a high-resolution face image that is highly similar to the input blurred face image.

[0056] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for deblurring a blurred face image using a latent coding library, characterized in that: The following steps are involved: 1) Downsample the input blurred face image, extract the feature map and hidden code of the corresponding resolution image each time, and finally obtain the hidden code matrix and feature maps of different resolutions, as follows: The encoder downsamples the input blurred face image at N levels. First, RRDBNet extracts the feature map of the blurred face image input, and then gradually reduces the feature resolution through the following formula: i =E i (f i-1 ),i=1,2,3,...,N, where f i is the feature map extracted by RRDBNet at layer i; f i-1 is the feature map extracted by RRDBNet at layer i-1, with a resolution of f i 2*2 times; E i Represents a stack of convolution with a stride of 2 and a convolution with a stride of 1, which is a separate downsampling layer representation for RRDBNet. Finally, the hidden coding matrix is ​​generated through the convolution and fully connected layers. The hidden coding matrix is ​​expressed as: C = E N+1 (f N ), where f N is the feature map obtained by the Nth layer, E N+1 For the last layer, i.e. the N+1th downsampling layer, step 1) finally produces feature maps f of different resolutions i And the hidden coding matrix C; 2) The obtained latent coding matrix and feature maps of different resolutions are layered and passed into the latent coding library; 3) The hidden coding library uses the final downsampled feature map as the original input, continuously upsampling and convolving the hidden coding style and the corresponding resolution feature map to achieve image reconstruction; f N is the feature map obtained by the Nth layer, specifically the convolution feature map of the minimum resolution after the input blurred face image size is downsampled by N layers. It serves as the initial input of the latent coding library. The structural operation of the latent coding library is expressed as: In the formula, S0 represents the network structure of the 0th layer, c0 is the 0th column in the hidden coding matrix C, g i represents the output feature of the i-th block, g i-1 The features generated by the previous block, f N-i is the feature map obtained by RRDBNet after Ni times of downsampling, S i Represents the network structure of the hidden code library from layer 1 to layer N, which is based on the style-based adversarial generation network; the hierarchical relationship of the network structure corresponds to the hierarchical structure of step 1); 4) Complete the image reconstruction of the latent coding library to form a high-resolution face image that is highly similar to the input blurred face image.

2. The method for deblurring a blurred face image using a latent coding library according to claim 1, wherein: In step 2), the hidden coding matrix is ​​C = (c0, c1, c2, ..., c p ,...,c N ),p=0,1,2,3,...,N, where c p represents the hidden code extracted by Np times downsampling of the pth column in C, which is the dimensionality reduction representation of the corresponding resolution image information, providing the hidden code library with high-dimensional information of features and style, helping it to restore the feature information of texture in the image reconstruction process; i f is the multi-scale convolution feature map from the original size of the input blurred face image to each resolution after i-layer downsampling; i Contains the feature information of the blurred face image itself, which is used to supervise and correct the feature expression of the image during reconstruction. The number of structural layers of the hidden coding library is the same as the number of downsampling times in step 1), and each layer requires a feature map of corresponding resolution f i And the latent code c extracted from the corresponding resolution p As the feature information input of the image.

3. The method for deblurring a blurred face image using a latent coding library according to claim 2, wherein: In step 3), each level of the hidden code library network structure includes the following four steps: The first step is upsampling operation: k(x,y)=k(i+u,j+v) k(i+u,j+v)=(1-u)*(1-v)*k(i,j)+(1-u)*v*k(i,j+1) +u*(1-v)*k(i+1,j)+u*v*k(i+1,j+1) Where k(x,y) represents the pixel value of the single channel at coordinate (x,y) after upsampling; (i, j) is the original coordinate before upsampling; u and v are the horizontal and vertical steps of the interpolation upsampling method; the upsampling step ultimately obtains features with a resolution twice that of the original face image; In the second step, the style application module AdaIN, or instance normalization operation, is used to apply style and feature to the incoming image. The feature style information extracted by the latent code at the corresponding level is applied to the upsampled image, and finally a feature map containing the features contained in the latent code extracted at the corresponding level of resolution is obtained. In the third step, a random noise adding module is used to add noise to the feature map of the added features, aiming to improve the fidelity of the feature expression and reduce the artificial traces caused by the feature expression when applied to the image; The fourth step is to perform a convolution operation on the feature map passed in by the corresponding layer and the feature map generated after the style is applied, using the following formula: Where h[x,y] is the pixel value of the new image at (x,y) after convolution, g i-1 [x,y] is the pixel value at (x,y) of the feature map generated by the previous level, f N-i [x,y] is the pixel value at (x,y) of the feature map input at the current level, and n1 and n2 are the pixel values ​​of the single channel located around (x,y) for offset. By convolving the feature maps extracted from the input blurred face images of different resolutions, the aim is to supervise the style expression of the image and converge the basic pose generation of the image. Back to the hidden code library network structure representation, the initial input S0 is represented as the network structure of layer 0. i Lack of feature input generated by the previous block; The 0th layer of the hidden coding library uses c0 as input, where c0 is the hidden coding matrix C = E N+1 (f N ) in the 0th column, perform adaptive instance normalization, and add random noise to randomize the image feature texture to enhance the realism of image reconstruction; finally, N Perform convolution operation to make the feature map supervise the style application operation, reduce the difference between the feature expression and the original image caused by the error, and obtain g0 as the input of the next layer, where g0 represents the feature generated by the first layer; g i represents the output feature of the i-th block, g i-1 Features generated for the previous block; The network structure of the i-th layer first generates g i-1 After upsampling to improve the feature resolution, the feature g generated by the previous network structure block is processed. i-1 and c p Perform convolution and instance normalization operations, and perform feature map f N-i Convolution, where c p C = E N+1 (f N ) in the pth column; f N-i is the feature map obtained by RRDBNet after Ni times of downsampling; until the final feature with the same resolution as the original face image is obtained, the instance normalization operation is that the style-based generative adversarial network uses a hybrid regularization method to perform explicit style control. The instance normalization operation aims to generate the intersection of two latent codes in the trained latent code library, obtain the latent code and its feature expression after the two latent codes are regularized, and apply it to the feature map to be processed to generate an image after the feature style is applied.

4. The method for deblurring a blurred face image using a latent coding library according to claim 3, wherein: In step 4), g N As the feature map of the same resolution generated by the hidden coding library, after entering the decoder to perform channel conversion on the image, the leakyRelu activation function is used to delinearize the data. The formula is as follows: Where z is the value of each pixel under a single channel. After delinearization, the channel-transformed image is generated to obtain a high-resolution face image that is highly similar to the input blurred face image.

Images