Low-illumination face image super-resolution reconstruction method, device and equipment
By using shallow feature extraction, spatial feature transformation and dedegradation color correction modules of the image reconstruction model under low light conditions, the problem of image details or structural distortion during the reconstruction of low-light faces is solved, and high-quality super Resolution reconstruction.
Patent Information
- Application Number
- CN202411460404.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-10-18
AI Technical Summary
Under low light conditions, distortions in image details or structures are prone to occur during the reconstruction of face images, resulting in a degradation of subsequent processing tasks.
A low-light face image super-resolution reconstruction method is adopted to reconstruct the model using pre-trained image, including shallow feature extraction module, spatial feature transformation module and dedegradation color correction module, through convolution processing, multi-scale feature extraction and Iterative color correction to restore high-resolution face images and reduce the degree of distortion of image details or structure.
It effectively reduces the distortion of image details or structure during image reconstruction, maintains the stability of facial structure, restores texture details, and improves the lighting recovery and color correction performance of image reconstruction model.
Smart Images

Figure CN118967451A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image enhancement technology, and in particular to a method, device and equipment for super-resolution reconstruction of low-light facial images. Background Art
[0002] Facial image super-resolution technology is usually based on facial prior knowledge to restore clear and realistic facial details. However, due to factors such as low light and shadow interference in the external environment, facial images acquired under low light conditions cannot provide accurate geometric prior and texture information. In addition, during the transmission and propagation of facial images, severe degradation will also destroy the structure and details of the reproduced high-resolution facial images, thereby reducing the performance of super-resolution. In more complex actual scenarios, images are subject to various degradations such as noise and artifacts, and the dimensionality reduction process may become more complicated. In particular, facial images acquired in low-light scenes often have low brightness, and the reconstruction process is prone to distortion of image details or structures, thereby reducing the performance of subsequent processing tasks. Summary of the invention
[0003] In view of this, the present invention provides a method, device and equipment for super-resolution reconstruction of low-light facial images to solve the problem of image detail or structure distortion in the process of reconstructing facial images under low light.
[0004] In a first aspect, the present invention provides a method for super-resolution reconstruction of low-light facial images, the method comprising: Obtaining a low-light and low-resolution face image to be reconstructed; The low-light low-resolution face image is input into the pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein: The shallow feature extraction module is used to perform convolution processing on low-light and low-resolution face images to obtain shallow features; The spatial feature transformation module is used to transform the shallow features to obtain multi-scale feature information; The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on multi-scale feature information to obtain a super-resolution face image.
[0005] The low-light facial image super-resolution reconstruction method provided by the present invention uses a spatial feature transformation module to restore brightness and a de-degradation color correction module to perform color correction, thereby reducing the degree of distortion of image details or structures during the facial image reconstruction process, and can both maintain the stability of the facial structure and restore texture details.
[0006] In an optional implementation, the training process of the image reconstruction model includes: Randomly obtain multiple original high-resolution face images from a preset public dataset; Adding a preset darkness factor and noise to each original high-resolution face image to obtain a low-light low-resolution face image corresponding to the original high-resolution face image; An image pair consisting of a low-light low-resolution face image and a corresponding original high-resolution face image is used as a sample, a preset neural network model is trained, and the trained model is used as an image reconstruction model.
[0007] The low-light facial image super-resolution reconstruction method provided by the present invention uses a darkness factor to control the illumination intensity of high-resolution facial images in a public data set, reduces the resolution by adding noise, and obtains a low-light low-resolution-high-resolution image pair, thereby solving the problem of lack of public data on low-light low-resolution facial images. The image reconstruction model obtained by training has a better effect of super-resolution reconstruction.
[0008] In an optional implementation, performing feature changes on shallow features to obtain multi-scale feature information includes: Input shallow features into multiple serially connected encoders and encode them multiple times in sequence to obtain multiple encoded feature maps; Multiple decoders connected in series are used to upsample multiple encoded feature maps to extract multi-scale feature information of low-light and low-resolution face images. The number of decoders is the same as that of encoders.
[0009] The low-light facial image super-resolution reconstruction method provided by the present invention utilizes the same number of tandem encoders and tandem decoders to gradually restore illumination based on shallow features, extract multi-scale feature information, improve the image reconstruction model's ability to represent features at different scales, and improve the model's ability to restore image illumination.
[0010] In an optional implementation, the shallow features are input into an encoder for encoding to obtain an encoded feature map, including: The shallow features are input into multiple branches with different sampling scales, and multiple semantic information of the shallow features are extracted by downsampling. After the spatial and channel reconstruction convolution module and the multi-dimensional collaborative attention mechanism module, the output features of each branch are obtained. Sort the output features of branches with different sampling scales from low to high according to the sampling scale; Upsample the output features of the branch with the lowest sampling scale by a preset multiple and use them as the fusion features of the current branch; The fusion features of the current branch are upsampled by a preset multiple in sequence, and fused with the output features of the next high sampling scale branch adjacent to the current branch to obtain a fusion result, where the sampling scale of the next high sampling scale branch is a preset multiple of the sampling scale of the current branch; The fusion result is convolved with the shallow features for residual processing, and then convolved again to obtain the encoded feature map.
[0011] The low-light facial image super-resolution reconstruction method provided by the present invention utilizes spatial and channel reconstruction convolution modules to suppress redundant spatial information and amplify the weight of key feature information, and utilizes a multi-dimensional collaborative attention mechanism module to capture the interdependencies in the spatial dimension and the interactions between channels, thereby significantly improving the model's ability to capture important facial structure pixels.
[0012] In an optional implementation, the encoder includes three branches with different sampling scales, the sampling scale of the first branch is 8 times, the sampling scale of the second branch is 4 times, and the sampling scale of the third branch is 2 times; The fusion features of the current branch are upsampled by a preset multiple in sequence, and fused with the output features of the next high-sampling scale branch adjacent to the current branch to obtain the fusion result, including: The output features of the first branch are upsampled by 2 times to obtain the first fusion features; The first fusion feature and the output feature of the second branch are fused to obtain a second fusion feature; The second fused feature is upsampled by 2 times and fused with the output feature of the third branch to obtain the third fused feature as the fusion result.
[0013] In the low-light facial image super-resolution reconstruction method provided by the present invention, the sampling scales between the branches differ by two times, and the sampling multiple when the fusion features corresponding to the branches are up-sampled is also two times, so that the features of adjacent branches have the same dimension when they are fused, which facilitates data processing and improves the efficiency of feature fusion.
[0014] In an optional implementation, the process of upsampling the fused features of the current branch by a preset multiple and fusing them with the output features of the next high sampling scale branch adjacent to the current branch includes: The fusion feature of the current branch is used as the first feature, the output feature of the next branch is used as the second feature, the first feature and the second feature are respectively input into the preset channel attention module to obtain the first channel output and the second channel output; The output of the first channel is multiplied by the output of the second channel to obtain a cross matrix; Performing a residual connection between the cross matrix and the first feature to obtain a first spatial input feature, and performing a residual connection between the cross matrix and the second feature to obtain a second spatial input feature; The first spatial input feature and the second spatial input feature are input into a preset spatial attention module to obtain a feature after spatial attention adjustment, and the feature after spatial attention adjustment is input into a fully connected layer to obtain a fusion feature.
[0015] The low-light facial image super-resolution reconstruction method provided by the present invention utilizes a channel attention module and a spatial attention module to perform feature fusion of adjacent branches, while taking into account the spatial layout of facial images and the importance of channel features, thereby improving the image processing performance of the model.
[0016] In an optional implementation, color correction and de-degradation processing are iteratively performed on multi-scale feature information to obtain a super-resolution face image, including: The multi-scale feature information is processed by self-attention mechanism and forward feedback to obtain the color information in the multi-scale feature information, and the multi-scale feature information is interpolated and convolved to obtain the degraded features; The color information and degradation features are respectively input into the de-degradation color correction module for iterative processing to obtain a super-resolution face image.
[0017] The low-light facial image super-resolution reconstruction method provided by the present invention uses an iterative approach to achieve complementary learning enhancement of color correction and de-degradation super-resolution, effectively processes facial images under low-light conditions, and achieves high-quality super-resolution reconstruction.
[0018] In an optional implementation, the de-degradation color correction module includes a plurality of correction submodules connected in series, the number of the correction submodules is the number of iterations, and the correction submodules include a color estimation module, a super-resolution module, a color adjustment module, and a color refinement module; The color information and degradation features are respectively input into the de-degradation color correction module for iterative processing to obtain a super-resolution face image, including: Input the color information into the color estimation module of the current correction submodule to obtain the color coefficient, and input the degradation feature into the super-resolution module of the current correction submodule to obtain the de-degraded feature; Inputting the color coefficient and the de-degradation feature into the color adjustment module of the current correction submodule, and performing color adjustment on the de-degradation feature using the color coefficient to obtain a color adjustment result; Input the color adjustment result and the color coefficient into the color refinement module of the current correction submodule to obtain the color refinement result; The color refinement result is input as the updated color information into the color estimation module of the next correction submodule connected in series with the current correction submodule, and the color adjustment result is input as the updated degradation feature into the super-resolution module of the next correction submodule. The iterative processing is performed in this way, and the color adjustment result output by the last correction submodule is used as the super-resolution face image.
[0019] The low-light facial image super-resolution reconstruction method provided by the present invention utilizes a color estimation module to process color information to obtain a color coefficient, utilizes a super-resolution module to process degradation features to obtain a de-degraded feature, utilizes the color coefficient to adjust the de-degraded feature, thereby improving the super-resolution of the image in the color adjustment result, and feeds the color adjustment result back to the color refinement module to perform more precise complementary refinement on the color coefficient, thereby improving the model color correction and de-degradation performance, and obtaining a clearer super-resolution facial image.
[0020] In a second aspect, the present invention provides a low-light facial image super-resolution reconstruction device, the device comprising: An image acquisition module, used for acquiring a low-light and low-resolution face image to be reconstructed; An image reconstruction module is used to input the low-light low-resolution face image into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; wherein the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein: The shallow feature extraction module is used to perform convolution processing on low-light and low-resolution face images to obtain shallow features; The spatial feature transformation module is used to perform feature transformation on the shallow features to obtain multi-scale feature information; The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution face image.
[0021] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0022] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to execute the method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0024] Figure 1is a schematic flow chart of a method for super-resolution reconstruction of a low-light face image according to an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of an image reconstruction model in a method for super-resolution reconstruction of a low-light face image according to an embodiment of the present invention; Figure 3 is a flowchart of a process of processing shallow features by a spatial feature transformation module in a method for super-resolution reconstruction of a low-light facial image according to an embodiment of the present invention; Figure 4 is a structural schematic diagram of a de-degradation color correction module in a low-light facial image super-resolution reconstruction method according to an embodiment of the present invention; Figure 5 is a schematic diagram of the internal structure of an encoder in a method for super-resolution reconstruction of a low-light face image according to an embodiment of the present invention; Figure 6 is a structural schematic diagram of a specific example of an encoder in a method for super-resolution reconstruction of a low-light face image according to an embodiment of the present invention; Figure 7 2 is a schematic diagram of the structure of a cross-attention module preset inside an encoder in a method for super-resolution reconstruction of a low-light face image according to an embodiment of the present invention; Figure 8 is a structural schematic diagram of a Transformer block of a de-degradation color correction module in a low-light facial image super-resolution reconstruction method according to an embodiment of the present invention; Fig. 9 is a schematic structural diagram of another de-degradation color correction module in the low-light facial image super-resolution reconstruction method according to an embodiment of the present invention; Fig.10 is a schematic structural diagram of a super-resolution module in a de-degradation color correction module in a method for super-resolution reconstruction of a low-light facial image according to an embodiment of the present invention; Fig.11 is a structural schematic diagram of a color adjustment module in a de-degradation color correction module in a low-light facial image super-resolution reconstruction method according to an embodiment of the present invention; Fig.12 is a structural schematic diagram of a color refinement module in a de-degradation color correction module in a low-light facial image super-resolution reconstruction method according to an embodiment of the present invention; Fig.13 is a structural block diagram of a low-light facial image super-resolution reconstruction device according to an embodiment of the present invention; Fig.14 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0026] The embodiment of the present invention provides a low-light facial image super-resolution reconstruction method, which is suitable for early facial image reconstruction of downstream tasks such as face recognition and expression recognition, and enhances the actual scene application capability of downstream tasks.
[0027] According to an embodiment of the present invention, an embodiment of a method for super-resolution reconstruction of low-light facial images is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.
[0028] In this embodiment, a low-light facial image super-resolution reconstruction method is provided, which can be used in the above-mentioned computer system. Figure 1 FIG. 1 is a flow chart of a method for super-resolution reconstruction of a low-light face image according to an embodiment of the present invention. Figure 1 As shown, the method includes: Step S1: Obtain a low-light and low-resolution face image to be reconstructed.
[0029] Specifically, the low-light and low-resolution facial image to be reconstructed is a facial image in a low-light environment captured by an image capture device before performing face recognition or expression recognition.
[0030] Step S2: input the low-light low-resolution face image to be reconstructed into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image.
[0031] Figure 2 is a schematic diagram of the structure of an image reconstruction model in a low-light facial image super-resolution reconstruction method according to an embodiment of the present invention. Figure 2 As shown, the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module.
[0032] The shallow feature extraction module is used to perform convolution processing on low-light and low-resolution face images to obtain shallow features.
[0033] Specifically, shallow features refer to low-level features extracted from the neural network, which are usually located in the early stages of the network and are closer to the input data. Shallow features include fine-grained information such as color, texture, edge, and angle. The shallow feature extraction module uses a common 3×3 convolutional layer for feature extraction. The input of the shallow feature extraction module is a low-light, low-resolution face image. , where C, H, and W represent the channel, height, and width of the input face image, respectively. The shallow feature extraction module first standardizes and normalizes the input face image, and then performs ordinary convolution operations to reduce the complexity of the model and reduce the computing resource consumption in the deep feature extraction stage. The network structure formula of the shallow feature extraction module is: ,in, Represents the shallow features obtained by the shallow feature extraction module of the low-light and low-resolution face image.
[0034] The spatial feature transformation module is used to transform shallow features to obtain multi-scale feature information.
[0035] Specifically, the deep features are processed by multi-layer networks and have a large receptive field and rich semantic information. In the deep feature extraction stage, the spatial feature transformation module (SFTM), de-degraded color correction module (DCCM) and self-attention mechanism module (SA) are used to enhance the shallow feature information extracted by the shallow feature extraction module. Specifically, SFTM changes the spatial structure and channel characteristics of the input shallow features to enhance or suppress certain spatial features; SA automatically weights the input shallow features to strengthen important information parts and ignore redundant content; DCCM combines the output of SFTM and SA to correct color distortion in the image and improve color authenticity and consistency. The network structure formula for the deep feature extraction stage is: ,in, Represents the deep features after deep feature extraction.
[0036] like Figure 3 As shown in the figure, the spatial feature transformation module processes shallow features by inputting shallow features into multiple serial encoders, and then upsampling through the same number of decoders to gradually restore the image illumination, and by using activation functions and residual connections, avoid the loss of illuminated pixels. The entire spatial feature transformation module is implemented through an encoder-decoder architecture, which not only improves the model's ability to represent features at different scales, but also significantly enhances the model's ability to restore image illumination.
[0037] The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on multi-scale feature information to obtain a super-resolution face image.
[0038] Specifically, Figure 4As shown in the figure, the de-degradation color correction module is a dual-branch structure, including: a color correction branch and a de-degradation branch. The color correction branch uses a Transformer block to encode the color information, and the de-degradation branch convolves the multi-scale feature information after the bicubic linear interpolation operation to obtain the degradation feature. The de-degradation color correction module includes multiple correction submodules. The color information and the degradation feature are used as the two inputs of the first correction submodule. Multiple iterative processes are performed in multiple serial correction submodules to obtain a super-resolution face image.
[0039] The low-light facial image super-resolution reconstruction method provided in this embodiment uses a spatial feature transformation module to restore brightness and a de-degradation color correction module to perform color correction, thereby reducing the degree of distortion of image details or structures during the facial image reconstruction process, thereby maintaining the stability of the facial structure and restoring texture details.
[0040] In some optional implementations, the training process of the image reconstruction model includes: Step a1: randomly obtain multiple original high-resolution face images from a preset public dataset.
[0041] Specifically, due to the lack of public datasets for low-light face super-resolution tasks, simulation experiments are conducted on existing public face datasets to simulate low-light, low-resolution face images in real scenes. The preset public datasets include but are not limited to: CelebA-HQ and FFHQ datasets, which include a large number of high-resolution face images, from which 20,000 high-resolution face images are randomly selected for simulation experiments.
[0042] Step a2, adding a preset darkness factor and noise to each original high-resolution face image to obtain a low-light low-resolution face image corresponding to the original high-resolution face image.
[0043] Specifically, after selecting the picture, the original high-resolution face image is subjected to low-light operation. The specific process is to control the light intensity through a dark factor to realize low-light simulation, and set the value range of the dark factor to [0,1]. When simulating low light, the value of the dark factor is fixed in the range of [0.5,0.9]. The larger the value of the dark factor, the lower the light intensity. In addition, noise is added to the face image through the degradation model: mainly including five degradation models: motion blur, Gaussian noise, Rayleigh noise, salt and pepper noise, and mean blur. The specific process of adding noise is a mature existing technology and will not be repeated here. Finally, a low-light and low-resolution face image corresponding to the original high-resolution face image is obtained.
[0044] Step a3: Use an image pair consisting of a low-light low-resolution face image and a corresponding original high-resolution face image as a sample, train a preset neural network model, and use the trained model as an image reconstruction model.
[0045] Specifically, a dataset consisting of image pairs consisting of low-light low-resolution face images and corresponding original high-resolution face images is used as a sample dataset to train a preset neural network model. The structure of the preset neural network model is a spatial feature transform network (Spatial Transformer Networks, STNs) for example only, but not limited to this, to obtain an image reconstruction model. The model training process is a mature existing technology and will not be repeated here.
[0046] The low-light facial image super-resolution reconstruction method provided in this embodiment uses a darkness factor to control the light intensity of high-resolution facial images in a public dataset, and reduces the resolution by adding noise to obtain a low-light low-resolution-high-resolution image pair, thereby solving the problem of lack of public data on low-light low-resolution facial images. The trained image reconstruction model has better effect in super-resolution reconstruction.
[0047] In some optional embodiments, such as Figure 3 As shown in the figure, the shallow features are transformed to obtain multi-scale feature information, including: Step S11, input the shallow features into multiple serially connected encoders for multiple encodings in sequence to obtain multiple encoded feature maps.
[0048] Specifically, the shallow features are sequentially input into the serial encoders. The input of the first encoder is the shallow features. Starting from the next encoder connected to the first encoder, the input of the next encoder is the output of the previous encoder. Based on this, each encoder outputs a feature map, and multiple encoded feature maps are obtained, which can be expressed as follows: ,in represents the i-th encoder, Indicates The feature map output by the encoder.
[0049] Step S12, using multiple serially connected decoders to upsample the multiple encoded feature maps to extract multi-scale feature information of low-light and low-resolution facial images, and the number of decoders is the same as the number of encoders.
[0050] Specifically, each encoder corresponds to a decoder, and the feature maps output by each corresponding encoder are upsampled by the same number of decoders, gradually restoring the illumination of the image and avoiding the loss of illumination pixels by using the Sigmoid function and residual connection, thus extracting the multi-scale feature information of low-light and low-resolution face images. The formula is expressed as: , where Sigmoid represents the activation function, Indicates The feature map output by the decoder is Indicates The decoder is The feature map output by the encoder and the The feature map output by the encoder is decoded.
[0051] The low-light facial image super-resolution reconstruction method provided in this embodiment utilizes the same number of tandem encoders and tandem decoders to gradually restore illumination based on shallow features and extract multi-scale feature information, thereby improving the image reconstruction model's ability to represent features at different scales and improving the model's ability to restore image illumination.
[0052] In some optional embodiments, such as Figure 5 As shown in the figure, the internal structure of the encoder is shown. The shallow features are input into the encoder for encoding to obtain the encoded feature map, including: In step S111, the shallow features are input into multiple branches with different sampling scales, and multiple semantic information of the shallow features are extracted by downsampling respectively. After the spatial and channel reconstruction convolution module and the multi-dimensional collaborative attention mechanism module, the output features of each branch are obtained.
[0053] Specifically, each encoder has the same internal structure, and has multiple branches with different sampling scales. For the input features of each encoder, downsampling with different sampling scales is performed in each branch to extract the semantic information of the input features. Then, the Spatial and Channel Reconstruction Convolution for Feature Redundancy (SCConv) module is used to suppress redundant spatial information and amplify the weight of key feature information. The multi-dimensional collaborative attention mechanism module (MCA) captures the interdependence in the spatial dimension and the interaction between channels to obtain the output features of each branch. The output features of each branch are expressed as: ,in, Indicates that the input features are The semantic information obtained by downsampling by times, The sampling scale is The output features of the branch.
[0054] Step S112, sorting the output features of branches with different sampling scales from low to high according to the sampling scale.
[0055] Specifically, the output features of branches with different sampling scales are sorted from low to high according to the sampling scale. For example, the sampling scales of each branch are: 2 times, 4 times, 8 times, 16 times, 32 times, and 64 times, respectively. This is only used as an example, but is not limited to this. Due to downsampling, the sampling scales of each branch are sorted from low to high as follows: 64 times, 32 times, 16 times, 8 times, 4 times, and 2 times. Multiple branches extract features of different scales, and the sampling scales between adjacent branches differ by two times, which is convenient for subsequent feature fusion. Preferably, three branches with sampling scales of 8 times, 4 times, and 2 times can be selected through experimental verification.
[0056] Step S113, upsampling the output features of the lowest sampling scale branch by a preset multiple to serve as the fusion features of the current branch.
[0057] Specifically, the output features of the lowest sampling scale branch are upsampled by a preset multiple as the fused features of the current branch. For the lowest sampling scale branch, its output features are not fused, and its upsampling result is equivalent to the fused features of the lowest sampling scale branch.
[0058] Step S114, upsampling the fusion features of the current branch by a preset multiple in sequence, and fusion with the output features of the next high sampling scale branch adjacent to the current branch to obtain a fusion result, wherein the sampling scale of the next high sampling scale branch is a preset multiple of the sampling scale of the current branch.
[0059] Specifically, Figure 5 As shown, for the output features of multiple branches of different scales, it is necessary to upsample the fused features of the current branch by a preset multiple in turn, and then fuse them with the output features of the next high sampling scale branch to obtain the fused features of the next high sampling scale branch. The fused features of the last high sampling scale branch are used as the fusion result of the output features of each branch, and the sampling scale of the next high sampling scale branch is a preset multiple of the sampling scale of the current branch.
[0060] In step S115, the fusion result is convolved with the shallow features and residual processing is performed, and convolution processing is performed again to obtain the encoded feature map.
[0061] Specifically, the fusion result is passed through a convolutional layer and then residually connected with the input feature of the encoder, and the fusion result is passed through another convolutional layer to obtain the final output as the encoded feature map.
[0062] So the The output feature map of the encoder is expressed as: ,in, Indicates The input feature map of the encoder is Indicates The fused features of the highest sampling scale branch in the encoder.
[0063] The low-light facial image super-resolution reconstruction method provided in this embodiment utilizes the spatial and channel reconstruction convolution modules to suppress redundant spatial information and amplify the weights of key feature information, and utilizes the multi-dimensional collaborative attention mechanism module to capture the interdependencies in the spatial dimension and the interactions between channels, thereby significantly improving the model's ability to capture important facial structure pixels.
[0064] In some optional embodiments, such as Figure 6 As shown, the encoder includes three branches with different sampling scales, the sampling scale of the first branch is 8 times, the sampling scale of the second branch is 4 times, and the sampling scale of the third branch is 2 times.
[0065] Step S114 sequentially upsamples the fusion features of the current branch by a preset multiple, and fuses them with the output features of the next high sampling scale branch adjacent to the current branch to obtain a fusion result, including: Step b1, upsampling the output features of the first branch by a factor of 2 to obtain the first fusion features.
[0066] Specifically, the output features of the first branch are: , and upsample it by 2 times to get the first fusion feature: .
[0067] Step b2: Fusing the first fused feature with the output feature of the second branch to obtain a second fused feature.
[0068] Specifically, the first fusion feature and the output feature of the second branch are fused using a preset cross attention module to obtain the second fusion feature: ,in Represents the output features of the second branch: , represents the second fusion feature, Represents a new cross-attention module designed based on the Channel Attention Mechanism (CAM) and Spatial Attention Module (SAM) in the plug-and-play attention module (Convolutional Block Attention Module, CBAM).
[0069] Step b3, upsampling the second fused feature by 2 times, and fusing it with the output feature of the third branch to obtain the third fused feature as the fusion result.
[0070] Specifically, the second fusion feature and the output feature of the third branch are fused using a preset cross attention module to obtain the third fusion feature: ,in Represents the output features of the second branch: , represents the third fusion feature, It represents a new cross attention module designed based on CAM and SAM in CBAM. For an encoder with only three branches, the third fused feature of the third branch is the fusion result.
[0071] In the low-light facial image super-resolution reconstruction method provided by this embodiment, the sampling scales between the branches differ by two times, and the sampling multiple when the fusion features corresponding to each branch are upsampled is also doubled, so that the features of adjacent branches have the same dimension when they are fused, which facilitates data processing and improves the efficiency of feature fusion.
[0072] In some optional implementations, in step S114, the process of upsampling the fused features of the current branch by a preset multiple and fusing them with the output features of the next high sampling scale branch adjacent to the current branch includes: Step S1141, taking the fusion feature of the current branch as the first feature, taking the output feature of the next branch as the second feature, inputting the first feature and the second feature into the preset channel attention module respectively, and obtaining the first channel output and the second channel output.
[0073] Specifically, Figure 7 As shown in the figure, the internal structure of the preset cross attention module is shown. For the preset cross attention module CAFM in the encoder, it includes two main modules: the preset channel attention module (CAM) and the preset spatial attention module (SAM). Its input is the fusion feature of the current branch and the output feature of the next branch, so the fusion feature of the current branch is used as the first feature. , the output feature of the next branch is used as the second feature , the first feature and the second feature are input into the preset channel attention module respectively to obtain the first channel output and the second channel output. The shapes of the first channel output and the second channel output are , the channel output is expressed as: ,in Indicates aisle.
[0074] Step S1142: multiply the first channel output and the second channel output to obtain a cross matrix.
[0075] Specifically, the first channel output is multiplied by the second channel output to obtain a shape of The cross matrix is: ,in, , They are the first channel output and the second channel output respectively. Step S1143, performing a residual connection on the cross matrix and the first feature to obtain a first spatial input feature, and performing a residual connection on the cross matrix and the second feature to obtain a second spatial input feature.
[0076] Specifically, the cross matrix is residually connected with the first feature to obtain the first spatial input feature: , perform residual connection between the cross matrix and the second feature to obtain the second spatial input feature: .
[0077] Step S1144, input the first spatial input feature and the second spatial input feature into a preset spatial attention module to obtain a feature after spatial attention adjustment, and input the feature after spatial attention adjustment into a fully connected layer to obtain a fusion feature.
[0078] Specifically, the first spatial input feature and the second spatial input feature are input into a preset spatial attention module to obtain the feature after spatial attention adjustment: , the features adjusted by spatial attention are input into the fully connected layer to obtain the fused features: ,in, represents the preset weight adjustment matrix, Represents the bias term.
[0079] The low-light facial image super-resolution reconstruction method provided in this embodiment utilizes a channel attention module and a spatial attention module to fuse features of adjacent branches, while taking into account the spatial layout of facial images and the importance of channel features, thereby improving the image processing performance of the model.
[0080] In some optional embodiments, such as Figure 3 As shown, the multi-scale feature information is iteratively processed for color correction and de-degradation to obtain a super-resolution face image, including: Step S21, performing self-attention mechanism and forward feedback processing on the multi-scale feature information to obtain color information in the multi-scale feature information, and performing interpolation and convolution processing on the multi-scale feature information to obtain degraded features.
[0081] Specifically, the de-degradation color correction module has a dual-branch structure, including: a color correction branch and a de-degradation branch. The color correction branch first uses the Transformer block to encode the multi-scale feature information to obtain color information. ,like Figure 8 The figure shows the internal structure of the Transformer block, including the self-attention mechanism and the forward feedback processing, which are mature existing technologies and will not be described here. The input of the Transformer block is the output feature of the spatial feature transformation module in the previous stage. After removing the degenerate branch bicubic linear interpolation operation, convolution is performed to extract features to obtain degenerate features .
[0082] Step S22: input the color information and the degradation features into a de-degradation color correction module for iterative processing to obtain a super-resolution face image.
[0083] Specifically, the de-degraded color correction module includes multiple correction sub-modules connected in series, the number of correction sub-modules is the number of iterations, color information and degradation features are used as two inputs of the first correction sub-module, the correction sub-module is used to encode the dependency between the two inputs, and multiple correction sub-modules are used to perform iterative processing in sequence to obtain a super-resolution face image.
[0084] The low-light facial image super-resolution reconstruction method provided in this embodiment uses an iterative approach to achieve complementary learning enhancement of color correction and de-degradation super-resolution, effectively processes facial images under low-light conditions, and achieves high-quality super-resolution reconstruction.
[0085] In some optional embodiments, such as Fig. 9 As shown, the de-degradation color correction module includes multiple correction sub-modules connected in series, the number of correction sub-modules is the number of iterations, and the correction sub-modules include a color estimation module, a super-resolution module, a color adjustment module and a color refinement module.
[0086] The above step S22 inputs the color information and the degradation features into the de-degradation color correction module for iterative processing to obtain a super-resolution face image, including: Step S221, inputting color information into the color estimation module of the current correction submodule to obtain color coefficients, and inputting degradation features into the super-resolution module of the current correction submodule to obtain de-degradation features.
[0087] Specifically, the color information Input the color estimation module of the current correction submodule, and get the color coefficient through three convolution layers , the degenerate features Input the super-resolution module of the current correction submodule to obtain the de-degraded features , Fig.10 , which is the internal structure of the super-resolution module. The internal structures of the color estimation module and the super-resolution module are prior arts and will not be described in detail here.
[0088] Step S222: input the color coefficient and the de-degradation feature into the color adjustment module of the current correction submodule, and perform color adjustment on the de-degradation feature using the color coefficient to obtain a color adjustment result.
[0089] Specifically, the color coefficient and de-degradation features The color adjustment module of the current correction submodule is input together, and the color coefficient is used to adjust the color of the de-degraded features to obtain the color adjustment result. ,in Indicates Correction submodule. Fig.11 The internal structure of the color adjustment module is shown, and the color coefficient and de-degradation features After different convolutional layers and activation functions, we get and , and then use the color adjustment function to get the color adjustment result: .
[0090] Step S223, inputting the color adjustment result and the color coefficient into the color refinement module of the current correction submodule to obtain a color refinement result.
[0091] Specifically, since accurate color coefficients can significantly improve image super-resolution, similarly, high-resolution images are also beneficial for color correction. Therefore, we use an iterative approach to achieve complementary learning enhancement of color correction and de-degradation super-resolution. and color coefficients The color refinement module of the current correction submodule is input, and the color coefficient is refined using the color adjustment result to improve the color correction accuracy, such as Fig.12 The internal structure of the color refinement module and the color adjustment result are shown. and color coefficients The Q, K, and V values are obtained through the fully connected layer respectively, and the cross attention is calculated based on the Q, K, and V values: ,in, represents the cross attention score, represents the softmax function, and d represents the vector length, which is used to ensure that some eigenvalues are not removed by the softmax function. Then the cross attention score is passed through the feedforward network to obtain the color refinement result, which is used as the color coefficient of the next correction submodule.
[0092] Step S224, the color refinement result is input as the updated color information into the color estimation module of the next correction submodule connected in series with the current correction submodule, and the color adjustment result is input as the updated degradation feature into the super-resolution module of the next correction submodule, and so on for iterative processing, and the color adjustment result output by the last correction submodule is used as the super-resolution face image.
[0093] Specifically, after deep feature extraction, the deep features are obtained, and the image is reconstructed using a sub-pixel convolution module (HR reconstruction module). Specifically, the HR reconstruction module consists of three convolutional layers and two Pixel-Shuffle layers alternating to obtain the final output super-resolution face image: ,in, represents the final super-resolution face image, The specific structure of the HR reconstruction module represents the deep features and is a mature existing technology, which will not be described here.
[0094] The low-light facial image super-resolution reconstruction method provided in this embodiment uses a color estimation module to process color information to obtain a color coefficient, uses a super-resolution module to process degradation features to obtain de-degraded features, uses the color coefficient to adjust the de-degraded features, thereby improving the super-resolution of the image in the color adjustment result, and feeds the color adjustment result back to the color refinement module to perform more precise complementary refinement on the color coefficient, thereby improving the model color correction and de-degradation performance, and the obtained super-resolution facial image is clearer.
[0095] In a specific embodiment, as shown in Table 1, the evaluation index results of each model on the Helen dataset, the PFHQ dataset and the SCUT_FBP dataset are listed respectively. The image reconstruction model provided in this embodiment has achieved the best or suboptimal results in terms of peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and learned perceptual image patch similarity (LPIPS), which fully proves that the image reconstruction model provided in this embodiment is competitive. It is worth noting that although the image reconstruction model LLFSR provided in this embodiment has a slightly insufficient performance on LPIPS, the LPIPS index on the Helen dataset has also achieved suboptimal results, which is only 0.0086 different from the best. However, LLFSR shows excellent performance in the PSNR and SSIM evaluation indicators. LLFSR always obtains the highest average PSNR and SSIM scores on the Helen, PFHQ and SCUT_FPB datasets, demonstrating the remarkable ability of LLFSR in preserving facial structure and reconstructing facial details. Importantly, LLFSR outperforms the next best model by a significant margin on all three test sets, with an average PSNR lead of 1.039 and an average SSIM lead of 0.0102.
[0096] Table 1
[0097] For the quantitative analysis of the experimental results, LLFSR is better than other models in restoring low-light and low-resolution face images, solving the problem of damaged facial structure under low-light conditions and achieving good restoration of texture details.
[0098] In this embodiment, a low-light facial image super-resolution reconstruction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0099] This embodiment provides a low-light facial image super-resolution reconstruction device. Fig.13 As shown, including: An image acquisition module 101 is used to acquire a low-light and low-resolution face image to be reconstructed; The image reconstruction module 102 is used to input the low-light low-resolution face image into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; wherein the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein: The shallow feature extraction module is used to perform convolution processing on low-light and low-resolution face images to obtain shallow features; The spatial feature transformation module is used to perform feature transformation on the shallow features to obtain multi-scale feature information; The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution face image.
[0100] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0101] The low-light facial image super-resolution reconstruction device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.
[0102] The embodiment of the present invention also provides a computer device having the above Fig.13 The low-light face image super-resolution reconstruction device shown.
[0103] See also Fig.14 , Fig.14 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.14 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.14 A processor 10 is taken as an example.
[0104] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0105] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0106] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0107] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0108] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0109] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0110] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A low-light facial image super-resolution reconstruction method, characterized in that: The method comprises: Obtaining a low-light and low-resolution face image to be reconstructed; The low-light low-resolution face image is input into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; wherein the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein: The shallow feature extraction module is used to perform convolution processing on low-light and low-resolution face images to obtain shallow features; The spatial feature transformation module is used to perform feature transformation on the shallow features to obtain multi-scale feature information; The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution face image.
2. The method according to claim 1, characterized in that The training process of the image reconstruction model includes: Randomly obtain multiple original high-resolution face images from a preset public dataset; Adding a preset darkness factor and noise to the original high-resolution facial image to obtain a low-light low-resolution facial image corresponding to the original high-resolution facial image; An image pair consisting of a low-light low-resolution face image and a corresponding original high-resolution face image is used as a sample, a preset neural network model is trained, and the trained model is used as an image reconstruction model.
3. The method according to claim 1, characterized in that The step of performing feature changes on the shallow features to obtain multi-scale feature information includes: Inputting the shallow features into a plurality of serially connected encoders for multiple encodings in sequence to obtain a plurality of encoded feature maps; The plurality of encoded feature maps are upsampled by using a plurality of decoders connected in series to extract multi-scale feature information of the low-light and low-resolution facial image, and the number of the decoders is the same as the number of the encoders.
4. The method according to claim 3, characterized in that The shallow features are input into the encoder for encoding to obtain an encoded feature map, including: The shallow features are input into multiple branches with different sampling scales, and multiple semantic information of the shallow features are extracted by downsampling respectively, and the output features of each branch are obtained through the spatial and channel reconstruction convolution module and the multi-dimensional collaborative attention mechanism module; Sort the output features of branches with different sampling scales from low to high according to the sampling scale; Upsample the output features of the branch with the lowest sampling scale by a preset multiple and use them as the fusion features of the current branch; The fusion features of the current branch are sequentially upsampled by a preset multiple, and fused with the output features of the next high sampling scale branch adjacent to the current branch to obtain a fusion result, wherein the sampling scale of the next high sampling scale branch is a preset multiple of the sampling scale of the current branch; The fusion result is convolved and residual processing is performed with the shallow features, and convolution processing is performed again to obtain an encoded feature map.
5. The method according to claim 4, characterized in that The process of upsampling the fusion features of the current branch by a preset multiple and fusing them with the output features of the next high sampling scale branch adjacent to the current branch includes: The fusion feature of the current branch is used as the first feature, the output feature of the next branch is used as the second feature, the first feature and the second feature are respectively input into the preset channel attention module to obtain the first channel output and the second channel output; The first channel output is multiplied by the second channel output to obtain a cross matrix; Performing a residual connection between the cross matrix and the first feature to obtain a first spatial input feature, and performing a residual connection between the cross matrix and the second feature to obtain a second spatial input feature; The first spatial input feature and the second spatial input feature are input into a preset spatial attention module to obtain a feature after spatial attention adjustment, and the feature after spatial attention adjustment is input into a fully connected layer to obtain a fusion feature.
6. The method according to claim 1 or 5, characterized in that: The iterative color correction and de-degradation processing of the multi-scale feature information to obtain a super-resolution face image includes: Performing a self-attention mechanism and a forward feedback process on the multi-scale feature information to obtain color information in the multi-scale feature information, and performing an interpolation and a convolution process on the multi-scale feature information to obtain a degraded feature; The color information and the degradation features are respectively input into a de-degradation color correction module for iterative processing to obtain a super-resolution face image.
7. The method according to claim 6, characterized in that The de-degradation color correction module includes a plurality of correction submodules connected in series, the number of the correction submodules is the number of iterations, and the correction submodules include a color estimation module, a super-resolution module, a color adjustment module and a color refinement module; The color information and the degradation feature are respectively input into a de-degradation color correction module for iterative processing to obtain a super-resolution face image, including: Inputting the color information into the color estimation module of the current correction submodule to obtain a color coefficient, and inputting the degradation feature into the super-resolution module of the current correction submodule to obtain a de-degraded feature; Inputting the color coefficient and the de-degradation feature into a color adjustment module of the current correction submodule, and performing color adjustment on the de-degradation feature using the color coefficient to obtain a color adjustment result; Inputting the color adjustment result and the color coefficient into the color refinement module of the current correction submodule to obtain a color refinement result; The color refinement result is input as updated color information into the color estimation module of the next correction submodule connected in series with the current correction submodule, and the color adjustment result is input as updated degradation feature into the super-resolution module of the next correction submodule. The iterative processing is performed in this way, and the color adjustment result output by the last correction submodule is used as the super-resolution face image.
8. A low-light facial image super-resolution reconstruction device, characterized in that: The device comprises: An image acquisition module, used for acquiring a low-light and low-resolution face image to be reconstructed; An image reconstruction module is used to input the low-light low-resolution face image into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; wherein the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein: The shallow feature extraction module is used to perform convolution processing on low-light and low-resolution face images to obtain shallow features; The spatial feature transformation module is used to perform feature transformation on the shallow features to obtain multi-scale feature information; The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution face image.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Low-illumination image enhancement method based on feature fusion and attention embedding
CN116797488A
Double-branch fusion low-illumination image enhancement method
CN116977208A
Face image super-resolution reconstruction method under low illumination condition
CN117830096A
Infrared image super-resolution reconstruction method based on complementary reference
CN118096534A
Low-illumination image enhancement method fusing cross mixing attention and receptive field expansion
CN118212163A
Cited By
Image super-resolution reconstruction method
CN122115217A
