A method, device and equipment for super-resolution reconstruction of low-light face images

Abstract

The present invention relates to the field of image enhancement technology, and discloses a method, device and equipment for super-resolution reconstruction of low-light human face images. The method comprises: inputting a low-light low-resolution human face image to be reconstructed into a pre-trained image reconstruction model to obtain a reconstructed super-resolution human face image; the image reconstruction model comprises: a shallow feature extraction module for performing convolution processing on the low-light low-resolution human face image to obtain shallow features; a spatial feature transformation module for performing feature changes on the shallow features to obtain multi-scale feature information; and a de-degradation color correction module for iteratively performing color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution human face image. The present invention utilizes the spatial feature transformation module for brightness restoration and utilizes the de-degradation color correction module for color correction, thereby reducing the degree of distortion of image details or structures in the process of reconstructing human face images, and can both maintain the stability of facial structures and restore texture details.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, specifically to a method, apparatus, and device for super-resolution reconstruction of low-light facial images. Background Technology

[0002] Face image super-resolution techniques typically rely on prior knowledge of the face to recover clear and realistic facial details. However, due to factors such as low light and shadow interference in the external environment, face images acquired under low-light conditions cannot provide accurate geometric priors and texture information. Furthermore, severe degradation during the transmission and propagation of face images can also damage the structure and details of the reconstructed high-resolution face image, thus reducing the performance of super-resolution. In more complex real-world scenarios, images are subject to various forms of degradation, such as noise and artifacts, which can make the dimensionality reduction process even more complex. Especially in low-light scenes, face images often have low brightness, and the reconstruction process is prone to distortion of image details or structure, thereby reducing the performance of subsequent processing tasks. Summary of the Invention

[0003] In view of this, the present invention provides a method, apparatus and device for super-resolution reconstruction of face images under low light conditions, so as to solve the problem of image detail or structural distortion during the reconstruction of face images under low light conditions.

[0004] In a first aspect, the present invention provides a method for super-resolution reconstruction of low-light face images, the method comprising:

[0005] Acquire low-light, low-resolution face images to be reconstructed;

[0006] Low-light, low-resolution face images are input into a pre-trained image reconstruction model to obtain reconstructed super-resolution face images. The image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module.

[0007] The shallow feature extraction module is used to perform convolution processing on low-light, low-resolution face images to obtain shallow features;

[0008] The spatial feature transformation module is used to transform shallow features to obtain multi-scale feature information;

[0009] The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on multi-scale feature information to obtain super-resolution face images.

[0010] The low-light face image super-resolution reconstruction method provided by this invention uses a spatial feature transformation module for brightness restoration and a de-degradation color correction module for color correction, which reduces the degree of distortion of image details or structure during the reconstruction of face images, thus maintaining the stability of facial structure and restoring texture details.

[0011] In one alternative implementation, the training process of the image reconstruction model includes:

[0012] Multiple raw high-resolution face images are randomly obtained from a pre-set public dataset;

[0013] Preset dark factors and noise are added to each original high-resolution face image to obtain a low-light, low-resolution face image corresponding to the original high-resolution face image;

[0014] Image pairs consisting of low-light, low-resolution face images and their corresponding original high-resolution face images are used as samples to train a pre-defined neural network model, and the trained model is used as the image reconstruction model.

[0015] The low-light face image super-resolution reconstruction method provided by this invention uses the dark factor to control the illumination intensity of high-resolution face images in public datasets, and reduces the resolution by adding noise to obtain low-light low-resolution-high-resolution image pairs. This solves the problem of the lack of public data on low-light low-resolution face images, and the trained image reconstruction model performs better in super-resolution reconstruction.

[0016] In one optional implementation, feature transformation is performed on shallow features to obtain multi-scale feature information, including:

[0017] Shallow features are input into multiple cascaded encoders and encoded multiple times in sequence to obtain multiple encoded feature maps;

[0018] Multiple decoders are used in series to upsample multiple encoded feature maps to extract multi-scale feature information of low-light, low-resolution face images. The number of decoders is the same as that of encoders.

[0019] The low-light face image super-resolution reconstruction method provided by this invention utilizes the same number of cascaded encoders and cascaded decoders to gradually restore illumination based on shallow features, extract multi-scale feature information, improve the image reconstruction model's ability to represent features at different scales, and enhance the model's ability to restore image illumination.

[0020] In one optional implementation, the shallow features are input into an encoder for encoding to obtain an encoded feature map, including:

[0021] Shallow features are input into multiple branches with different sampling scales, and downsampling is performed to extract multiple semantic information from the shallow features. After passing through the spatial and channel reconstruction convolution module and the multidimensional collaborative attention mechanism module, the output features of each branch are obtained.

[0022] The output features of branches with different sampling scales are sorted from low to high according to the sampling scale;

[0023] The output features of the lowest sampling scale branch are upsampled by a preset factor and used as the fused features of the current branch.

[0024] The fusion features of the current branch are upsampled by a preset multiple and then fused with the output features of the next high-sampling-scale branch adjacent to the current branch to obtain the fusion result. The sampling scale of the next high-sampling-scale branch is a preset multiple of the sampling scale of the current branch.

[0025] The fusion result is convolved and then subjected to residual processing with shallow features, followed by another convolution process to obtain the encoded feature map.

[0026] The low-light face image super-resolution reconstruction method provided by this invention uses spatial and channel reconstruction convolution modules to suppress redundant spatial information and amplify the weights of key feature information. It also uses a multi-dimensional collaborative attention mechanism module to capture the interdependencies in the spatial dimension and the interactions between channels, which significantly improves the model's ability to capture important facial structure pixels.

[0027] In one alternative implementation, the encoder includes three branches with different sampling scales: the first branch has a sampling scale of 8 times, the second branch has a sampling scale of 4 times, and the third branch has a sampling scale of 2 times.

[0028] The fusion features of the current branch are upsampled by a preset factor and then fused with the output features of the next high-sampling-scale branch adjacent to the current branch to obtain the fusion result, including:

[0029] The output features of the first branch are upsampled by a factor of 2 to obtain the first fused features;

[0030] The second fusion feature is obtained by fusing the output features of the first fusion feature and the second branch;

[0031] The second fusion feature is upsampled by a factor of 2 and fused with the output feature of the third branch to obtain the third fusion feature as the fusion result.

[0032] The low-light face image super-resolution reconstruction method provided by this invention has a sampling scale difference of two times between each branch, and the sampling multiple of the fusion features corresponding to each branch is also doubled when upsampling. This makes the features of adjacent branches have the same dimension when fused, which facilitates data processing and improves the efficiency of feature fusion.

[0033] In one optional implementation, the process of upsampling the fused features of the current branch by a preset factor and fusing them with the output features of the next high-sampling-scale branch adjacent to the current branch includes:

[0034] The fusion feature of the current branch is used as the first feature, and the output feature of the next branch is used as the second feature. The first feature and the second feature are respectively input into the preset channel attention module to obtain the first channel output and the second channel output.

[0035] Multiplying the output of the first channel by the output of the second channel yields the cross matrix;

[0036] The first spatial input feature is obtained by residual concatenating the cross matrix with the first feature, and the second spatial input feature is obtained by residual concatenating the cross matrix with the second feature.

[0037] The first spatial input features and the second spatial input features are input into a preset spatial attention module to obtain spatial attention-adjusted features. The spatial attention-adjusted features are then input into a fully connected layer to obtain fused features.

[0038] The low-light face image super-resolution reconstruction method provided by this invention utilizes a channel attention module and a spatial attention module to fuse features of adjacent branches, while taking into account the spatial layout of the face image and the importance of channel features, thereby improving the performance of the model in image processing.

[0039] In one optional implementation, multi-scale feature information is iteratively processed for color correction and de-degradation to obtain a super-resolution face image, including:

[0040] The color information in the multi-scale feature information is obtained by applying a self-attention mechanism and a feedback mechanism to the multi-scale feature information. The degenerate feature is obtained by interpolating and convolution of the multi-scale feature information.

[0041] Color information and degradation features are input into the de-degradation color correction module for iterative processing to obtain a super-resolution face image.

[0042] The low-light face image super-resolution reconstruction method provided by this invention utilizes an iterative approach to achieve complementary learning enhancement of color correction and de-degradation super-resolution, effectively processing face images under low-light conditions and achieving high-quality super-resolution reconstruction.

[0043] In one optional implementation, the de-degradation color correction module includes multiple correction sub-modules connected in series, the number of correction sub-modules being the number of iterations, and the correction sub-modules including a color estimation module, a super-resolution module, a color adjustment module, and a color refinement module.

[0044] Color information and degradation features are input into the de-degradation color correction module for iterative processing to obtain a super-resolution face image, including:

[0045] The color information is input into the color estimation module of the current correction submodule to obtain the color coefficients, and the degradation features are input into the super-resolution module of the current correction submodule to obtain the de-degradation features.

[0046] The color coefficients and de-degradation features are input into the color adjustment module of the current correction submodule. The color coefficients are used to adjust the color of the de-degradation features to obtain the color adjustment result.

[0047] Input the color adjustment results and color coefficients into the color refinement module of the current correction submodule to obtain the color refinement results;

[0048] The color refinement result is used as the updated color information and input into the color estimation module of the next correction submodule, which is connected in series with the current correction submodule. The color adjustment result is used as the updated degradation feature and input into the super-resolution module of the next correction submodule. This process is repeated iteratively until the color adjustment result output by the last correction submodule is used as the super-resolution face image.

[0049] The low-light face image super-resolution reconstruction method provided by this invention uses a color estimation module to process color information to obtain color coefficients, uses a super-resolution module to process degradation features to obtain de-degradation features, and uses color coefficients to adjust the de-degradation features, thereby improving the super-resolution of the image in the color adjustment result. The color adjustment result is fed back to the color thinning module to perform more accurate complementary thinning of the color coefficients, thereby improving the model's color correction and de-degradation performance, and the resulting super-resolution face image is clearer.

[0050] Secondly, the present invention provides a low-light face image super-resolution reconstruction device, the device comprising:

[0051] The image acquisition module is used to acquire low-light, low-resolution face images to be reconstructed;

[0052] The image reconstruction module is used to input the low-light, low-resolution face image into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; wherein the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein:

[0053] The shallow feature extraction module is used to perform convolution processing on low-light, low-resolution face images to obtain shallow features;

[0054] The spatial feature transformation module is used to perform feature transformation on the shallow features to obtain multi-scale feature information;

[0055] The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution face image.

[0056] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0057] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0058] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating a low-light face image super-resolution reconstruction method according to an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of the image reconstruction model in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention;

[0061] Figure 3 This is a flowchart illustrating the processing of shallow features by the spatial feature transformation module in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention.

[0062] Figure 4 This is a schematic diagram of the de-degradation color correction module in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention.

[0063] Figure 5 This is a schematic diagram of the internal structure of the encoder in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of a specific example of an encoder in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention.

[0065] Figure 7 This is a schematic diagram of the structure of the preset cross-attention module inside the encoder in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention;

[0066] Figure 8This is a schematic diagram of the Transformer block of the de-degradation color correction module in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention.

[0067] Figure 9 This is a schematic diagram of another de-degradation color correction module in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention.

[0068] Figure 10 This is a schematic diagram of the structure of the super-resolution module in the de-degradation color correction module of the low-light face image super-resolution reconstruction method according to an embodiment of the present invention;

[0069] Figure 11 This is a schematic diagram of the color adjustment module in the de-degradation color correction module of the low-light face image super-resolution reconstruction method according to an embodiment of the present invention.

[0070] Figure 12 This is a schematic diagram of the color refinement module in the de-degradation color correction module of the low-light face image super-resolution reconstruction method according to an embodiment of the present invention.

[0071] Figure 13 This is a structural block diagram of a low-light face image super-resolution reconstruction device according to an embodiment of the present invention;

[0072] Figure 14 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] This invention provides a method for super-resolution reconstruction of low-light face images, which is applicable to the initial face image reconstruction for downstream tasks such as face recognition and expression recognition, thereby enhancing the practical application capabilities of downstream tasks.

[0075] According to an embodiment of the present invention, a method for super-resolution reconstruction of low-light face images is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0076] This embodiment provides a method for super-resolution reconstruction of low-light face images, which can be used in the aforementioned computer system. Figure 1 This is a schematic flowchart of a low-light face image super-resolution reconstruction method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0077] Step S1: Obtain the low-light, low-resolution face image to be reconstructed.

[0078] Specifically, the low-light, low-resolution face image to be reconstructed is a face image captured by an image acquisition device under low-light conditions before face recognition or expression recognition is performed.

[0079] Step S2: Input the low-light, low-resolution face image to be reconstructed into the pre-trained image reconstruction model to obtain the reconstructed super-resolution face image.

[0080] Figure 2 This is a schematic diagram of the image reconstruction model in the low-light face image super-resolution reconstruction method according to an embodiment of the present invention, as shown below. Figure 2 As shown, the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module.

[0081] The shallow feature extraction module is used to perform convolution processing on low-light, low-resolution face images to obtain shallow features.

[0082] Specifically, shallow features refer to low-level features extracted from a neural network. These features typically reside in the early stages of the network and are closely related to the input data. Shallow features include fine-grained information such as color, texture, edges, and corners. The shallow feature extraction module uses a standard 3×3 convolutional layer for feature extraction. The input to the shallow feature extraction module is a low-light, low-resolution face image. Where C, H, and W represent the channels, height, and width of the input face image, respectively. The shallow feature extraction module first standardizes and normalizes the input face image, and then performs ordinary convolution operations to reduce the complexity of the model and the computational resource consumption of the deep feature extraction stage. The network structure formula of the shallow feature extraction module is as follows: ,in, This represents the shallow features obtained from a low-light, low-resolution face image after processing by the shallow feature extraction module.

[0083] The spatial feature transformation module is used to transform shallow features to obtain multi-scale feature information.

[0084] Specifically, deep features, processed through multiple network layers, possess a large receptive field and rich semantic information. In the deep feature extraction stage, the Spatial Feature Transformation (SFTM) module, the Degradation Color Correction (DCCM) module, and the Self-Attention Mechanism (SA) module enhance the shallow feature information extracted by the shallow feature extraction module. Specifically, SFTM alters the spatial structure and channel features of the input shallow features, enhancing or suppressing certain spatial features; SA automatically weights the input shallow features, strengthening important information parts and ignoring redundant content; DCCM combines the outputs of SFTM and SA to correct color distortion in the image, improving color realism and consistency. The network structure formula for the deep feature extraction stage is: ,in, This represents the deep features after deep feature extraction.

[0085] like Figure 3 As shown, the spatial feature transformation module processes shallow features as follows: shallow features are input into multiple cascaded encoders, and then upsampled using the same number of decoders to gradually recover the image's illumination. Activation functions and residual connections are used to prevent the loss of illuminated pixels. The entire spatial feature transformation module is implemented using an encoder-decoder architecture, which not only improves the model's ability to represent features at different scales but also significantly enhances the model's ability to recover image illumination.

[0086] The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on multi-scale feature information to obtain super-resolution face images.

[0087] Specifically, such as Figure 4 As shown, the de-degradation color correction module has a dual-branch structure, including a color correction branch and a de-degradation branch. The color correction branch uses Transformer blocks to encode color information, while the de-degradation branch convolves the multi-scale feature information after bicubic interpolation to obtain degradation features. The de-degradation color correction module includes multiple correction sub-modules. Color information and degradation features serve as the two inputs to the first correction sub-module, and multiple iterations are performed in multiple concatenated correction sub-modules to obtain the super-resolution face image.

[0088] The low-light face image super-resolution reconstruction method provided in this embodiment uses a spatial feature transformation module for brightness restoration and a de-degradation color correction module for color correction, which reduces the degree of distortion of image details or structure during the reconstruction of face images, thus maintaining the stability of facial structure and restoring texture details.

[0089] In some alternative implementations, the training process of the image reconstruction model includes:

[0090] Step a1: Randomly obtain multiple original high-resolution face images from a preset public dataset.

[0091] Specifically, due to the lack of publicly available datasets for low-light face super-resolution tasks, simulation experiments are conducted on existing publicly available face datasets to simulate low-light, low-resolution face images in real-world scenarios. The pre-selected publicly available datasets include, but are not limited to, the CelebA-HQ and FFHQ datasets, which contain a large number of high-resolution face images. 20,000 high-resolution face images are randomly selected from these datasets for the simulation experiments.

[0092] Step a2: Add a preset dark factor and noise to each original high-resolution face image to obtain a low-light, low-resolution face image corresponding to the original high-resolution face image.

[0093] Specifically, after selecting the image, a low-light operation is performed on the original high-resolution face image. The process involves controlling the light intensity using a dark factor to simulate low-light conditions. The value range of the dark factor is set to [0,1]. When simulating low light, the value of the dark factor is fixed within the range of [0.5,0.9]. The larger the value of the dark factor, the lower the light intensity. In addition, noise is added to the face image using degradation models. These models mainly include five types of degradation models: motion blur, Gaussian noise, Rayleigh noise, salt-and-pepper noise, and mean blur. The specific process of adding noise is a mature existing technology and will not be elaborated here. Finally, a low-light, low-resolution face image corresponding to the original high-resolution face image is obtained.

[0094] Step a3: Use the image pair consisting of the low-light, low-resolution face image and the corresponding original high-resolution face image as samples to train the preset neural network model, and use the trained model as the image reconstruction model.

[0095] Specifically, a dataset consisting of image pairs—low-light, low-resolution face images and their corresponding original high-resolution face images—is used as a sample dataset to train a preset neural network model. The preset neural network model is a Spatial Transformer Network (STNs), which is only an example and not a limitation. The resulting image reconstruction model is a mature existing technology and will not be described in detail here.

[0096] The low-light face image super-resolution reconstruction method provided in this embodiment uses the dark factor to control the illumination intensity of high-resolution face images in the public dataset, and reduces the resolution by adding noise to obtain low-light low-resolution-high-resolution image pairs. This solves the problem of lack of public data on low-light low-resolution face images, and the trained image reconstruction model performs better in super-resolution reconstruction.

[0097] In some alternative implementations, such as Figure 3 As shown, multi-scale feature information is obtained by performing feature transformations on shallow features, including:

[0098] Step S11: Input the shallow features into multiple cascaded encoders and encode them multiple times in sequence to obtain multiple encoded feature maps.

[0099] Specifically, shallow features are sequentially input into a series of encoders. The input to the first encoder is the shallow feature. Starting with the next encoder connected to the first encoder, the input to the next encoder is the output of the previous encoder. Based on this, each encoder outputs a feature map, resulting in multiple encoded feature maps, as expressed by the formula: ,in This represents the i-th encoder. Indicates the first The feature map output by the encoder.

[0100] Step S12: Upsample multiple encoded feature maps using multiple cascaded decoders to extract multi-scale feature information of low-light, low-resolution face images. The number of decoders is the same as that of encoders.

[0101] Specifically, each encoder corresponds to one decoder. The feature maps output by each corresponding encoder are upsampled using the same number of decoders to gradually restore the image's illumination. By utilizing the Sigmoid function and residual connections, the loss of illuminated pixels is avoided, thus extracting multi-scale feature information from low-illumination, low-resolution face images. The formula is expressed as: Where Sigmoid represents the activation function. Indicates the first Feature maps output by each decoder Indicates the first The decoder for the first... The feature map output by the encoder and the first encoder The feature maps output by each encoder are used for decoding.

[0102] The low-light face image super-resolution reconstruction method provided in this embodiment uses the same number of cascaded encoders and cascaded decoders to gradually recover illumination based on shallow features, extract multi-scale feature information, improve the image reconstruction model's ability to represent features at different scales, and enhance the model's ability to recover image illumination.

[0103] In some alternative implementations, such as Figure 5 The diagram shows the internal structure of the encoder. Shallow features are input into the encoder for encoding, resulting in an encoded feature map, including:

[0104] Step S111: Input the shallow features into multiple branches with different sampling scales, perform downsampling to extract multiple semantic information of the shallow features, and obtain the output features of each branch through the spatial and channel reconstruction convolution module and the multidimensional collaborative attention mechanism module.

[0105] Specifically, each encoder has an identical internal structure with multiple branches at different sampling scales. For the input features of each encoder, downsampling is performed at different sampling scales in each branch to extract the semantic information of the input features. Then, the input features are passed through a Spatial and Channel Reconstruction Convolution for Feature Redundancy (SCConv) module to suppress redundant spatial information and amplify the weights of key feature information. Finally, a Multidimensional Collaborative Attention (MCA) module captures the interdependencies in the spatial dimension and the interactions between channels to obtain the output features of each branch. The output features of each branch are represented as follows: ,in, This indicates that the input features will be processed. The semantic information obtained by downsampling by a factor of 10 The sampling scale is indicated as The output characteristics of the branch.

[0106] Step S112: Sort the output features of different sampling scale branches in order of sampling scale from low to high.

[0107] Specifically, the output features of branches with different sampling scales are sorted from low to high according to the sampling scale. For example, the sampling scales of each branch are 2x, 4x, 8x, 16x, 32x, and 64x, which are just examples and not a limitation. Since it is downsampling, the sampling scales of each branch are sorted from low to high as follows: 64x, 32x, 16x, 8x, 4x, and 2x. Multiple branches extract features at different scales, and the sampling scales between adjacent branches differ by a factor of two, which facilitates subsequent feature fusion. Preferably, through experimental verification, three branches with sampling scales of 8x, 4x, and 2x can be selected.

[0108] Step S113: Upsample the output features of the lowest sampling scale branch by a preset multiple and use them as the fusion features of the current branch.

[0109] Specifically, the output features of the lowest sampling scale branch are upsampled by a preset multiple and used as the fusion features of the current branch. For the lowest sampling scale branch, its output features are not fused, and its upsampled result is equivalent to the fusion features of the lowest sampling scale branch.

[0110] Step S114: The fusion features of the current branch are upsampled by a preset multiple and fused with the output features of the next high-sampling-scale branch adjacent to the current branch to obtain the fusion result. The sampling scale of the next high-sampling-scale branch is a preset multiple of the sampling scale of the current branch.

[0111] Specifically, such as Figure 5 As shown, for the output features of multiple branches at different scales, the fusion features of the current branch need to be upsampled by a preset multiple and then fused with the output features of the next high-sampling scale branch to obtain the fusion features of the next high-sampling scale branch. The fusion features of the last high-sampling scale branch are used as the fusion result of the output features of each branch. The sampling scale of the next high-sampling scale branch is a preset multiple of the sampling scale of the current branch.

[0112] Step S115: After convolving the fusion result, perform residual processing with the shallow features, and then perform convolution processing again to obtain the encoded feature map.

[0113] Specifically, the fusion result is passed through a convolutional layer and then residually connected to the input features of the encoder. The fused result is then passed through another convolutional layer to obtain the final output, which serves as the encoded feature map.

[0114] Therefore, the first The output feature map of each encoder is represented as follows: ,in, Indicates the first The input feature map of each encoder, Indicates the first The fusion feature of the highest sampling scale branch in each encoder.

[0115] The low-light face image super-resolution reconstruction method provided in this embodiment uses spatial and channel reconstruction convolution modules to suppress redundant spatial information and amplify the weights of key feature information. It also uses a multi-dimensional collaborative attention mechanism module to capture the interdependencies in the spatial dimension and the interactions between channels, which significantly improves the model's ability to capture important facial structure pixels.

[0116] In some alternative implementations, such as Figure 6 As shown, the encoder includes three branches with different sampling scales: the first branch has a sampling scale of 8 times, the second branch has a sampling scale of 4 times, and the third branch has a sampling scale of 2 times.

[0117] Step S114 sequentially upsamples the fusion features of the current branch by a preset factor and fuses them with the output features of the next high-sampling-scale branch adjacent to the current branch to obtain the fusion result, including:

[0118] Step b1: Upsample the output features of the first branch by a factor of 2 to obtain the first fused features.

[0119] Specifically, the output characteristics of the first branch are: The first fusion feature is obtained by upsampling it by a factor of 2: .

[0120] Step b2: Fuse the first fusion feature and the output features of the second branch to obtain the second fusion feature.

[0121] Specifically, the output features of the first fusion feature and the second branch are fused using a preset cross-attention module to obtain the second fusion feature: ,in Indicates the output characteristics of the second branch: , Indicates the second fusion feature, This represents a new cross-attention module designed based on the Channel Attention Mechanism (CAM) and Spatial Attention Module (SAM) in the Convolutional Block Attention Module (CBAM).

[0122] Step b3: Upsample the second fusion feature by a factor of 2 and fuse it with the output feature of the third branch to obtain the third fusion feature as the fusion result.

[0123] Specifically, the third fusion feature is obtained by fusing the second fusion feature and the output feature of the third branch using a preset cross-attention module: ,in Indicates the output characteristics of the second branch: , Indicates the third fusion feature, This represents a new cross-attention module designed based on CAM and SAM in CBAM. For an encoder with only three branches, the third fusion feature of the third branch is the fusion result.

[0124] The low-light face image super-resolution reconstruction method provided in this embodiment has a sampling scale difference of two times between each branch, and the sampling multiple of the fusion features corresponding to each branch is also doubled when upsampling. This makes the features of adjacent branches have the same dimension when fused, which facilitates data processing and improves the efficiency of feature fusion.

[0125] In some optional implementations, step S114, which involves upsampling the fused features of the current branch by a preset factor and fusing them with the output features of the next high-sampling-scale branch adjacent to the current branch, includes:

[0126] Step S1141: Take the fusion feature of the current branch as the first feature, take the output feature of the next branch as the second feature, and input the first feature and the second feature into the preset channel attention module respectively to obtain the first channel output and the second channel output.

[0127] Specifically, such as Figure 7 The diagram shows the internal structure of the preset cross-attention module (CAFM). The CAFM module in the encoder comprises two main modules: the preset channel attention module (CAM) and the preset spatial attention module (SAM). Its inputs are the fused features of the current branch and the output features of the next branch. Therefore, the fused features of the current branch are used as the first feature. The output feature of the next branch is used as the second feature. The first feature and the second feature are input into a preset channel attention module, respectively, to obtain the first channel output and the second channel output. The shapes of the first channel output and the second channel output are as follows: The channel output is represented as: ,in Indicates the first aisle.

[0128] Step S1142: Multiply the first channel output by the second channel output to obtain the cross matrix.

[0129] Specifically, multiplying the output of the first channel by the output of the second channel yields a shape of... Cross matrix: ,in, , These are the first channel output and the second channel output, respectively.

[0130] Step S1143: Perform a residual connection between the cross matrix and the first feature to obtain the first spatial input feature, and perform a residual connection between the cross matrix and the second feature to obtain the second spatial input feature.

[0131] Specifically, the first spatial input feature is obtained by residually concatenating the cross matrix with the first feature: The second spatial input feature is obtained by residually concatenating the cross matrix with the second feature: .

[0132] Step S1144: Input the first spatial input feature and the second spatial input feature into the preset spatial attention module to obtain the spatial attention-adjusted feature, and input the spatial attention-adjusted feature into the fully connected layer to obtain the fused feature.

[0133] Specifically, the first spatial input features and the second spatial input features are input into a preset spatial attention module to obtain spatially attention-adjusted features: The spatial attention-adjusted features are input into the fully connected layer to obtain the fused features: ,in, This represents the preset weight adjustment matrix. This indicates the bias term.

[0134] The low-light face image super-resolution reconstruction method provided in this embodiment utilizes a channel attention module and a spatial attention module to fuse features of adjacent branches. It also takes into account the spatial layout of the face image and the importance of channel features, thereby improving the performance of the model in image processing.

[0135] In some alternative implementations, such as Figure 3 As shown, multi-scale feature information is iteratively processed for color correction and de-degradation to obtain a super-resolution face image, including:

[0136] Step S21: Perform self-attention mechanism and pre-feedback processing on the multi-scale feature information to obtain the color information in the multi-scale feature information. Perform interpolation and convolution processing on the multi-scale feature information to obtain the degradation features.

[0137] Specifically, the de-degradation color correction module has a dual-branch structure, including a color correction branch and a de-degradation branch. The color correction branch first uses Transformer blocks to encode multi-scale feature information to obtain color information. ,like Figure 8 The diagram shows the internal structure of the Transformer block, which includes a self-attention mechanism and pre-feedback processing. These are mature existing technologies and will not be elaborated upon here. The input to the Transformer block is the output features of the previous stage spatial feature transformation module. After removing the degenerate branches and performing bicubic interpolation, convolution is then performed to extract features, resulting in degenerate features. .

[0138] Step S22: Input the color information and degradation features into the de-degradation color correction module for iterative processing to obtain a super-resolution face image.

[0139] Specifically, the de-degradation color correction module includes multiple cascaded correction sub-modules, the number of which is equal to the number of iterations. Color information and degradation features are used as the two inputs of the first correction sub-module, which encodes the dependency between the two inputs. Multiple correction sub-modules are used for iterative processing in sequence to obtain a super-resolution face image.

[0140] The low-light face image super-resolution reconstruction method provided in this embodiment utilizes an iterative approach to achieve complementary learning enhancement of color correction and de-degradation super-resolution, effectively processing face images under low-light conditions and achieving high-quality super-resolution reconstruction.

[0141] In some alternative implementations, such as Figure 9 As shown, the de-degradation color correction module includes multiple cascaded correction sub-modules. The number of correction sub-modules is equal to the number of iterations. The correction sub-modules include a color estimation module, a super-resolution module, a color adjustment module, and a color refinement module.

[0142] Step S22 above inputs color information and degradation features into the de-degradation color correction module for iterative processing to obtain a super-resolution face image, including:

[0143] Step S221: Input the color information into the color estimation module of the current correction submodule to obtain the color coefficients, and input the degradation features into the super-resolution module of the current correction submodule to obtain the de-degradation features.

[0144] Specifically, color information The color estimation module of the current correction submodule is input, and the color coefficients are obtained after passing through three convolutional layers. Degradation characteristics The de-degradation features are obtained by inputting the super-resolution module of the current correction submodule. , Figure 10 The internal structure of the super-resolution module is shown below. The internal structures of the color estimation module and the super-resolution module are existing technologies and will not be described in detail here.

[0145] Step S222: Input the color coefficients and de-degradation features into the color adjustment module of the current correction submodule, and use the color coefficients to adjust the color of the de-degradation features to obtain the color adjustment result.

[0146] Specifically, color coefficients and de-degeneration characteristics The color adjustment module of the current correction submodule is input together, and the color coefficients are used to adjust the color of the de-degradation features to obtain the color adjustment result. ,in Indicates the first A correction submodule. For example... Figure 11The diagram shows the internal structure of the color adjustment module, including the color coefficients. and de-degeneration characteristics After different convolutional layers and activation functions, the following results were obtained: and Then, the color adjustment function is used to process the colors to obtain the color adjustment result: .

[0147] Step S223: Input the color adjustment result and color coefficient into the color refinement module of the current correction submodule to obtain the color refinement result.

[0148] Specifically, since accurate color coefficients can significantly improve image super-resolution, high-resolution images are also beneficial for color correction. Therefore, we employ an iterative approach to achieve complementary learning enhancement of color correction and de-degradation super-resolution. The color adjustment results... and color coefficient Input the color refinement module of the current correction submodule, and refine the color coefficients using the color adjustment results to improve color correction accuracy, such as... Figure 12 The image shows the internal structure of the color refinement module and the color adjustment results. and color coefficient The Q, K, and V values ​​are obtained through fully connected layers respectively, and cross-attention is calculated based on the Q, K, and V values: ,in, Indicates the cross-attention score. The function represents the softmax function, and d represents the vector length. Its purpose is to ensure that some feature values ​​are not removed by the softmax function. Then, the cross-attention score is passed through a feedforward network to obtain the color refinement result, which serves as the color coefficient for the next correction submodule.

[0149] Step S224: The color refinement result is used as the updated color information and input into the color estimation module of the next correction submodule connected in series with the current correction submodule. The color adjustment result is used as the updated degradation feature and input into the super-resolution module of the next correction submodule. This process is repeated iteratively until the color adjustment result output by the last correction submodule is used as the super-resolution face image.

[0150] Specifically, after deep feature extraction, deep features are obtained, and image reconstruction is performed using a sub-pixel convolutional module (HR reconstruction module). Specifically, the HR reconstruction module consists of three alternating convolutional layers and two pixel-shuffle layers to obtain the final super-resolution face image. ,in, This represents the final super-resolution face image. The specific structure of the HR reconstruction module, which represents deep features, is a mature existing technology and will not be elaborated here.

[0151] The low-light face image super-resolution reconstruction method provided in this embodiment uses a color estimation module to process color information to obtain color coefficients, a super-resolution module to process degradation features to obtain de-degradation features, and color coefficients to adjust the de-degradation features, thereby improving the super-resolution of the image in the color adjustment result. The color adjustment result is fed back to the color thinning module to perform more accurate complementary thinning of the color coefficients, thereby improving the model's color correction and de-degradation performance, and resulting in a clearer super-resolution face image.

[0152] In one specific embodiment, as shown in Table 1, the evaluation metrics results for each model on the Helen, PFHQ, and SCUT_FBP datasets are listed respectively. The image reconstruction model provided in this embodiment achieves best or near-best performance in Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), fully demonstrating the competitiveness of the image reconstruction model provided in this embodiment. It is worth noting that although the image reconstruction model LLFSR provided in this embodiment performs slightly worse on LPIPS, it still achieves near-best performance on the Helen dataset, differing from the best by only 0.0086. However, LLFSR demonstrates excellent performance in PSNR and SSIM evaluation metrics. LLFSR consistently achieves the highest average PSNR and SSIM scores on the Helen, PFHQ, and SCUT_FPB datasets, demonstrating LLFSR's significant ability to preserve facial structure and reconstruct facial details. Most importantly, LLFSR significantly outperformed the suboptimal model on all three test sets, leading by an average of 1.039 in PSNR and 0.0102 in SSIM.

[0153] Table 1

[0154]

[0155] Quantitative analysis of the experimental results shows that LLFSR outperforms other models in restoring low-light, low-resolution face images, solving the problem of facial structure damage under low-light conditions, while achieving good texture detail restoration.

[0156] This embodiment also provides a low-light face image super-resolution reconstruction apparatus, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0157] This embodiment provides a device for super-resolution reconstruction of low-light facial images, such as... Figure 13 As shown, it includes:

[0158] Image acquisition module 101 is used to acquire low-light, low-resolution face images to be reconstructed;

[0159] Image reconstruction module 102 is used to input the low-light, low-resolution face image into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; wherein the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein:

[0160] The shallow feature extraction module is used to perform convolution processing on low-light, low-resolution face images to obtain shallow features;

[0161] The spatial feature transformation module is used to perform feature transformation on the shallow features to obtain multi-scale feature information;

[0162] The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution face image.

[0163] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0164] In this embodiment, the low-light face image super-resolution reconstruction device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0165] This invention also provides a computer device having the above-described features. Figure 13 The device shown is a super-resolution reconstruction device for low-light human face images.

[0166] Please see Figure 14 , Figure 14 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 14As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 14 Take a processor 10 as an example.

[0167] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0168] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0169] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0170] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0171] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0172] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0173] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A low-light facial image super-resolution reconstruction method, characterized in that: The method comprises: Obtaining a low-light and low-resolution face image to be reconstructed; The low-light low-resolution face image is input into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; wherein the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein: The shallow feature extraction module is used to perform convolution processing on low-light and low-resolution face images to obtain shallow features; The spatial feature transformation module is used to perform feature transformation on the shallow features to obtain multi-scale feature information; The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution face image, including: Performing a self-attention mechanism and a forward feedback process on the multi-scale feature information to obtain color information in the multi-scale feature information, and performing an interpolation and a convolution process on the multi-scale feature information to obtain a degraded feature; Inputting the color information and the degradation features into a de-degradation color correction module for iterative processing to obtain a super-resolution face image, wherein the de-degradation color correction module includes a plurality of correction submodules connected in series, the number of the correction submodules being the number of iterations, and the correction submodules including a color estimation module, a super-resolution module, a color adjustment module and a color refinement module; The color information and the degradation features are respectively input into a de-degradation color correction module for iterative processing to obtain a super-resolution face image, including: Inputting the color information into the color estimation module of the current correction submodule to obtain a color coefficient, and inputting the degradation feature into the super-resolution module of the current correction submodule to obtain a de-degraded feature; Inputting the color coefficient and the de-degradation feature into a color adjustment module of the current correction submodule, and performing color adjustment on the de-degradation feature using the color coefficient to obtain a color adjustment result; Inputting the color adjustment result and the color coefficient into the color refinement module of the current correction submodule to obtain a color refinement result; The color refinement result is input as updated color information into the color estimation module of the next correction submodule connected in series with the current correction submodule, and the color adjustment result is input as updated degradation feature into the super-resolution module of the next correction submodule. The iterative processing is performed in this way, and the color adjustment result output by the last correction submodule is used as the super-resolution face image.

2. The method according to claim 1, characterized in that The training process of the image reconstruction model includes: Randomly obtain multiple original high-resolution face images from a preset public dataset; Adding a preset darkness factor and noise to the original high-resolution facial image to obtain a low-light low-resolution facial image corresponding to the original high-resolution facial image, wherein the preset darkness factor is used to control the light intensity to achieve low-light simulation; An image pair consisting of a low-light low-resolution face image and a corresponding original high-resolution face image is used as a sample, a preset neural network model is trained, and the trained model is used as an image reconstruction model.

3. The method according to claim 1, characterized in that: The step of performing feature changes on the shallow features to obtain multi-scale feature information includes: Inputting the shallow features into a plurality of serially connected encoders for multiple encodings in sequence to obtain a plurality of encoded feature maps; The plurality of encoded feature maps are upsampled by using a plurality of decoders connected in series to extract multi-scale feature information of the low-light and low-resolution facial image, and the number of the decoders is the same as the number of the encoders.

4. The method according to claim 3, characterized in that The shallow features are input into multiple serially connected encoders for multiple encodings in sequence to obtain multiple encoded feature maps, including: The shallow features are input into multiple branches with different sampling scales, and multiple semantic information of the shallow features are extracted by downsampling respectively. The output features of each branch are obtained through a spatial and channel reconstruction convolution module and a multi-dimensional collaborative attention mechanism module. The spatial and channel reconstruction convolution module is used to suppress redundant spatial information and amplify the weight of key feature information. The multi-dimensional collaborative attention mechanism module is used to capture the interdependence in the spatial dimension and the interaction between channels. Sort the output features of branches with different sampling scales from low to high according to the sampling scale; Upsample the output features of the branch with the lowest sampling scale by a preset multiple and use them as the fusion features of the current branch; The fusion features of the current branch are sequentially upsampled by a preset multiple, and fused with the output features of the next high sampling scale branch adjacent to the current branch to obtain a fusion result, wherein the sampling scale of the next high sampling scale branch is a preset multiple of the sampling scale of the current branch; The fusion result is convolved and residual processing is performed with the shallow features, and convolution processing is performed again to obtain an encoded feature map.

5. The method according to claim 4, characterized in that The process of upsampling the fusion features of the current branch by a preset multiple and fusing them with the output features of the next high sampling scale branch adjacent to the current branch includes: The fusion feature of the current branch is upsampled by a preset multiple as the first feature, the output feature of the next high sampling scale branch is used as the second feature, the first feature and the second feature are respectively input into the preset channel attention module to obtain the first channel output and the second channel output; The first channel output is multiplied by the second channel output to obtain a cross matrix; Performing a residual connection between the cross matrix and the first feature to obtain a first spatial input feature, and performing a residual connection between the cross matrix and the second feature to obtain a second spatial input feature; The first spatial input feature and the second spatial input feature are input into a preset spatial attention module to obtain a feature after spatial attention adjustment, and the feature after spatial attention adjustment is input into a fully connected layer to obtain a fusion feature.

6. A low-light facial image super-resolution reconstruction device, characterized in that: The device comprises: An image acquisition module, used for acquiring a low-light and low-resolution face image to be reconstructed; An image reconstruction module is used to input the low-light low-resolution face image into a pre-trained image reconstruction model to obtain a reconstructed super-resolution face image; wherein the image reconstruction model includes: a shallow feature extraction module, a spatial feature transformation module, and a de-degradation color correction module, wherein: The shallow feature extraction module is used to perform convolution processing on low-light and low-resolution face images to obtain shallow features; The spatial feature transformation module is used to perform feature transformation on the shallow features to obtain multi-scale feature information; The de-degradation color correction module is used to iteratively perform color correction and de-degradation processing on the multi-scale feature information to obtain a super-resolution face image, including: Performing a self-attention mechanism and a forward feedback process on the multi-scale feature information to obtain color information in the multi-scale feature information, and performing an interpolation and a convolution process on the multi-scale feature information to obtain a degraded feature; Inputting the color information and the degradation features into a de-degradation color correction module for iterative processing to obtain a super-resolution face image, wherein the de-degradation color correction module includes a plurality of correction submodules connected in series, the number of the correction submodules being the number of iterations, and the correction submodules including a color estimation module, a super-resolution module, a color adjustment module and a color refinement module; The color information and the degradation features are respectively input into a de-degradation color correction module for iterative processing to obtain a super-resolution face image, including: Inputting the color information into the color estimation module of the current correction submodule to obtain a color coefficient, and inputting the degradation feature into the super-resolution module of the current correction submodule to obtain a de-degraded feature; Inputting the color coefficient and the de-degradation feature into a color adjustment module of the current correction submodule, and performing color adjustment on the de-degradation feature using the color coefficient to obtain a color adjustment result; Inputting the color adjustment result and the color coefficient into the color refinement module of the current correction submodule to obtain a color refinement result; The color refinement result is input as updated color information into the color estimation module of the next correction submodule connected in series with the current correction submodule, and the color adjustment result is input as updated degradation feature into the super-resolution module of the next correction submodule. The iterative processing is performed in this way, and the color adjustment result output by the last correction submodule is used as the super-resolution face image.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images