A dynamic illumination face image quality enhancement method based on multi-scale attention mechanism

Abstract

A method for enhancing the quality of dynamic illumination facial images based on a multi-scale attention mechanism belongs to the field of digital image / video signal processing. The present invention designs a cascaded encoding and decoding network structure, which can be divided into a multi-scale wavelet feature extraction module and an image enhancement module. The facial image affected by illumination is subjected to key feature extraction by the multi-scale wavelet feature extraction module and then input into the image enhancement module to complete the enhancement of the image affected by dynamic illumination. In the multi-scale wavelet feature extraction module, wavelet transform downsampling is used to obtain feature information maps of different scales. The global context information in the image is extracted and fused through the attention unit inside the module. In the image enhancement module, based on the multi-scale attention idea, the network's perception of dynamically changing illumination is improved by obtaining information at different scales and a dual attention mechanism to complete the enhancement process. This technology has broad application prospects in criminal investigation, military reconnaissance, multimedia vision and other fields.

Description

Technical Field

[0001] The present invention belongs to the field of digital image / video signal processing, and in particular relates to a dynamic illumination face image quality enhancement method based on a multi-scale attention mechanism. Background Art

[0002] With the advancement of computer vision and multimedia technologies, people receive massive amounts of multimedia information, including images and videos, every day. Furthermore, with the development and widespread adoption of intelligent imaging devices and software, the means by which people acquire these images and videos are becoming increasingly diverse, including through smartphones, video cameras, remote sensing satellites, medical imaging equipment, and surveillance cameras. The acquisition process is often limited by equipment quality and environmental factors, which can affect image quality to varying degrees. This can lead to issues such as blur, uneven lighting, poor contrast, and unclear details. For example, in video surveillance of dynamic scenes, facial images are susceptible to changes in lighting, posing challenges to subsequent subjective viewing and automated analysis algorithms.

[0003] In recent years, deep learning technology has made significant progress in image quality enhancement. This has provided an opportunity to effectively improve the quality of facial images in dynamic lighting scenarios. In dynamic scenes, captured facial images are affected by factors such as object motion and changing lighting, resulting in complex lighting variations. Therefore, effective methods for enhancing the quality of dynamic lighting facial images are needed to improve the effectiveness of subsequent face recognition tasks. This approach has broad application value in the public security field.

[0004] Deep learning-based convolutional neural networks have been widely used in low-light image enhancement and have achieved remarkable results. However, existing methods focus on image restoration in static scenes with unchanged lighting, and lack attention to the impact of dynamic lighting changes in real scenes. However, dynamic lighting changes are common in real scenes, especially outdoor sports scenes. Vision science shows that the local receptive fields of neurons in the same area of ​​the primate visual cortex vary. This results in the influence of various complex lighting conditions (non-uniform, backlit, backlit, overexposure, etc.) that may reduce people's subjective perception and affect the performance of face recognition models.

[0005] Typically, captured images contain both high-frequency and low-frequency information. High-frequency information generally contains detailed texture information, while low-frequency information primarily contains illumination information. Wavelet transforms can reduce information loss while separating image information into high-frequency and low-frequency components. Introducing wavelet transforms can help networks perform time-frequency analysis. Furthermore, introducing multi-scale information into the network can help it capture global context, expand its receptive field, and better address the effects of dynamically changing illumination. Furthermore, attention mechanisms, based on the characteristics of human visual attention, have been observed to be suitable for addressing the coexistence of multiple dynamic illumination conditions. The attention mechanism can be used to adaptively select weights, improving feature expression and network recovery.

[0006] In response to the above-mentioned problems and observations, the present invention aims to design a facial image enhancement network that can adapt to various illumination variations and restore ideal illumination facial images. To this end, the present invention proposes a method for dynamic illumination facial image quality enhancement based on a multi-scale attention mechanism. The present invention designs a cascaded network structure, which can be divided into a multi-scale wavelet feature extraction module and an image enhancement module. The multi-scale wavelet feature extraction module extracts key features from the illumination-affected facial image, which is then input into the image enhancement module for further image restoration and enhancement, ultimately achieving an enhanced result. Both the multi-scale wavelet feature extraction module and the image enhancement module utilize a codec architecture. In the feature extraction module, wavelet transforms are used instead of the downsampling operation of conventional deep codec networks. Wavelet transform downsampling yields feature information maps at three different scales. Simultaneously, dual attention units and channel attention units process and fuse multi-scale contextual information. By incorporating multi-scale attention into the image enhancement module, feature information at different scales is processed to improve the network's ability to perceive dynamically changing illumination, thereby achieving an adaptive enhancement process. This model combines speed with dynamic illumination adjustment capabilities. This technology has broad application prospects in criminal investigation, military reconnaissance, multimedia vision and other fields. Summary of the Invention

[0007] The purpose of the present invention is to alleviate the problem of reduced image quality in dynamic lighting scenarios, which has rarely been paid attention to in existing studies. It mainly improves the influence of various non-uniform lighting degradation factors in facial images affected by dynamic lighting, improves its subjective effect and the performance of subsequent objective analysis. To this end, a method for enhancing the quality of facial images under dynamic lighting based on a multi-scale attention mechanism is provided. This method can enable the network to pay attention to situations under various different changing lighting conditions, effectively improving the quality of facial images. Among them, the multi-scale wavelet feature extraction module and the image enhancement module can significantly improve the network's ability to adjust to dynamic lighting, and effectively restore facial images under normal lighting from images affected by various different lighting conditions, while retaining the texture detail information of the background and the face.

[0008] The present invention is achieved by adopting the following technical means:

[0009] A method for enhancing the quality of facial images under dynamic lighting based on a multi-scale attention mechanism. This method adopts a multi-level encoding and decoding network framework. First, the facial image affected by dynamic lighting is input into the first-level multi-scale wavelet feature extraction module to extract and fuse multi-scale feature information. These feature information are used to guide the enhancement process of subsequent modules; then, the feature information of the previous level is input into the image enhancement module composed of the multi-scale residual attention idea to complete the final image enhancement process; finally, according to the dynamic lighting characteristics, a combination of three loss functions is selected to complete the training and optimization process of the model. The overall network of this method is shown in the attached figure. Figure 1 As shown in the attached figure, it is mainly divided into two modules: multi-scale wavelet feature extraction module and image enhancement module. Figure 2 , as shown in 3.

[0010] Existing research has mostly focused on image restoration in static scenes with unchanged illumination, but has lacked attention to the impact of dynamic illumination changes in actual scenes. However, dynamic illumination changes are common in actual scenes, and in particular, they can affect the recognition quality of facial images taken in the wild. To address the issue of dynamic illumination effects, the present invention uses a multi-scale wavelet feature extraction module to extract feature information from a variety of different illumination conditions, and employs a multi-scale attention mechanism to enable the network to adaptively assign different weights to different features, helping the network to recover facial and background detail information contained in normal illumination from the influence of dynamic illumination. At the same time, through experiments, a loss function combination suitable for model learning of facial images with dynamically changing illumination is proposed. Due to the current lack of large-scale facial datasets under the influence of multiple illumination conditions, a relighting method is used to construct a facial dataset affected by multiple illumination conditions for training and verification.

[0011] The method of the present invention is divided into four parts: construction of a multi-scale wavelet feature extraction module, construction of an image enhancement module, selection of a loss function for dynamic lighting effects, and preparation of a multi-lighting face dataset.

[0012] The method specifically comprises the following steps:

[0013] 1) Construction of multi-scale wavelet feature extraction module:

[0014] The main function of this module is to extract a larger receptive field and global context information, thereby guiding the subsequent network to complete the image restoration process. This network is mainly composed of an encoder-decoder network structure.

[0015] In the first step, the input image is passed through a 3×3 convolutional layer to expand the number of image channels. Since the image input to the network is a three-channel RGB image, when the network needs to obtain more information from the image, the input image needs to be increased in dimension to improve the network's feature information extraction performance.

[0016] The second step is to downsample the image through a cascade of three wavelet transforms. The wavelet transform method has the effect of reducing information loss during the downsampling process and can extract both high-frequency and low-frequency information of the image. High-frequency information generally contains texture details of the image, while low-frequency information contains illumination information. This enables the network to acquire time-frequency analysis capabilities.

[0017] In the third step, after generating feature information at three different scales through cascaded three wavelet transforms, the feature map is adjusted to the required dimension through a convolutional layer and a PReLU activation function, and then input into the dual attention unit. The dual attention unit can effectively extract useful feature information at different scales while suppressing the propagation of less useful features downward;

[0018] In the fourth step, the output features of each dual-attention unit are concatenated with the feature image information of the previous level after inverse wavelet transformation. After dimensionality reduction through the convolution layer and activation function, they are fused with the channel attention unit and then fed to the next level. After this process is repeated twice, the output feature map is concatenated with the feature map obtained in the first step, and the convolution layer and activation function are used to perform the final feature extraction and fusion of the channel attention unit to obtain image enhancement feature information. This step extracts features in multiple channel dimensions. The channel attention unit can help the network focus on useful feature information on different channels.

[0019] The Dual Attention Unit (DAU) primarily consists of a spatial attention branch (SA) and a channel attention branch (CA). The SA branch exploits the spatial correlation of features, while the CA branch exploits the inter-channel relationship of convolutional feature maps. In the dual attention unit, the feature map undergoes convolution and activation before being fed into the SA and CA branches, respectively. The SA branch first applies global average pooling and max pooling to the features along the channel dimension. The two are then concatenated and multiplied with the corresponding channels to form a spatial attention feature map. The feature map is then convolved and sigmoid activated to obtain the spatial attention feature map. The CA branch first encodes global context through global average pooling to generate a channel feature map. The feature map passes through a convolutional layer and then a sigmoid activation. The resulting weights are multiplied with the corresponding channels to generate the channel attention feature map. Finally, the feature maps of the two branches are concatenated and multiplied with the input feature map to form the feature map of the dual attention unit. The channel attention unit (CAU) is mainly composed of the dual attention unit (DAU) without the spatial attention branch.

[0020] 2) Image enhancement module construction:

[0021] After obtaining the global context information, the network needs to start focusing on the image enhancement process. Unlike the current images under the influence of a single lighting type, it is necessary to pay attention to scenes under a variety of different lighting conditions, including backlight, overexposure, uneven lighting, etc. Therefore, the network of the enhancement process needs to pay attention not only to the changes in light, but also to the parts hidden by bright or dark. The present invention adopts a multi-scale residual block (Multi-Scale Residual Block MSRB) proposed in the MIRnet network to complete the construction of the image enhancement module (Image Enhancement Block IEB).

[0022] Two multi-scale residual modules (MSRBs) are adopted as the backbone structure in the image restoration module. This choice can generate spatially accurate output by maintaining high-resolution representation while receiving rich contextual information from low resolution, maintaining a trade-off between computational effort and performance.

[0023] The Multi-Scale Residual Module (MSRB) consists of multiple fully convolutional parallel streams. It allows information exchange across parallel streams to integrate high-resolution feature maps with the help of low-resolution feature maps, thereby receiving rich contextual information. The information received by the MSRB is downsampled three times to obtain a multi-scale feature map, which helps the network obtain spatial context and semantic information from coarse to fine. The feature information between different scales obtained after input into the dual attention unit is exchanged through continuous up- and down-sampling. The feature selection module receives feature information from different scales for feature fusion selection. The feature information obtained from different scales can be integrated through the feature selection module to ultimately generate a feature information map mapping of global information. At the same time, the introduction of residual operations makes network training easier.

[0024] The feature selection module dynamically adjusts the network's receptive field through two operations: fusion and selection. The fusion operation generates a global feature map by splicing information from multiple resolution scales. After the feature map undergoes global average pooling and convolution activation, it leverages the performance of the Softmax activation function to multiply it with feature information from other scales to generate three different selected mappings, which are then summed to adaptively recalibrate the output feature map.

[0025] 3) Selection of loss function for dynamic lighting effects

[0026] The L1 and L2 loss functions are two pixel-level loss functions commonly used in image quality enhancement. They help the network recover lost color and brightness. They each have their own advantages. The L1 loss function effectively preserves color and brightness because errors are equally weighted regardless of local structure. The L2 loss tends to penalize larger errors but is more tolerant of smaller ones. For restoring facial image quality under dynamic lighting, facial images can be affected by lighting that is too dark or too bright, and distributed across different regions. We want the network to fully restore the image content of the facial portion while placing lower demands on other areas, so we choose the L2 loss as the primary loss function.

[0027] However, when using L2 loss alone, it was found that although most of the information of facial images affected by lighting can be restored, the results of preserving facial features such as face shape, skin color, and eye color are not good enough, and artifacts are prone to occur. Therefore, we refer to the structural similarity (SSIM) loss to further preserve the characteristics of structure and texture information, so we added the structural similarity loss to the total loss function.

[0028] Although pixel-level loss provides very accurate guidance for image restoration, it tends to ignore some difficult-to-measure features. For example, when two pixels are constrained by pixel-level distance, a pixel can take the value of any pixel within the error radius, leading to the neglect of some detailed information and the occurrence of color difference and artifacts. Therefore, we also add perceptual loss. Perceptual loss can detect changes at the feature level. For facial images under dynamically changing lighting, it can help the network model pay attention to the differences in color and texture details, and supervise the model to learn to more accurately reconstruct facial images under normal lighting. Therefore, the total loss function is as follows:

[0029] L Total =L L2oss +L SSIM +L Perceptual (1)

[0030] 4) Preparation of face datasets with various illuminations

[0031] To address the proposed problem of dynamically illuminated facial images, end-to-end network training requires a large number of paired datasets to ensure network effectiveness. However, existing datasets mostly focus on natural images in low light or at varying exposure levels, lacking datasets that address the impact of dynamic illumination changes. Furthermore, capturing paired facial images of people with dynamically changing illumination in outdoor scenes is extremely difficult. Therefore, a generative approach is employed to address this problem. First, a publicly available dataset of disguised and made-up faces is used, and after alignment and cropping, a subset of facial images is selected as the target dataset.

[0032] The relighting method is used to generate face images under different lighting environments. This method can generate face images under a variety of different lighting environments according to the light probe specification. The present invention collects face images in the target data set and generates a large number of paired face data under different lighting environments through this method. The data is divided into training set, test set and validation set. Some of the generated results are shown in the attached figure. Figure 8 shown.

[0033] The deep learning network in this technology is trained in an end-to-end process. The network input is randomly cropped image data and then trained until the loss function converges. During the testing process, a self-built test set is used to obtain enhanced results.

[0034] Compared with the prior art, the present invention has obvious advantages and significant effects:

[0035] The present invention addresses the issue of facial image quality affected by dynamic lighting, which has been largely ignored in current research, by proposing a method for enhancing the quality of facial images under dynamic lighting based on a multi-scale attention mechanism. The method first generates a large-scale dataset of facial images under various lighting conditions through relighting. Wavelet transform is then used to generate multi-scale input variations with different spatial resolutions when constructing an enhancement network. Through feature extraction and fusion of dual-attention units and channel-attention units, the network obtains sufficient global contextual information and detail information at an earlier stage. This information can be dynamically and adaptively restored to facial images under different lighting conditions in an image enhancement module composed of multi-scale residual blocks. Objective and subjective comparisons with various advanced methods after training on dynamic lighting facial datasets show that the method can handle a variety of different lighting distributions, retain more key information about the face and background objects without generating color difference, and restore facial images under normal lighting to exhibit better enhancement effects, demonstrating significant advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 , overall network architecture diagram;

[0037] Figure 2 , the network architecture of the multi-scale wavelet feature extraction module;

[0038] Figure 3 ,Network architecture of image enhancement module;

[0039] Figure 4 , the network architecture of the multi-scale residual module;

[0040] Figure 5 , the network architecture of dual attention units;

[0041] Figure 6 , the network architecture of the channel attention unit;

[0042] Figure 7 ,Network architecture of feature selection module;

[0043] Figure 8 ,Partial results of generated face data under various lighting conditions;

[0044] Figure 9 Comparison of some subjective results of different methods; (Horizontally, they are: input image, LIME, GLAD, DSLR, MBLLEN, MIRNet, Ours, GroundTruth; Vertically, they are: different input test images) DETAILED DESCRIPTION

[0045] The following describes an embodiment of the present invention with reference to the accompanying drawings:

[0046] This paper consists of three parts: first, preparing a dataset of faces under dynamic lighting. We use a relighting method to embed different lighting types into normal lighting face images to generate a large-scale paired dataset, addressing the lack of dynamic lighting datasets. Second, we construct a network model based on a multi-scale attention mechanism and a loss function. Finally, we load the data and set up the training. The specific processes of each of these three parts are described below:

[0047] (1) Preparation of face dataset under dynamic lighting

[0048] a) Face dataset preprocessing

[0049] The facial data in the dataset used in this method all come from a dataset of faces with disguise and makeup proposed by Wang et al. in 2016. Because facial images with makeup and disguise and in different backgrounds are more similar to facial images captured in real environments, the present invention detects and aligns facial images using the RetinaFace face detection algorithm. The obtained images are then uniformly cropped and divided into a size of 256×256 pixels, and 520 facial images are selected as the target dataset.

[0050] b) Generation of face data with multiple dynamic lighting conditions and data set division

[0051] The method for generating multiple dynamic lighting face data adopts the relighting method proposed by Jiang et al. in 2021. We input the face image data in the target dataset into the network to generate face images with different lighting conditions. Some of the results are shown in the attached figure. Figure 8 As shown. The data set used in the present invention generated a total of 520 groups of 72 continuous illumination changes in each group, totaling 37,440 face images. Taking into account the trade-off between training time and performance factors, 10 typical illumination conditions were selected from each group, totaling 5,200 images. Among these 520 groups, 16 groups were randomly selected as test sets, and the remaining 504 groups were training sets. The training set totaled 5,040 images, and the test set totaled 160 images. In order to verify the generalization ability of the network, that is, the recovery performance under other illumination conditions that have not been trained, the 16 groups of original face images with all 72 continuous illumination changes in the test set constitute a verification set, totaling 1,152 images.

[0052] (2) Construction of network model and loss function based on multi-scale attention mechanism

[0053] In the second part, we will introduce the multi-scale wavelet feature extraction module, image enhancement module and the construction of the overall network, and explain the detailed settings of the loss function used.

[0054] a) Network construction of multi-scale wavelet feature extraction module

[0055] Multi-scale wavelet feature extraction module such as Figure 2As shown in the figure, the main implementation method of the multi-scale wavelet feature extraction module is to replace the traditional downsampling process with wavelet transform to achieve the transformation of image scale and channel dimension. After the transformation, the feature information at each level of different scales is extracted by the dual attention unit and then the wavelet inverse transform is performed step by step to complete the upsampling process. The channel attention unit is used for fusion step by step to complete the extraction of feature information. Specifically, the image input to the module first undergoes a layer of 3×3 convolution to expand the original 3 channels of the image to 64 channels without changing the image size. The convolution layer sets the convolution kernel to 3, the padding to 1, and the bias to False. Then, the obtained feature map is downsampled and expanded to 3 different scales through three consecutive first-order two-dimensional discrete wavelet transforms. After each wavelet transform, it is necessary to pass a layer of 3×3 convolution layer and activation function to adjust the transformed features to 128, 256, and 512 channels and reduce the scale by 2 times, 4 times, and 8 times respectively as the input of the first, second, and third levels. The number and size of the output channels of each level are the same as the input The inputs remain consistent. This operation can expand the network's receptive field while controlling the number and scale of feature channels within the required range. The convolution layer settings used are all set to a convolution kernel of 3, a padding of 1, a stride of 1, and a bias of False. The activation function uses the PRelu activation function. The wavelet transform is a two-dimensional Haar wavelet transform. The wavelet transform can decompose an image into high-frequency information and low-frequency information. The frequency of an image is an indicator of the severity of grayscale value changes and is the gradient of grayscale values ​​in a plane space. High frequency refers to a large change in grayscale values ​​between adjacent areas of the image, that is, a large gradient. Low frequency refers to a small change in grayscale values ​​between adjacent areas of the image, that is, a small gradient. The wavelet transform operation helps reduce the loss of feature information during downsampling, allowing the network to adaptively focus on features that require attention, allowing the network to acquire time-frequency analysis capabilities. Secondly, the feature information at three different scales is upsampled after extracting useful feature information through dual attention units. First, the third-level output is upsampled by the first-order two-dimensional discrete wavelet inverse transform. The feature information obtained after the 1×1 convolution layer and the feature information obtained in the second level are concatenated and input into the 3×3 convolution layer and the channel attention unit for efficient feature fusion extraction, and then upsampled again by the first-order two-dimensional discrete wavelet inverse transform. The 1×1 convolution layer uses the convolution kernel set to 1, padding to 0, step size to 1, and bias to False. The 3×3 convolution layer uses the convolution kernel set to 3, padding to 1, step size to 1, and bias to False.Finally, the output of the second level and the output of the first level are concatenated again and input into the 3×3 convolution layer and the channel attention unit for efficient feature fusion extraction. The result is concatenated with the 64-dimensional channel feature map obtained by the first layer of convolution and input into the structure of the 3×3 convolution layer and the channel attention unit again to generate a fused multi-scale feature information map to guide the subsequent reconstruction and enhancement process. The 3×3 convolution layer sets the convolution kernel to 3, the stride to 1, the padding to 1, and the bias to False.

[0056] Dual attention units such as Figure 5As shown in the figure, the dual attention unit is designed to accept feature information from different scales, efficiently extract useful features, and suppress the propagation of useless features. The entire module is built on a residual dual-branch structure. First, the input feature information is fully cross-channeled through a 3×3 convolution, an activation function, and a 3×3 convolution. This increased nonlinearity in the feature information helps the network converge more easily. The number of input and output channels of the convolutional layer is consistent with the 64-dimensional output of the previous level. The 3×3 convolution uses a kernel size of 3, a stride of 1, a padding of 1, a bias of False, and a PReLU activation function. The feature information then passes through the spatial attention branch (SA) and the channel attention branch (CA). The outputs of the two branches are concatenated and fused through a 1×1 convolution. The outputs are then added to the dual attention unit input features to form a residual connection, ultimately obtaining the dual attention feature map. The 1×1 convolution layer uses a kernel of 1, a padding of 0, a stride of 1, and a bias of False. In the SA branch, global average pooling and maximum pooling operations are first applied to the input features respectively. After splicing, the two are multiplied with the SA branch input through a 5×5 convolution and sigmoid activation operation to form a spatial attention feature map. The 5×5 convolution sets the convolution kernel to 5, padding to 1, step size to 1, and bias to False. The CA branch first encodes the global context information through adaptive global average pooling to generate a channel feature map. The CA branch input feature map first passes through 1 1×1 convolution, 1 Relu activation function, 1×1 convolution, and 1 sigmoid activation. The generated weights are multiplied with the CA branch input feature map to generate a channel attention feature map. The 1×1 convolution layer uses the convolution kernel set to 1, padding to 0, step size to 1, and bias to False. The first convolution reduces the channel number feature dimension and divides the output channel number by 8. The second convolution increases the feature dimension and multiplies the output channel number by 8. Finally, the attention feature maps from the two branches are concatenated and fused with the input feature map to form a dual attention feature map. The 1×1 convolution is used to reduce the number of output feature channels to the same number of input feature channels. The 1×1 convolution layer uses a convolution kernel of 1, padding of 0, stride of 1, and bias of False. The channel attention unit is formed by removing the spatial attention branch SA from the dual attention unit. The basic parameter settings remain the same, such as Figure 6 shown.

[0057] b) Network construction of image enhancement module

[0058] Image enhancement modules such as Figure 3 As shown in Figure 1, the network construction of the image enhancement module is mainly composed of two cascaded multi-scale residual modules (MSRB). The output of the last-level MSRB module undergoes a convolution with a kernel of 3, padding of 1, stride of 1, and bias of False, and is added to the input of the image enhancement module to form a residual connection.

[0059] Multi-scale residual modules such as Figure 4 As shown in the figure, in the multi-scale residual module (MSRB), first, the input features are first subjected to two consecutive downsampling operations with magnifications of 2 and 4 respectively (the scale is reduced by 2 times, 4 times, and the number of channels is increased by 2 times, 4 times) to obtain feature information on three scales (input scale, scale reduced by 2 times, and scale reduced by 4 times). After the feature information at each scale is extracted by the dual attention unit, the feature information at different scales is exchanged in the intermediate stage through upsampling (the scale is increased by 2 times, 4 times, and the number of channels is reduced by 2 times, 4 times) or downsampling operations. The feature information from the three scales input at each level is input into the feature selection module for feature fusion selection. Then, the output of the feature selection module at each level is input into the feature selection module through the dual attention unit and then through the upsampling operation. Finally, the output of the module is added to the input feature map of the MSRB module after 1 convolution with a kernel of 3, padding of 1, stride of 1, and bias of False to form a residual connection and then output. The downsampling process uses anti-aliasing downsampling. The input image is first smoothed using box_filter and the image scale is reduced by 3×3 convolution. The 3×3 convolution sets the kernel to 3, padding to 0, step size to 1, and bias to False. Then, 1×1 convolution is used to complete the downsampling operation of increasing the number of feature channels. The 1×1 convolution sets the kernel to 1, padding to 0, step size to 1, and bias to False. The upsampling process uses bilinear interpolation to increase the image scale and 1×1 convolution to complete the downsampling of reducing the number of feature channels. The 1×1 convolution sets the kernel to 1, padding to 0, step size to 1, and bias to False.

[0060] Feature selection modules such as Figure 7As shown in the figure, the feature selection module dynamically adjusts the receptive field of the network through two operations of fusion and selection to extract useful feature information. After combining the feature information from three different scales, the module successively reduces the number of input feature channels to the number of input channels divided by 8 through one global average pooling, one convolution layer with a kernel of 1, padding of 0, step size of 1, and bias of False, and then increases the number of feature channels to the same as the number of input channels through one PRelu activation function, one convolution layer with a kernel of 1, padding of 0, step size of 1, and bias of False. After one Softmax layer, the selected fused features are obtained. Finally, this set of features are multiplied and added with the feature maps of the corresponding scale of the input to complete the multi-scale feature selection and fusion operation.

[0061] c) Overall network construction

[0062] The overall network framework is as follows Figure 1 As shown, considering the trade-off between performance and computational efficiency, the present invention uses a cascaded multi-scale wavelet feature extraction module and two image enhancement modules. The cascade order is: multi-scale wavelet feature extraction module, image enhancement module, and image enhancement module. At the same time, to reduce the occurrence of gradient vanishing and overfitting problems, the present invention adds the input and the output of the last module to form a global residual connection, making the network more flexible.

[0063] d) Loss function setting

[0064] During the network training process, the total loss function is shown in formula (1):

[0065] L Total =L L2oss +L SSIM +L Perceptual (1)

[0066] The definition of loss function is crucial to the performance of the network model. The loss function of the generative network designed in this paper consists of three parts: pixel-level loss (L L2oss ), institutional similarity loss (L SSIM ) and perceptual loss (L Perceptual ), the weights are consistent during training.

[0067] Specifically, L L2oss Used to calculate the mean square loss between the enhanced network results and the Ground-truth, L SSIM It is used to calculate the brightness contrast structure difference between the enhanced network result and the Ground-truth, and L Perceptualis the perceptual loss, which is used to calculate the mean square loss between the feature maps obtained by the enhanced network result and the ground-truth after the pre-trained VGG network, as shown in formulas (2), (3) and (4):

[0068]

[0069]

[0070]

[0071] Among them, W and H refer to the width and height of the input image respectively, and F i Refers to the enhanced result of the network, GT refers to the Ground-truth corresponding to the input, V gg (·) corresponds to the pre-trained VGG network operation. This paper selects the output results of the first 30 layers of the pre-trained VGG network for calculation, μ Fi and μ GT Represent the average value of the enhanced result and Ground-truth, and Represents the variance of the enhanced result and Ground-truth, Represents the covariance between the enhanced result and the Ground-truth, C1=(k1L) 2 C2=(k2L) 2 Represents two stable variables, L is the dynamic range of the pixel is 255, k1 = 0.01, k2 = 0.03.

[0072] (3) Data loading and training and testing settings

[0073] a) Data loading

[0074] The paired images in the training set are randomly cut into a size of 128×128 pixels, and image preprocessing methods such as folding and rotation are performed to enhance the data before inputting into the network for training.

[0075] b) Model training

[0076] The model training was performed using the Pytorch 1.4.0 framework on a Windows system with an Nvidia TITAN XP GPU and an Intel(R) Core(TM) i5-10600KF CPU. The batch normalization size was set to 4 and the initial learning rate was set to 10. -4, the number of iterations is set to 200, and the learning rate is halved every 25 iterations starting from the 150th iteration. The optimizer used is the Adam optimizer with β1=0.9, β2=0.999, and eps=10 -8 , weight_decay=10 -8 .

[0077] c) Model testing

[0078] In order to verify the effectiveness of the present invention, subjective visual effects and objective numerical indicators are used to evaluate the enhancement effect. The present invention and other existing related methods (LIME, GLAD, DSLR, MBLLEN, MIRNet) are trained using the same training data set. The subjective visual effects of the method of the present invention and other existing methods are compared as shown in the accompanying figure. The objective results use two commonly used image quality evaluation indicators, namely Peak Signal to Noise Ratio PSNR and Structural Similarity (SSIM). In the present invention, the network can receive RGB color images of any size, and the data set may not contain paired clear-low-quality image pairs. However, in order to facilitate the measurement of PSNR and SSIM indicators, we used paired clear-low-quality image pairs in the test set and validation set in the test. The results are shown in Table 1.

[0079] Our subjective comparison test results are as follows Figure 9 As shown, from the subjective results, it can be seen that compared with other existing methods, the method proposed in the present invention can handle many different lighting distributions (including lighting types involved in and not involved in training), and can well restore facial images under normal lighting from the influence of various dynamic lighting problems. At the same time, more facial and object features will be retained without color difference. From the perspective of objective indicators, the present invention also shows great advantages compared with other methods. Therefore, considering both subjective and objective aspects, the results of the present invention are better than those of existing methods.

[0080] Table 1 Objective results of different methods on the test set and validation set

[0081]

Claims

1. A dynamic illumination face image quality enhancement method based on multi-scale attention mechanism, characterized by The following steps are involved: (1) Preparation of face dataset under dynamic lighting a) Face dataset preprocessing The face images are detected and aligned using the RetinaFace face detection algorithm. The resulting images are then uniformly cropped and divided into 256×256 pixel sizes as the target dataset. b) Generation of face data with multiple dynamic lighting conditions and data set division Various dynamic lighting face data generation methods use relighting methods; (2) Construction of network model and loss function based on multi-scale attention mechanism a) Network construction of multi-scale wavelet feature extraction module The main implementation method of the multi-scale wavelet feature extraction module is to replace the traditional downsampling process with wavelet transform to achieve the transformation of image scale and channel dimension; specifically, the image input to the module first passes through a convolution layer to expand the original 3 channels of the image to 64 channels without changing the image size. The convolution layer sets the convolution kernel to 3, padding to 1, and bias to False; then, the obtained feature map is downsampled and expanded to 3 different scales through 3 consecutive 2D discrete wavelet transforms. It should be noted that the downsampling operation is performed at the feature channel level rather than the pixel level. After each wavelet transform, a 3×3 convolution layer and an activation function are required to adjust the transformed features to 128, 256, and 512 channels and reduce the scale by 2 times, 4 times, and 8 times respectively as the input of the first, second, and third levels respectively. The number of output channels and size of each level are consistent with the input. The convolution layer is used to set the convolution kernel to 3, padding to 1, step size to 1, bias to False, the PRelu activation function used, and the wavelet transform used is the 2D Haar wavelet transform. The wavelet transform decomposes the image into high-frequency information and low-frequency information. Secondly, the feature information at three different scales is respectively extracted through the dual attention unit to extract useful feature information. The feature information obtained by upsampling the third level through the 2D discrete wavelet inverse transform and the feature information obtained by the second level are input into the convolution layer plus the channel attention unit through the splicing operation for efficient feature fusion extraction. Finally, the output of the second level is subjected to the same operation steps as the first level and the initial input feature map is input again into the structure of the convolution plus channel attention unit to generate a multi-scale feature information map to guide the subsequent reconstruction and enhancement process. The convolution layer settings used are: convolution kernel 3, step size 1, padding 1, bias False. The dual attention unit is used to receive feature information from different scales; the input The feature information is fully cross-channel interacted by performing one convolution, one activation function, and one convolution layer to increase the nonlinearity of the feature information. The number of input and output channels of the convolution layer is the same, the convolution kernel size is 3, the bias is False, and the activation function is the PRelu activation function. Then, the feature information after the above steps passes through the spatial attention branch and the channel attention branch respectively. The outputs of the two branches are fused by splicing and 1×1 convolution, and then added to the input features to form a residual connection, and finally a dual attention feature map is obtained. The spatial attention branch first applies global average pooling and maximum pooling operations to the features respectively. After splicing the two, they are multiplied with the corresponding channels through a 5×5 convolution and a sigmoid activation operation to form a spatial attention feature map; the channel attention branch first encodes the global context information through global average pooling to generate a channel feature map. The feature map passes through a 1×1 convolution, a Relu activation function, a 1×1 convolution, and a sigmoid activation. The generated weights are multiplied with the corresponding channels to generate a channel attention feature map; finally, the attention maps from the two branches are fused through a splicing operation and a 1×1 convolution and added to the input feature map to form a dual attention feature map; b) Network construction of image enhancement module The image enhancement module is composed of two multi-scale residual modules in cascade. The input of the first-level module is added to the output of the second-level module after passing through a convolutional layer with a kernel of 3, padding of 1, stride of 1, and bias of False to form a residual connection. In the multi-scale residual module, first, the input features are down-sampled twice with magnifications of 2 and 4 respectively to obtain feature information at three scales. The feature information at each scale is extracted by the dual attention unit. Secondly, in the intermediate stage, the feature information at different scales is exchanged through upsampling or downsampling operations. The feature information from the three scales of each level is input into the feature selection module for feature fusion selection. Then, the output of each level of feature selection module is input into the feature selection module through the dual attention unit and then through the upsampling operation. Finally, the output of the module is convolved with a kernel of 3, padding of 1, stride of 1, and bias of False, and is added to the initial input feature map to form a residual connection and then output. The feature selection module dynamically adjusts the network's receptive field through two operations: fusion and selection. This module adds feature information from three different scales, and then passes through one global average pooling, one convolution layer with a kernel of 1, padding of 0, step size of 1, and bias of False, one PRelu activation function, three convolution layers with a kernel of 1, padding of 0, step size of 1, and bias of False, and one Softmax layer to obtain the selected fused features. Finally, this set of features is multiplied with the input feature maps of different scales and then added together to complete the multi-scale feature selection and fusion operation. c) Overall network construction The overall network is composed of a multi-scale wavelet feature extraction module and two image enhancement modules in cascade order: multi-scale wavelet feature extraction module, image enhancement module, and image enhancement module. The input is added to the output of the last module to form a global residual connection. d) Loss function setting During the network training process, the total loss function is shown in formula (1): L Total =L L2oss +L SSIM +L Perceptual (1) The definition of loss function is crucial to the performance of the network model. The loss function of the designed generative network consists of three parts: pixel-level loss (L L2oss ), structural similarity loss (L SSIM ) and perceptual loss (L Perceptual ), the weights are consistent during training; L L2oss Used to calculate the mean square loss between the enhanced network results and the Ground-truth, L SSIM It is used to calculate the brightness contrast structure difference between the enhanced network results and the Ground-truth, and L Perceptual is the perceptual loss, which is used to calculate the mean square loss between the feature maps obtained by the enhanced network result and the ground-truth after the pre-trained VGG network, as shown in formulas (2), (3) and (4): Among them, W and H refer to the width and height of the input image respectively, and F i Refers to the enhanced result of the network, GT refers to the Ground-truth corresponding to the input, V gg (·) corresponds to the pre-trained VGG network operation, and the output results of the first 30 layers of the pre-trained VGG network are selected for calculation. and μ GT Represent the average value of the enhanced result and Ground-truth, and Represents the variance of the enhanced result and Ground-truth, Represents the covariance between the enhanced result and the Ground-truth, C1=(k1L) 2 and C2=(k2L) 2 They represent two stable variables, L is the dynamic range of the pixel is 255, k1 = 0.01, k2 = 0.03; (3) Data loading and training and testing settings a) Data loading The paired images in the training set are randomly cut into 128×128 pixel values, and then input into the network for training after data augmentation; b) Model training The optimizer used for model training is the Adam optimizer.

Images