Face super-resolution reconstruction detection method, device and equipment

Abstract

The invention discloses a face super-resolution reconstruction detection method, device and equipment, and relates to the field of computer vision, and the key points of the technical scheme are as follows: detecting whether a low-resolution image has a to-be-confirmed face region or not; when it is detected that the low-resolution image has a to-be-confirmed face region, inputting the image of the to-be-confirmed face region into a face super-resolution reconstruction network for super-resolution reconstruction so as to confirm the face region in the image; wherein an enhanced generative adversarial network constructed by a generator and a discriminator serves as the face super-division reconstruction network, a DB residual dense block of the generator is replaced by a plurality of stacked Swin Transform block structures, and a reconstruction loss function of the discriminator is introduced to be used for judging whether the reconstruction loss function is a cross entropy loss function of a face region or not. The problem that the quality of the generated image is affected due to the fact that certain noise and artifacts still exist in the process of reconstructing the image resolution by an ESRGAN provided in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and more specifically, to a method, device and equipment for super-resolution reconstruction of human faces. Background Art

[0002] Face detection is an important branch of computer vision and the first key step in all face image processing tasks (such as face recognition, face tracking, pose estimation, etc.). It is widely used in many fields such as digital video processing, human-computer interaction, social media applications, and user behavior analysis. With the development of technology, the application background of face detection has exceeded the scope of traditional face recognition systems and has become an important tool for ensuring public safety and improving user experience.

[0003] In practical applications, due to the limitations of shooting conditions (such as shooting distance, lighting conditions, etc.), the acquired face images are often of low quality, manifested as low resolution, blur, high noise and other problems. These low-quality face images bring great challenges to subsequent face detection and recognition, reducing the accuracy and reliability of the recognition system. In order to overcome the challenges brought by low-quality face images, super-resolution reconstruction technology has been introduced into the field of face detection and recognition. Super-resolution reconstruction technology is an image processing technology that can generate high-resolution images through a series of low-resolution images. This method utilizes interpolation technology, deep learning and other technical means in image processing. By interpolating or learning low-resolution images, the pixel values ​​in the high-resolution image are obtained, thereby making the details of the image clearer. Traditional super-resolution reconstruction methods mainly rely on interpolation algorithms or reconstruction filters. Although these methods can improve the resolution of the image to a certain extent, they often cannot effectively restore the high-frequency detail information in the image, resulting in limited quality of the reconstructed image.

[0004] ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) is an image super-resolution algorithm based on generative adversarial networks (GANs), which aims to convert low-resolution (LR) images into high-resolution (HR) images while maintaining or even improving the image quality. However, ESRGAN may have difficulty accurately reconstructing all high-frequency information for images with complex textures, details, or dynamic scenes, especially in areas such as edges and texture details, where artifacts or blurring may occur, which affects the quality of the generated image (such as blurring, jagged edges, etc.). Summary of the invention

[0005] The present application provides a face super-resolution reconstruction detection method, device and equipment, which solves the problem that the ESRGAN network provided by the prior art still has certain noise and artifacts in the process of reconstructing image resolution, which affects the quality of the generated image.

[0006] In a first aspect of the present application, a method for super-resolution reconstruction detection of a face is provided, the method comprising:

[0007] Detect whether there is a face area to be confirmed in the low-resolution image;

[0008] When it is detected that there is a face area to be confirmed in a low-resolution image, the image of the face area to be confirmed is input into a face super-resolution reconstruction network for super-resolution reconstruction to confirm the face area in the image; wherein, an enhanced generative adversarial network constructed by a generator and a discriminator is used as the face super-resolution reconstruction network, and the DB residual dense block of the generator is replaced by multiple stacked Swin Transformer block structures, and the reconstruction loss function of the discriminator is introduced into a cross entropy loss function for determining whether it is a face area.

[0009] In some embodiments, the low-resolution image is detected to see if there is a face area to be confirmed in the following manner: calling a face detection network to detect the low-resolution image to obtain a face detection area;

[0010] The face detection area is divided according to the confidence threshold. If it is greater than the first confidence threshold, the face detection area is a confirmed face area. If it is less than the first confidence threshold and greater than the second confidence threshold, the face detection area is a face area to be confirmed. If it is less than the second confidence threshold, the face detection area is a non-face area.

[0011] In some embodiments, the generator includes a shallow feature extraction network and a deep feature extraction network;

[0012] The shallow feature extraction network is a 3×3 convolutional layer, which is used to map the image of the face area to be confirmed into a first feature map of size C×W×H; where C is the number of channels, W and H are the width and height of the image respectively.

[0013] The deep feature extraction network includes multiple stacked Swin Transformer block structures, multiple 3×3 convolutional layers, an upsampling layer and at least one activation function layer.

[0014] In some embodiments, each of the Swin Transformer block structures includes a first Swin Transformer sub-block structure, a first convolutional layer, a second Swin Transformer sub-block structure, and a second convolutional layer; wherein the first Swin Transformer sub-block structure and the second Swin Transformer sub-block structure are both composed of three Swin Transformer blocks connected in sequence;

[0015] The first Swin Transformer sub-block structure captures the local features and context information of the first feature map through the self-attention mechanism and relative position encoding of three Swin Transformer blocks to obtain a second feature map of size 2C×W×H;

[0016] The first convolutional layer is used to fuse the first feature map with the second feature map to obtain a third feature map of size 2C×W×H;

[0017] The second Swin Transformer sub-block structure captures the local features and context information of the third feature map through the self-attention mechanism and relative position encoding of three Swin Transformer blocks to obtain a fourth feature map of size 3C×W×H;

[0018] The second convolutional layer is used to normalize the first feature map, the third feature map, and the fourth feature map to a fifth feature map of size C×W×H.

[0019] In some embodiments, the Swin Transformer block is composed of two layer normalization layers, a multi-head self-attention mechanism module, and a multi-layer perceptron module;

[0020] Among them, the first feature map is used as the input of a layer normalization layer, the output of a layer normalization layer is used as the input of the multi-head self-attention mechanism module, the output of the multi-head self-attention mechanism module and the first feature map are fused to obtain a first fusion result as the input of another layer normalization layer, the output of another layer normalization layer is used as the input of the multi-layer perceptron module, and the output of the multi-layer perceptron module and the second fusion result obtained by fusion of the first fusion result are used as the output of the SwinTransformer block.

[0021] In some embodiments, the reconstruction loss function includes a pixel loss function and an adversarial loss function, wherein the pixel loss function adopts a mean square error loss function.

[0022] In some embodiments, the network structure of the discriminator includes eight 3×3 convolutional layers, an LReLU activation function layer and a batch normalization layer; the numbers of convolution kernels of the eight 3×3 convolutional layers are 64, 128, 128, 128, 256, 256, 512, and 512 respectively; the last convolutional layer with 512 convolution kernels is connected to a fully connected layer, the output of the fully connected layer is connected to the LReLU activation function layer, and the output of the LReLU activation function layer is further connected to two fully connected layers to construct a binary classification network, which is used to determine whether the image of the input face area to be confirmed is a high-resolution image and whether the image of the input face area to be confirmed is a face area.

[0023] In a second aspect of the present application, a face super-resolution reconstruction detection device is provided, the device comprising:

[0024] A face region detection module is used to detect whether there is a face region to be confirmed in the low-resolution image;

[0025] A face area reconstruction module is used to input the image of the face area to be confirmed into the face super-resolution reconstruction network for super-resolution reconstruction to confirm the face area in the image when a face area to be confirmed is detected in a low-resolution image; wherein an enhanced generative adversarial network constructed by a generator and a discriminator is used as the face super-resolution reconstruction network, and the DB residual dense block of the generator is replaced by multiple stacked Swin Transformer block structures, and the reconstruction loss function of the discriminator is introduced into the cross entropy loss function for determining whether it is a face area.

[0026] In some embodiments, the face region detection module includes:

[0027] The detection module is used to call the face detection network to detect the low-resolution image and obtain the face detection area;

[0028] The area confirmation module is used to divide the face detection area according to the confidence threshold. If it is greater than the first confidence threshold, the face detection area is a confirmed face area. If it is less than the first confidence threshold and greater than the second confidence threshold, the face detection area is a face area to be confirmed. If it is less than the second confidence threshold, the face detection area is a non-face area.

[0029] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a face super-resolution reconstruction detection method as provided in the first aspect of the present application is implemented.

[0030] Compared with the prior art, this application has the following beneficial effects:

[0031] In a face super-resolution reconstruction detection method provided in the present application, the generator of ESRGAN is deeply integrated with SwinTransformer to construct a face super-resolution reconstruction network. Specifically, the hierarchical structure and local attention mechanism of Swin Transformer enable the model to extract features at different scales and perform multi-scale feature fusion through cross-layer connections. It helps to reduce the generation of these artifacts, thereby improving image quality. At the same time, Swin Transformer has a stronger feature extraction capability, so it can better restore the detail information in the face image, such as skin texture, hair, etc. This makes the super-resolution reconstructed face image more realistic and natural. Swin Transformer can capture the global contextual information in the image through the self-attention mechanism, such as the structural relationship between the details in the face image (such as eyes, nose, mouth, etc.) and the entire face. The introduction of global information helps to better restore these details, thereby improving the feature extraction capability and reconstruction quality of the face super-resolution reconstruction network, and ensuring the image quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the embodiments of the present application, constitute a part of the present application, and do not constitute a limitation on the embodiments of the present application. In the drawings:

[0033] Figure 1 A flowchart of a face super-resolution reconstruction detection method provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the structure of a generator of a face super-resolution reconstruction network provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of the Swin Transformer block structure provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the structure of the Swin Transformer block provided in the embodiment of the present application;

[0037] Figure 5 A schematic diagram of the structure of the discriminator of the face super-resolution reconstruction network provided in an embodiment of the present application;

[0038] Figure 6 A principle block diagram of a face super-resolution reconstruction detection device provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with examples and drawings. The illustrative implementation scheme of the present application and its description are only used to explain the present application and are not intended to limit the present application.

[0041] It should be noted that the terms "include" or "may include" used in various embodiments of the present application indicate the presence of the function, operation or element applied for, and do not limit the addition of one or more functions, operations or elements. In addition, as used in various embodiments of the present application, the terms "include", "have" and their cognates are only intended to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or the possibility of adding one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items.

[0042] It should be understood that terms such as "first", "second", "third", "fourth", and "fifth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0043] First, the embodiment of the present application first describes the existing enhanced generative adversarial network ESRGAN, which uses Residual-in-Residual Dense Block (RRDB) as the main building block of the generator. This structure integrates the ideas of multi-level residual networks and dense connections, which helps to extract richer image features. RRDB (Residual-in-Residual Dense Block) is a network structure proposed in ESRGAN (Enhanced Super-ResolutionGenerative Adversarial Networks), which combines the advantages of residual networks (Residual Network) and densely connected blocks (Dense Block) to improve the performance of image super-resolution tasks. The RRDB network structure is mainly composed of a plurality of RRDB modules cascaded, and each RRDB module contains several Residual Dense Blocks (RDBs) and a residual connection (Residual Connection). Through the residual link, RRDB implements a "residual and residual" structure, so that residual information can be effectively utilized at different levels. RDB is the basic building block of RRDB, which combines Residual Block with Dense Block and extracts rich local features through densely connected convolutional layers.

[0044] Swin Transformer is a model based on the Transformer architecture, specially designed for computer vision tasks such as image classification, object detection, and semantic segmentation. It effectively solves the problem of high computational complexity of traditional Transformer when processing images by introducing the Window Attention Mechanism and hierarchical feature learning strategy. It can not only process multi-scale image information like CNN, but also retain the advantages of Transformer in global information modeling and feature representation. At the same time, the introduction of cross-level connections and cross-window attention mechanisms further enhances the model's global modeling ability and feature integration ability, enabling Swin Transformer to achieve excellent performance in multiple computer vision tasks.

[0045] However, the ESRGAN described above still has certain noise and artifact problems in some cases, which affects the quality of the generated image (such as blur, jagged, etc.).

[0046] Therefore, in order to solve the problem that the ESRGAN network provided by the prior art may be difficult to accurately reconstruct all high-frequency information for images including complex textures, details or dynamic scenes in the process of reconstructing image resolution, especially in areas such as edges and texture details, artifacts or blurring may occur, which affects the quality of the generated image (such as blurring, jaggedness, etc.). The embodiment of the present application provides a face super-resolution reconstruction detection method, which deeply integrates the generator of ESRGAN with SwinTransformer to construct a face super-resolution reconstruction network. Specifically, the hierarchical structure and local attention mechanism of Swin Transformer enable the model to extract features at different scales and perform multi-scale feature fusion through cross-layer connections. It helps to reduce the generation of these artifacts, thereby improving image quality. At the same time, Swin Transformer has a stronger feature extraction capability, so it can better restore detail information in face images, such as skin texture, hair, etc. This makes the super-resolution reconstructed face image more realistic and natural. Swin Transformer can capture global contextual information in the image through the self-attention mechanism, such as the structural relationship between the details in the face image (such as eyes, nose, mouth, etc.) and the entire face. The introduction of global information helps to better restore these details, thereby improving the feature extraction ability and reconstruction quality of the face super-resolution reconstruction network, and ensuring the image quality of the generated image.

[0047] The following will describe in detail the face super-resolution reconstruction detection method provided by the embodiment of the present application in combination with a specific implementation scheme. Figure 1 , Figure 1 A flowchart of a face super-resolution reconstruction detection method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0048] S101, detecting whether there is a face area to be confirmed in the low-resolution image.

[0049] In this embodiment, the following method is used to detect whether there is a face area to be confirmed in a low-resolution image: a face detection network is called to detect the low-resolution image to obtain a face detection area; the face detection area is divided according to a confidence threshold, if it is greater than a first confidence threshold, the face detection area is a confirmed face area, if it is less than the first confidence threshold and greater than the second confidence threshold, the face detection area is a face area to be confirmed, and if it is less than the second confidence threshold, the face detection area is a non-face area.

[0050] Specifically, the face detection network used in this embodiment is RetinaFace.

[0051] S102, when it is detected that there is a face area to be confirmed in the low-resolution image, the image of the face area to be confirmed is input into the face super-resolution reconstruction network for super-resolution reconstruction to confirm the face area in the image; wherein, the enhanced generative adversarial network constructed by the generator and the discriminator is used as the face super-resolution reconstruction network, and the DB residual dense block of the generator is replaced by a plurality of stacked Swin Transformer block structures, and the reconstruction loss function of the discriminator is introduced into the cross entropy loss function for determining whether it is a face area.

[0052] In this embodiment, please refer to Figure 2 , LR represents the image of the face area to be confirmed, that is, a low-resolution image with lower resolution, and HR represents the face area determined from the image of the face area to be confirmed, that is, a high-resolution image with higher resolution.

[0053] The generator includes a shallow feature extraction network and a deep feature extraction network; the shallow feature extraction network is a 3×3 convolutional layer, which is used to map the image of the face area to be confirmed into a first feature map of size C×W×H; where C is the number of channels, W and H are the width and height of the image respectively.

[0054] The deep feature extraction network includes multiple stacked Swin Transformer block structures, multiple 3×3 convolutional layers, upsampling layers, and at least one activation function layer.

[0055] It should be noted that the Swin Transformer block structure of the deep feature extraction network in this embodiment is similar to Figure 2 The dense Swin Transformer block in is equivalent to the 3×3 convolutional layer. Figure 2 The 3×3Conv in . The upsampling layer is Figure 2 The upsampling in the activation function layer is Figure 2 The LreLU in .

[0056] Further, such as Figure 3 As shown, each Swin Transformer block structure includes a first Swin Transformer sub-block structure, a first convolutional layer, a second Swin Transformer sub-block structure and a second convolutional layer; wherein the first Swin Transformer sub-block structure and the second Swin Transformer sub-block structure are both composed of three Swin Transformer blocks connected in sequence;

[0057] The first Swin Transformer sub-block structure captures the local features and contextual information of the first feature map through the self-attention mechanism and relative position encoding of three Swin Transformer blocks to obtain a second feature map of size 2C×W×H;

[0058] The first convolutional layer is used to fuse the first feature map with the second feature map to obtain a third feature map of size 2C×W×H;

[0059] The second Swin Transformer sub-block structure captures the local features and context information of the third feature map through the self-attention mechanism and relative position encoding of three Swin Transformer blocks to obtain a fourth feature map of size 3C×W×H;

[0060] The second convolutional layer is used to normalize the first feature map, the third feature map, and the fourth feature map to a fifth feature map of size C×W×H.

[0061] exist Figure 3 In the dense Swin Transformer block structure, six Swin Transformer blocks ST and two convolutional layers Conv are combined. In this structure, every three consecutive ST blocks are regarded as a group, which are connected to the subsequent convolutional layers in a densely connected manner, greatly enhancing the flow and reuse of information. In the dense Swin Transformer block structure, there are two levels of dense connections: first, a group of ST blocks captures rich local and contextual information through self-attention mechanism and relative position encoding; second, these enhanced feature representations are injected into the subsequent convolutional layers in a densely connected manner to achieve further fusion and enhancement of features. Assume that the size of the input feature map is C×W×H, where C is the channel, W and H are the width and height respectively. The size of the feature maps output by the two groups of ST are 2C×W×H and 3C×W×H respectively. Finally, these feature maps are normalized to C×W×H through the convolutional layer, maintaining the richness of features and the consistency of data.

[0062] Given that facial images contain rich texture details, such as fine hair, complex skin wrinkles, and scattered spots, these features often present repeated or similar patterns in the global range of the image. This repeated information is crucial for building a robust face reconstruction model. Therefore, in the deep feature extraction network stage, by stacking multiple dense Swin Transformer block structure modules, the model can deeply mine and learn these global similarity information layer by layer, thereby more effectively capturing the key features in the facial image and laying a solid foundation for subsequent recognition, analysis, or reconstruction tasks.

[0063] In some embodiments, the Swin Transformer block is composed of two layer normalization layers, a multi-head self-attention mechanism module and a multi-layer perceptron module; wherein the first feature map is used as the input of a layer normalization layer, the output of a layer normalization layer is used as the input of the multi-head self-attention mechanism module, the output of the multi-head self-attention mechanism module and the first feature map are fused to obtain a first fusion result, which is used as the input of another layer normalization layer, the output of the other layer normalization layer is used as the input of the multi-layer perceptron module, and the output of the multi-layer perceptron module and the first fusion result are fused to obtain a second fusion result, which is used as the output of the Swin Transformer block.

[0064] Specifically, if Figure 4 As shown in the figure, a ST (Swin Transformer) block is composed of LayerNorm layer, MSA (Multi-Head Self-Attention Mechanism Module) and MLP (Multi-Layer Perceptron Module). For example, the feature map of size C×W×H is first subjected to Layer Normalization (LN) to stabilize the training process and accelerate convergence. The input feature map of size C×W×H is divided into M×M local windows that do not overlap, and the total number of windows is HW / M 2 ), calculate the local attention of each window respectively. For the local window features, the calculation formulas of query matrix, key matrix, value matrix Q, K, V are as follows.

[0065] Q = XP Q ,K=XP K ,V=XP K ; where P Q , P K and P K is the projection matrix shared among different windows.

[0066] The matrix is ​​calculated by the self-attention mechanism within the local window, and its calculation formula is as follows.

[0067] Attention(Q,K,V)=SoftMax((QK^T) / √d+B)V;where B is a learnable relative position code. The attention function is executed h times in parallel, and then the results of the multi-head self-attention mechanism module MSA are connected to recombine into an image of the same size as the input feature map, which is output after a layer of convolution.

[0068] The MLP module itself consists of two fully connected layers, between which a GELU (Gaussian Error Linear Unit) is embedded as a nonlinear activation function. The GELU activation function enhances the model's ability to fit complex feature patterns by introducing nonlinear factors, further improving the effect of feature conversion.

[0069] X=MSA(LN(X))+X; X=MLP(LN(X))+X; where MSA is the multi-head self-attention module, LN is the LayerNorm layer, and MLP is the multi-layer perceptron.

[0070] Therefore, in this embodiment, the generator provided by introducing the ST block and fusing the residual connection mechanism can significantly improve the model's ability to learn local and global features in image reconstruction tasks, thereby generating high-resolution images with richer details. This mechanism can build long-distance dependencies between various positions in the feature map, so that the model can more accurately capture the subtle textures and complex structures in the image. This feature helps to reduce image blur caused by information loss. At the same time, the application of residual connections further promotes the effective fusion and information transfer of internal features of the model. By directly connecting the input and output, the residual connection not only alleviates the problem of vanishing gradients in the deep network, but also enables shallow features to directly participate in the construction of deep features, promoting the use of multi-level feature information. Each time after dense connection, the convolutional layer will fuse the information of the previous layer and enhance the ST block using the spatial invariance of the convolution. In addition, another function of the convolutional layer is to normalize the channel so that it can be input into the next group of ST blocks.

[0071] In the upsampling stage, pixel-squeezing convolution is used to efficiently improve the spatial resolution of the feature map. The pixels in the channel dimension of the feature map are rearranged into a larger spatial dimension, while the number of channels is reduced to achieve spatial scale expansion.

[0072] This embodiment also provides improvements on the discriminator, such as Figure 5 As shown, the network structure of the discriminator includes eight 3×3 convolutional layers, an LReLU activation function layer and a batch normalization layer; the numbers of convolution kernels of the eight 3×3 convolutional layers are 64, 128, 128, 128, 256, 256, 512, and 512 respectively; the last convolutional layer with 512 convolution kernels is connected to a fully connected layer, the output of the fully connected layer is connected to the LReLU activation function layer, and the output of the LReLU activation function layer is further connected to two fully connected layers to construct a binary classification network, which is used to determine whether the image of the input face area to be confirmed is a high-resolution image and whether the image of the input face area to be confirmed is a face area.

[0073] Specifically, the adjustment of the discriminator adopts the Leaky ReLU activation function combined with the batch normalization (BN) strategy to enhance the nonlinear expression ability of the network and accelerate the training process, while completely abandoning the pooling operation to reduce information loss. The designed discriminator network contains a total of eight convolutional structures, each layer uses a 3×3 convolution kernel. The number of convolution kernels follows an exponential growth strategy, gradually increasing from the initial 64 to 512. The network controls feature extraction and spatial dimension reduction by alternating between layers with convolution steps of 1 and 2. Whenever the feature dimension doubles, a convolution layer with a step size of 2 is used to halve the spatial resolution of the image, thereby effectively aggregating contextual information. On top of the high-level features extracted by the deep convolutional layer, a fully connected layer (FC) is connected for global feature integration and compression. The fully connected layers set in parallel handle two binary classification tasks respectively: one is to distinguish whether the input image is a high-resolution (HR) image, and the other is to determine whether it is a face area.

[0074] Furthermore, given that the model needs to perform the dual tasks of image reconstruction and classification at the same time, its loss function is designed to include reconstruction loss and classification loss. The reconstruction loss is intended to encourage the generator network to accurately reconstruct high-resolution and detailed images from low-resolution inputs. The reconstruction loss function combines pixel-level loss and adversarial loss. The pixel-to-pixel loss function uses the mean square error (MSE) loss, which is defined as follows:

[0075] In the formula, represents a low-resolution image, represents a real high-resolution image sample, G is the generator network, and w is the parameter of the generator.

[0076] The adversarial loss function is defined as follows: In the formula, It represents the probability that the discriminator D regards the sample image generated by the generator G as the real sample image, where θ is the parameter of the discriminant network.

[0077] Since this embodiment also needs to determine whether it is a classification task of a face region, a cross entropy loss function is used to evaluate the performance of face region classification confirmation. Considering that during the training process, the discriminant network may receive input images containing low-resolution faces, in order to encourage the generator to generate more detailed high-resolution face images, an auxiliary loss is introduced to further promote the generator to generate more detailed high-resolution images, namely the cross entropy loss function, which is defined as follows:

[0078] In the formula, y i is the label corresponding to the data sample, y for human face i =1, non-face is y i=0. Combining the above loss formulas, we get the reconstruction loss function L, which is expressed as: L = L MSE +αL adv +βL cls , where α and β are weight coefficients.

[0079] The present application also provides a face super-resolution reconstruction detection device, which can be used to perform each step of a face super-resolution reconstruction detection method described in the above embodiment. Figure 6 , Figure 6 A principle block diagram of a face super-resolution reconstruction detection device provided in an embodiment of the present application, such as Figure 6 As shown, the device includes

[0080] A face region detection module 610 is used to detect whether there is a face region to be confirmed in the low-resolution image;

[0081] The face area reconstruction module 620 is used to input the image of the face area to be confirmed into the face super-resolution reconstruction network for super-resolution reconstruction to confirm the face area in the image when a face area to be confirmed is detected in the low-resolution image; wherein the enhanced generative adversarial network constructed by the generator and the discriminator is used as the face super-resolution reconstruction network, and the DB residual dense block of the generator is replaced by multiple stacked Swin Transformer block structures, and the reconstruction loss function of the discriminator is introduced into the cross entropy loss function for determining whether it is a face area.

[0082] Accordingly, in a face super-resolution reconstruction detection device provided in the present application, the generator of ESRGAN is deeply integrated with Swin Transformer to construct a face super-resolution reconstruction network. Specifically, the hierarchical structure and local attention mechanism of Swin Transformer enable the model to extract features at different scales and perform multi-scale feature fusion through cross-layer connections. It helps to reduce the generation of these artifacts, thereby improving image quality. At the same time, Swin Transformer has a stronger feature extraction capability, so it can better restore the detail information in the face image, such as skin texture, hair, etc. This makes the super-resolution reconstructed face image more realistic and natural. Swin Transformer can capture the global contextual information in the image through the self-attention mechanism, such as the structural relationship between the details in the face image (such as eyes, nose, mouth, etc.) and the entire face. The introduction of global information helps to better restore these details, thereby improving the feature extraction capability and reconstruction quality of the face super-resolution reconstruction network, and ensuring the image quality of the generated image.

[0083] In some embodiments, the face area detection module includes: a detection module, which is used to call the face detection network to detect the low-resolution image and obtain the face detection area; an area confirmation module, which is used to divide the face detection area according to the confidence threshold. If it is greater than the first confidence threshold, the face detection area is a confirmed face area; if it is less than the first confidence threshold and greater than the second confidence threshold, the face detection area is a face area to be confirmed; if it is less than the second confidence threshold, the face detection area is a non-face area.

[0084] The embodiment of the present application provides a face super-resolution reconstruction detection device, which is similar to the above Figure 1 The face super-resolution reconstruction detection method shown is an invention based on the same concept. Through the detailed description of the face super-resolution reconstruction detection method provided in the above embodiment, technical personnel in this field can clearly understand the implementation process of a face super-resolution reconstruction detection device in this embodiment. Therefore, for the sake of conciseness of the specification, it will not be repeated here.

[0085] Please refer to Figure 7 In an embodiment of the present application, a structural diagram of an electronic device is also provided, such as Figure 7 As shown, the electronic device may include: a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702 and the memory 303 communicate with each other through the communication bus 704. The processor 701 may call the logic instructions in the memory 703 to execute a face super-resolution reconstruction detection method, the method comprising: detecting whether there is a face area to be confirmed in a low-resolution image; when it is detected that there is a face area to be confirmed in the low-resolution image, inputting the image of the face area to be confirmed into the face super-resolution reconstruction network for super-resolution reconstruction to confirm the face area in the image; wherein, the enhanced generative adversarial network constructed by the generator and the discriminator is used as the face super-resolution reconstruction network, and the DB residual dense block of the generator is replaced by a plurality of stacked Swin Transformer block structures, and the reconstruction loss function of the discriminator is introduced into the cross entropy loss function for determining whether it is a face area.

[0086] In addition, the logic instructions in the above-mentioned memory 703 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] In an embodiment of the present application, a computer program product is also provided, which includes a computer program, and the computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a face super-resolution reconstruction detection method provided by the above methods, the method including: detecting whether there is a face area to be confirmed in a low-resolution image; when it is detected that there is a face area to be confirmed in the low-resolution image, inputting the image of the face area to be confirmed into a face super-resolution reconstruction network for super-resolution reconstruction to confirm the face area in the image; wherein, an enhanced generative adversarial network constructed by a generator and a discriminator is used as the face super-resolution reconstruction network, and the DB residual dense block of the generator is replaced by multiple stacked Swin Transformer block structures, and the reconstruction loss function of the discriminator is introduced into a cross-entropy loss function for determining whether it is a face area.

[0088] In an embodiment of the present application, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute a face super-resolution reconstruction detection method provided by the above methods. The computer-readable storage medium is a memory device in a computer device, which is used to store programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in a computer device and extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (nonvolatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the face super-resolution reconstruction detection method in the above embodiments. Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD ROM, optical storage, etc.) containing computer-usable program codes.

[0089] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A face super-resolution reconstruction detection method, characterized in that: Methods include: Detect whether there is a face area to be confirmed in the low-resolution image; When it is detected that there is a face area to be confirmed in a low-resolution image, the image of the face area to be confirmed is input into a face super-resolution reconstruction network for super-resolution reconstruction to confirm the face area in the image; wherein, an enhanced generative adversarial network constructed by a generator and a discriminator is used as the face super-resolution reconstruction network, and the DB residual dense block of the generator is replaced by multiple stacked Swin Transformer block structures, and the reconstruction loss function of the discriminator is introduced into a cross entropy loss function for determining whether it is a face area.

2. The method according to claim 1, characterized in that: The low-resolution image is detected to have a face area to be confirmed in the following manner: a face detection network is called to detect the low-resolution image to obtain a face detection area; The face detection area is divided according to the confidence threshold. If it is greater than the first confidence threshold, the face detection area is a confirmed face area. If it is less than the first confidence threshold and greater than the second confidence threshold, the face detection area is a face area to be confirmed. If it is less than the second confidence threshold, the face detection area is a non-face area.

3. The method according to claim 1, characterized in that The generator includes a shallow feature extraction network and a deep feature extraction network; The shallow feature extraction network is a 3×3 convolutional layer, which is used to map the image of the face area to be confirmed into a first feature map of size C×W×H; where C is the number of channels, W and H are the width and height of the image respectively. The deep feature extraction network includes multiple stacked Swin Transformer block structures, multiple 3×3 convolutional layers, an upsampling layer and at least one activation function layer.

4. The method according to claim 3, characterized in that Each of the Swin Transformer block structures includes a first Swin Transformer sub-block structure, a first convolutional layer, a second Swin Transformer sub-block structure, and a second convolutional layer; wherein the first Swin Transformer sub-block structure and the second Swin Transformer sub-block structure are both composed of three Swin Transformer blocks connected in sequence; The first Swin Transformer sub-block structure captures the local features and context information of the first feature map through the self-attention mechanism and relative position encoding of three Swin Transformer blocks to obtain a second feature map of size 2C×W×H; The first convolutional layer is used to fuse the first feature map with the second feature map to obtain a third feature map of size 2C×W×H; The second Swin Transformer sub-block structure captures the local features and context information of the third feature map through the self-attention mechanism and relative position encoding of three Swin Transformer blocks to obtain a fourth feature map of size 3C×W×H; The second convolutional layer is used to normalize the first feature map, the third feature map, and the fourth feature map to a fifth feature map of size C×W×H.

5. The method according to claim 4, characterized in that The Swin Transformer block is composed of two layer normalization layers, a multi-head self-attention mechanism module and a multi-layer perceptron module; Among them, the first feature map is used as the input of a layer normalization layer, the output of a layer normalization layer is used as the input of the multi-head self-attention mechanism module, the output of the multi-head self-attention mechanism module and the first feature map are fused to obtain a first fusion result as the input of another layer normalization layer, the output of another layer normalization layer is used as the input of the multi-layer perceptron module, and the output of the multi-layer perceptron module and the second fusion result obtained by fusion of the first fusion result are used as the output of the SwinTransformer block.

6. The method according to claim 1, characterized in that The reconstruction loss function includes a pixel loss function and an adversarial loss function, wherein the pixel loss function adopts a mean square error loss function.

7. The method according to claim 1, characterized in that The network structure of the discriminator includes eight 3×3 convolutional layers, an LReLU activation function layer and a batch normalization layer; the numbers of convolution kernels of the eight 3×3 convolutional layers are 64, 128, 128, 128, 256, 256, 512, and 512 respectively; the last convolutional layer with 512 convolution kernels is connected to a fully connected layer, the output of the fully connected layer is connected to the LReLU activation function layer, and the output of the LReLU activation function layer is further connected to two fully connected layers to construct a binary classification network, which is used to judge whether the image of the input face area to be confirmed is a high-resolution image and whether the image of the input face area to be confirmed is a face area.

8. A face super-resolution reconstruction detection device, characterized in that: The device includes: A face region detection module is used to detect whether there is a face region to be confirmed in the low-resolution image; A face area reconstruction module is used to input the image of the face area to be confirmed into the face super-resolution reconstruction network for super-resolution reconstruction to confirm the face area in the image when a face area to be confirmed is detected in a low-resolution image; wherein an enhanced generative adversarial network constructed by a generator and a discriminator is used as the face super-resolution reconstruction network, and the DB residual dense block of the generator is replaced by multiple stacked Swin Transformer block structures, and the reconstruction loss function of the discriminator is introduced into the cross entropy loss function for determining whether it is a face area.

9. The device according to claim 8, characterized in that Face area detection module, including: The detection module is used to call the face detection network to detect the low-resolution image and obtain the face detection area; The area confirmation module is used to divide the face detection area according to the confidence threshold. If it is greater than the first confidence threshold, the face detection area is a confirmed face area. If it is less than the first confidence threshold and greater than the second confidence threshold, the face detection area is a face area to be confirmed. If it is less than the second confidence threshold, the face detection area is a non-face area.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for super-resolution reconstruction and detection of faces as described in any one of claims 1 to 7 is implemented.

Images