Video super-resolution recognition method and system based on face feature coding

Abstract

The invention provides a video super-resolution recognition method based on face feature coding, and the method comprises the following steps: obtaining a low-resolution video stream, and carrying out the preprocessing of a video frame; extracting a face region in the preprocessed video frame, and constructing a face region sequence; inputting the face region sequence into a pre-trained feature coding network, and extracting a space-time associated face depth feature vector; based on the face depth feature vector, generating a high-resolution face image with reserved identity features by using a generative adversarial network; performing multi-scale feature fusion on the generated high-resolution face image and the original feature vector, and optimizing feature expression; and performing identity recognition based on the optimized feature expression, and outputting a recognition result. According to the invention, the face detection and recognition performance can be greatly improved, and powerful support is provided for public safety guarantee.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a video super-resolution recognition method and system based on facial feature coding. Background Art

[0002] As an important branch of biometric recognition, facial recognition technology has broad application prospects in public safety, intelligent security, financial payments, and other fields. However, video images captured in real-world applications are often affected by factors such as lighting variations, motion blur, and low resolution, making it difficult to accurately extract facial features and severely impacting recognition accuracy.

[0003] Traditional super-resolution reconstruction methods primarily focus on enhancing image detail, but lack the ability to effectively preserve facial identity. This can lead to improved visual quality in the resulting high-resolution facial images, but altered identity characteristics, impacting subsequent recognition performance. Furthermore, existing face recognition algorithms experience a significant decline in recognition accuracy at low resolutions due to a significant loss of feature information.

[0004] In recent years, deep learning technology has made significant progress in the field of computer vision. In particular, generative adversarial networks (GANs) have demonstrated powerful capabilities in image super-resolution reconstruction. However, when general super-resolution methods are directly applied to facial images, they often fail to effectively preserve the identity of the face, resulting in poor performance of the reconstructed face images in recognition systems.

[0005] Therefore, it is of great practical significance to develop a video super-resolution recognition method that can effectively maintain the facial identity features while improving the quality of low-resolution face images. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a video super-resolution recognition method and system based on facial feature coding. By combining facial feature coding with super-resolution reconstruction, the consistency of facial identity features is maintained while improving image resolution, thereby significantly improving the recognition accuracy of faces in low-resolution videos.

[0007] The technical solutions of the present invention are as follows:

[0008] A video super-resolution recognition method based on facial feature coding comprises the following steps:

[0009] Obtain low-resolution video stream and pre-process the video frames;

[0010] Extract the face region from the preprocessed video frame and construct a face region sequence;

[0011] Input the facial region sequence into the pre-trained feature encoding network to extract the spatiotemporal associated facial depth feature vector;

[0012] Based on the facial deep feature vector, using a generative adversarial network to generate a high-resolution facial image with preserved identity features;

[0013] Perform multi-scale feature fusion on the generated high-resolution face image and the original feature vector to optimize feature expression;

[0014] Perform identity recognition based on the optimized feature expression and output the recognition result.

[0015] Furthermore, the pre-processing of the video frame is specifically as follows:

[0016] Denoise the video frames;

[0017] Perform contrast enhancement on the denoised video frames to improve the visual features of the face area;

[0018] The enhanced video frames are subjected to color space conversion, the RGB color space is converted into the YCbCr color space, and the video frames subjected to color space conversion are subjected to sharpening processing.

[0019] Furthermore, the face region in the preprocessed video frame is extracted and the face region sequence is constructed as follows:

[0020] Detecting face regions in pre-processed video frames;

[0021] Align the detected face area and adjust the face to a standard posture based on the facial key point positioning algorithm;

[0022] The face regions in the video frames are sampled in time sequence to construct a face region sequence containing continuous time information.

[0023] Furthermore, the feature encoding network includes a 3D convolutional layer and an attention mechanism, wherein the 3D convolutional layer includes three convolutional modules, each convolutional module contains a 3×3×3 convolution kernel, a batch normalization layer and a ReLU activation function; the attention mechanism includes a spatial attention module and a channel attention module.

[0024] Furthermore, based on the facial depth feature vector, generating a high-resolution facial image with identity-preserving features using a generative adversarial network is specifically as follows:

[0025] Input the facial depth feature vector into the feature condition generator to generate a high-resolution facial structure;

[0026] The generated high-resolution facial structure is input into the detail enhancement network to restore facial texture details;

[0027] An identity-preserving discriminator is used to determine whether the generated high-resolution face image retains the original identity features. The identity-preserving discriminator includes a feature extraction branch and an identity verification branch.

[0028] Furthermore, the multi-scale feature fusion of the generated high-resolution face image and the original feature vector is specifically as follows:

[0029] The generated high-resolution face image is input into the feature pyramid network to extract feature maps of different scales;

[0030] Project the original feature vector into the feature space of the same dimension as the feature map through the fully connected layer;

[0031] Adopting adaptive attention mechanism to calculate the correlation weight between the original feature vector and the feature map of each scale;

[0032] Based on the calculated correlation weights, the feature maps of each scale are weightedly fused to obtain the optimized feature expression.

[0033] The present invention also provides a video super-resolution recognition system based on facial feature coding, comprising:

[0034] A video preprocessing unit configured to perform enhancement and face region extraction on an input low-resolution video;

[0035] a feature encoding unit configured to extract spatiotemporal and depth features of a face in a video sequence;

[0036] a super-resolution generation unit configured to generate a high-resolution face image based on the depth feature;

[0037] a feature fusion unit configured to fuse the generated image with the original features to optimize feature expression;

[0038] The recognition taxon is configured to perform identification based on optimized features.

[0039] Furthermore, the super-resolution generation unit includes:

[0040] Feature condition generator, which generates high-resolution face structure based on face feature vectors;

[0041] Detail enhancement network, used to restore facial texture details;

[0042] The identity-preserving discriminator ensures that the generated faces retain the original identity characteristics.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] The present invention organically combines facial feature encoding with super-resolution reconstruction, effectively maintaining the identity characteristics of the face while improving image resolution, and solves the problem of identity feature loss in face applications using traditional super-resolution methods.

[0045] By designing a generative adversarial network that includes an identity-preserving discriminator and introducing an identity-preserving loss function, we ensure that the generated high-resolution face images retain the original identity feature information while improving the visual quality.

[0046] It can make full use of feature information of different scales and dynamically adjust feature weights through an adaptive attention mechanism, thereby improving the discriminative ability and robustness of features. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings illustrate various embodiments generally by way of example and not limitation, and together with the description and claims, serve to explain embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive of the embodiments of the present apparatus or method.

[0048] Figure 1 A schematic flow chart of the method of the present invention is shown. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] like Figure 1 As shown, the present invention provides a video super-resolution recognition method based on face feature coding, comprising

[0051] 1. Video Preprocessing

[0052] After obtaining the low-resolution video stream, perform the following operations on the video frames in sequence:

[0053] Denoising: The non-local mean filtering algorithm is used. This algorithm searches for an area similar to the current pixel block in the entire image and calculates the weighted average of the similar area to replace the current pixel block. This removes noise while preserving the image's texture and edge details as much as possible, minimizing damage to facial feature information.

[0054] Contrast Enhancement: Using the CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm, the video frame is first divided into multiple small regions, and histogram equalization is performed on each small region to enhance local contrast. At the same time, the amplitude of contrast enhancement is limited to avoid noise amplification and artifacts caused by excessive enhancement, effectively improving the visual recognition of facial areas in the video frame.

[0055] Color space conversion: Converts video frames from the RGB color space to the YCbCr color space. The Y component represents luminance information, while the Cb and Cr components represent chrominance information. This conversion separates luminance and chrominance information, allowing subsequent processing to prioritize the luminance component, enhancing image clarity while reducing the complexity of color information processing.

[0056] Sharpening: Uses the Laplacian operator to sharpen the video frames after color space conversion. By calculating the second-order derivative of image pixels, it highlights edges and details in the image, enhancing facial contours and features, and making blurred facial images clearer.

[0057] 2. Face Region Extraction and Sequence Construction

[0058] Face Region Detection: MTCNN (Multi-Task Cascaded Convolutional Neural Network) performs face detection on preprocessed video frames. This network consists of three cascaded sub-networks. Through three stages of rapid screening, precise calibration, and key point detection, it efficiently and accurately locates the face region in the video frame and outputs the position of the face frame and the corresponding confidence score.

[0059] Face alignment: Based on the facial key points detected by MTCNN (such as the eyes, nose tip, and mouth corners), an affine transformation algorithm is used to adjust the face to a standard pose. By calculating the correspondence between standard facial key points and the detected key points, an affine transformation matrix is ​​generated. The facial image is then rotated, scaled, and translated to ensure consistent spatial position and orientation across different video frames, facilitating accurate and consistent feature extraction.

[0060] Constructing a facial region sequence: Following the time sequence of the video frames, the facial region is sampled at regular intervals (e.g., every 5 frames). The sampled facial images are arranged in chronological order to construct a facial region sequence containing continuous temporal information, providing the data foundation for subsequent extraction of the spatiotemporal features of the face.

[0061] 3. Feature encoding step

[0062] The constructed face region sequence is input into the pre-trained feature encoding network, and the network structure is as follows:

[0063] 3D convolution layer: Consists of three convolution modules connected in sequence. Each convolution module contains a 3×3×3 convolution kernel, a batch normalization layer, and a ReLU activation function. The 3D convolution kernel performs convolution operations on a sequence of facial regions in both spatial dimensions (height and width) and temporal dimensions, simultaneously extracting local spatial features (such as facial features) and dynamic temporal features (such as facial expressions). The batch normalization layer normalizes the convolutional features to accelerate network training convergence. The ReLU activation function introduces nonlinear factors into the network, enhancing its expressive power.

[0064] Attention mechanism: It includes a spatial attention module and a channel attention module. The spatial attention module generates spatial attention weights by analyzing the information distribution of the feature map at different spatial positions, weighting different spatial positions of the feature map to highlight the features of key facial parts (such as eyes, nose, and mouth). The channel attention module calculates channel attention weights based on the response of each channel of the feature map, enhancing feature channels that are important for identity recognition and suppressing irrelevant channels, thereby further optimizing the extracted facial feature vectors and making them more discriminative and robust.

[0065] 4. Super-resolution reconstruction

[0066] Based on the extracted facial deep feature vectors, a generative adversarial network (GAN) is used to generate high-resolution face images. The network consists of the following parts working together:

[0067] The feature conditional generator uses a U-Net architecture and takes the facial depth feature vector as the conditional input. The encoder gradually extracts semantic features from low-resolution facial images through multiple convolutional and downsampling layers, reducing the resolution of the feature map and increasing the number of channels. The decoder maps the semantic features back to the high-resolution image space through transposed convolutions and upsampling layers. Skip connections are used to introduce feature information from the corresponding level in the encoder into the decoder, supplementing the detailed information and generating a preliminary high-resolution facial structure.

[0068] Detail Enhancement Network: Built on a residual network structure, it contains multiple residual blocks. The high-resolution facial structure output by the feature condition generator is fed into this network. The residual blocks learn the residual information of the image to restore and enhance the texture details of the facial image (such as skin texture and hair detail), making the generated high-resolution facial image more realistic and detailed.

[0069] Identity-preserving discriminator: This uses the PatchGAN architecture and consists of a feature extraction branch and an identity verification branch. The feature extraction branch determines whether the generated high-resolution facial image is realistic and natural. It extracts features from the image using a convolutional neural network and outputs a probability that the image is real. The identity verification branch compares the features of the input image with the feature vectors of the original low-resolution image to verify that the identity features of the generated image are consistent with the original image. This ensures that the generated high-resolution facial image retains the original identity information while improving visual quality.

[0070] Network training: During training, a combined loss function consisting of adversarial loss, perceptual loss, identity-preserving loss, and pixel loss is used. The adversarial loss encourages the generator to produce more realistic images to deceive the discriminator. The perceptual loss, based on a pre-trained VGG network, calculates the difference in high-level semantic features between the generated and real images, ensuring semantic consistency. The identity-preserving loss ensures that the identity features of the generated images are consistent with those of the original images. The pixel loss constrains the similarity between the generated and real images at the pixel level. By continuously adjusting the parameters of the generator and discriminator, the generative adversarial network achieves a balance, generating high-quality, high-resolution facial images that preserve identity features.

[0071] 5. Feature Fusion Steps

[0072] The generated high-resolution face image is fused with the original feature vector using multi-scale features. The specific process is as follows:

[0073] Multi-scale feature extraction: A high-resolution face image is fed into a feature pyramid network, which extracts feature maps of different scales from the image through lateral connections and a top-down pathway. These feature maps capture both local detail features (small scale) and global semantic features (large scale), providing rich feature information for subsequent fusion.

[0074] Feature vector projection: The original extracted facial depth feature vector is transformed through the fully connected layer and projected into the feature space of the same dimension as the feature map of each scale, so that the two have a basis for fusion.

[0075] Adaptive Attention Calculation: An adaptive attention mechanism is used to calculate the correlation weights between the original feature vector and feature maps at each scale. By analyzing the similarity and complementarity between the feature vector and feature map in the feature space, the weights of feature maps at different scales are dynamically adjusted to highlight important features related to identity recognition.

[0076] Weighted feature fusion: Based on the calculated correlation weights, the feature maps at each scale are weighted and summed to obtain the fused feature representation. Simultaneously, the residual optimization module performs a residual connection and nonlinear transformation on the fused features and the original feature vectors to further optimize the feature representation and enhance the discriminative power of the features, making them more suitable for identity recognition.

[0077] VI. Identity Identification

[0078] The optimized feature representation is input into a pre-trained Softmax classifier for identity recognition. Based on a classification model learned on a large-scale face dataset, the classifier calculates the probability that the input feature belongs to each identity category and selects the category with the highest probability as the recognition result. To improve recognition accuracy, a voting mechanism is used to count the number of occurrences of each identity category across multiple consecutive frames (e.g., 10 frames) in the video sequence. The category with the highest number of occurrences is then output as the final identity recognition result.

[0079] The above is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, can make equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A video super-resolution recognition method based on facial feature coding, characterized in that: The following steps are involved: Obtain low-resolution video stream and pre-process the video frames; Extract the face region from the preprocessed video frame and construct a face region sequence; Input the facial region sequence into the pre-trained feature encoding network to extract the spatiotemporal associated facial depth feature vector; Based on the facial deep feature vector, using a generative adversarial network to generate a high-resolution facial image with preserved identity features; Perform multi-scale feature fusion on the generated high-resolution face image and the original feature vector to optimize feature expression; Perform identity recognition based on the optimized feature expression and output the recognition result.

2. The video super-resolution recognition method based on facial feature coding according to claim 1, characterized in that: The pre-processing of the video frame is specifically as follows: Denoise the video frames; Perform contrast enhancement on the denoised video frames to improve the visual features of the face area; The enhanced video frames are subjected to color space conversion, the RGB color space is converted into the YCbCr color space, and the video frames subjected to color space conversion are subjected to sharpening processing.

3. The video super-resolution recognition method based on facial feature coding according to claim 1, characterized in that: Extract the face area in the preprocessed video frame and construct the face area sequence as follows: Detecting face regions in pre-processed video frames; Align the detected face area and adjust the face to a standard posture based on the facial key point positioning algorithm; The face regions in the video frames are sampled in time sequence to construct a face region sequence containing continuous time information.

4. The video super-resolution recognition method based on facial feature coding according to claim 1, characterized in that: The feature encoding network includes a 3D convolutional layer and an attention mechanism, wherein the 3D convolutional layer includes three convolution modules, each convolution module contains a 3×3×3 convolution kernel, a batch normalization layer and a ReLU activation function; the attention mechanism includes a spatial attention module and a channel attention module.

5. The video super-resolution recognition method based on facial feature coding according to claim 1, characterized in that: Based on the facial depth feature vector, a high-resolution facial image with identity-preserving features is generated using a generative adversarial network as follows: Input the facial depth feature vector into the feature condition generator to generate a high-resolution facial structure; The generated high-resolution facial structure is input into the detail enhancement network to restore facial texture details; An identity-preserving discriminator is used to determine whether the generated high-resolution face image retains the original identity features. The identity-preserving discriminator includes a feature extraction branch and an identity verification branch.

6. The video super-resolution recognition method based on facial feature coding according to claim 1, characterized in that: The multi-scale feature fusion of the generated high-resolution face image and the original feature vector is specifically as follows: The generated high-resolution face image is input into the feature pyramid network to extract feature maps of different scales; Project the original feature vector into the feature space of the same dimension as the feature map through the fully connected layer; Adopting adaptive attention mechanism to calculate the correlation weight between the original feature vector and the feature map of each scale; Based on the calculated correlation weights, the feature maps of each scale are weightedly fused to obtain the optimized feature expression.

7. A video super-resolution recognition system based on facial feature coding, characterized in that: include: A video preprocessing unit configured to perform enhancement and face region extraction on an input low-resolution video; a feature encoding unit configured to extract spatiotemporal and depth features of a face in a video sequence; a super-resolution generation unit configured to generate a high-resolution face image based on the depth feature; a feature fusion unit configured to fuse the generated image with the original features to optimize feature expression; The recognition taxon is configured to perform identification based on optimized features.

8. The video super-resolution recognition system based on facial feature coding according to claim 7, characterized in that: The super-resolution generation unit includes: Feature condition generator, which generates high-resolution face structure based on face feature vectors; Detail enhancement network, used to restore facial texture details; The identity-preserving discriminator ensures that the generated faces retain the original identity characteristics.

Images