Multi-face high-speed recognition method and system based on video stream and medium

Through multi-task learning and multi-stage training, multi-task recovery recognition neural network model, combined with restoration and recognition tasks, the low recognition rate and identity inconsistency of face recognition in natural scenes is solved, and efficient and accurate face recognition is achieved .

CN119068528APending Publication Date: 2024-12-03ZHEJIANG TUXUN TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411136572.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing natural scene facial recognition algorithm has low recognition rate in complex environments, making it difficult to process images affected by a variety of low-quality factors. Separate training of face restoration and recognition models leads to identity inconsistency, affecting the accuracy of recognition.

Method used

Using multi-task learning method, multi-stage training of multi-task recovery recognition neural network model is combined with face recovery and recognition, and using restoration generation adversarial network and recognition network for individual and joint training, combining MSE, perception, identity and adversarial loss Function to realize efficient, accurate and universal face recognition for natural scene video streams.

Benefits of technology

It realizes efficient and accurate facial recognition in natural scenarios, retains identity information, improves recognition performance and robustness, and solves the low recognition rate and inconsistency of facial recognition in uncontrolled scenarios in the prior art. Sexual problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068528A_ABST
    Figure CN119068528A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-face high-speed recognition method and system based on a video stream and a medium. The method comprises the following steps: acquiring a test image based on a test video stream; inputting a test image into the trained multi-task restoration recognition neural network model to obtain model output; and performing feature extraction based on a target image output by the model to obtain identity features and confidence. According to the method, face restoration and face recognition are creatively combined together, a multi-task learning method is utilized, a plurality of loss functions are combined to carry out two-stage training on a network, specifically, the two-stage training comprises independent training and combined training, accurate, efficient and universal face recognition on natural scene video streams is achieved, identity information is reserved, and the face recognition efficiency is improved. The difficulty problem and the pain point problem of an existing natural scene face recognition algorithm are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition and image processing, and more specifically, to a multi-face high-speed recognition method, system and medium based on video stream. Background Art

[0002] In recent years, with the development of emerging information technologies such as big data, artificial intelligence, digital image processing, embedded systems, and the Internet of Things, especially thanks to the rise of deep learning theory and technology, face recognition technology has ushered in a historic change. At present, various applications of face recognition have spread throughout our work, study, and life. Common application areas include access control management, attendance management, mobile payment, software login authentication, etc. In these application areas, the recognition rate and detection speed of existing face recognition systems have reached a very high level.

[0003] The application areas mentioned above are usually referred to as controlled scenarios. This is because the various factors that affect face acquisition, detection and recognition are strictly controlled by humans. These factors can be mainly divided into three categories: internal factors, external factors and other factors: (1) Internal factors are related to the nature of the face itself, mainly including face posture, makeup, expression, age change, hairstyle, whether to wear glasses, whether to wear a mask, etc.; (2) External factors are related to the physical environment and the face acquisition system itself, mainly including ambient lighting, electromagnetic interference noise, resolution, equipment imaging quality, etc.; (3) Other factors mainly refer to whether the user is required to actively adjust his or her facial conditions to cooperate with and assist in face acquisition, detection and recognition.

[0004] However, the real environment is often quite complex. In uncontrolled scenarios, that is, natural scenarios, the quality of the above factors cannot be guaranteed, which poses a huge and even unrealistic challenge to the robustness and generalization ability of the face recognition algorithm itself. Specifically, in surveillance video streams, video faces generally have low-quality factors such as small scale, too strong or too weak lighting, low resolution, and strong noise interference, and there is a large domain difference between them and standard collected faces. If we only improve the structure of the recognition algorithm itself, this will have extremely limited impact on improving the recognition rate. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system and medium for high-speed recognition of multiple faces based on video streams, which creatively combine face restoration and face recognition, use a multi-task learning method, and combine several loss functions to perform two-stage training on the network, specifically including separate training and joint training, to achieve accurate, efficient and universal face recognition of natural scene video streams, and retain identity information, solving the difficult problems and pain points of existing natural scene face recognition algorithms.

[0006] The first aspect of the present invention provides a method for high-speed recognition of multiple faces based on video stream, comprising the following steps:

[0007] Acquire a test image based on a test video stream;

[0008] Input the test image into the trained multi-task restoration and recognition neural network model to obtain the model output;

[0009] Feature extraction is performed on the target image output by the model to obtain identity features and confidence.

[0010] In this solution, the training steps of the multi-task restoration and recognition neural network model specifically include:

[0011] Obtaining a surveillance video stream of a face image to be restored to construct a standard face image dataset;

[0012] Performing image processing based on the standard face image data set to obtain an image pair, wherein the processing method at least includes face detection and face alignment, and a preset image enhancement operation;

[0013] Based on the image pairs after image processing, the initial neural network model is trained in two stages to obtain the multi-task restoration and recognition neural network model, wherein:

[0014] In the first stage, the image is used to separately train the restoration generative adversarial network and the recognition network in the model;

[0015] The second stage uses the image pairs to jointly train the restoration generative adversarial network and the recognition network.

[0016] In this solution, face detection and face alignment are used for image processing, including:

[0017] Face detection and face alignment are performed based on the preset algorithm, where:

[0018] Detect the scale change of the face to obtain the key points of the face;

[0019] The rotation angle is calculated according to the detected facial key points, and the face alignment operation is performed based on the calculated rotation angle.

[0020] In this solution, the preset image enhancement operations are used, including:

[0021] The image is randomly flipped horizontally with a preset probability to obtain a left-right symmetrical processed image;

[0022] Performing random Gaussian noise processing on the processed image, and setting a disturbance range to adjust image parameters;

[0023] The image size is unified and the pixel values ​​are normalized. Bicubic interpolation is used for the unified size processing.

[0024] In this solution, in the first stage, the loss functions used when the restoration generative adversarial network and the recognition network in the separate training model using the image pair include MSE loss function, perceptual loss function, identity loss function and adversarial loss function.

[0025] In this solution, in the second stage, the image pairs are used to jointly train the restoration generative adversarial network, and the loss function used by the recognition network includes a joint loss function.

[0026] The second aspect of the present invention further provides a multi-face high-speed recognition system based on video stream, comprising a memory and a processor, wherein the memory comprises a multi-face high-speed recognition method program based on video stream, and when the multi-face high-speed recognition method program based on video stream is executed by the processor, the following steps are implemented:

[0027] Acquire a test image based on a test video stream;

[0028] Input the test image into the trained multi-task restoration and recognition neural network model to obtain the model output;

[0029] Feature extraction is performed on the target image output by the model to obtain identity features and confidence.

[0030] In this solution, the training steps of the multi-task restoration and recognition neural network model specifically include:

[0031] Obtaining a surveillance video stream of a face image to be restored to construct a standard face image dataset;

[0032] Performing image processing based on the standard face image data set to obtain an image pair, wherein the processing method at least includes face detection and face alignment, and a preset image enhancement operation;

[0033] Based on the image pairs after image processing, the initial neural network model is trained in two stages to obtain the multi-task restoration and recognition neural network model, wherein:

[0034] In the first stage, the image is used to separately train the restoration generative adversarial network and the recognition network in the model;

[0035] The second stage uses the image pairs to jointly train the restoration generative adversarial network and the recognition network.

[0036] In this solution, face detection and face alignment are used for image processing, including:

[0037] Face detection and face alignment are performed based on the preset algorithm, where:

[0038] Detect the scale change of the face to obtain the key points of the face;

[0039] The rotation angle is calculated according to the detected facial key points, and the face alignment operation is performed based on the calculated rotation angle.

[0040] In this solution, the preset image enhancement operations are used, including:

[0041] The image is randomly flipped horizontally with a preset probability to obtain a left-right symmetrical processed image;

[0042] Performing random Gaussian noise processing on the processed image, and setting a disturbance range to adjust image parameters;

[0043] The image size is unified and the pixel values ​​are normalized. Bicubic interpolation is used for the unified size processing.

[0044] In this solution, in the first stage, the loss functions used when the restoration generative adversarial network and the recognition network in the separate training model using the image pair include MSE loss function, perceptual loss function, identity loss function and adversarial loss function.

[0045] In this solution, in the second stage, the image pairs are used to jointly train the restoration generative adversarial network, and the loss function used by the recognition network includes a joint loss function.

[0046] The third aspect of the present invention provides a computer-readable storage medium, which includes a machine-based video stream-based high-speed multi-face recognition method program. When the video stream-based high-speed multi-face recognition method program is executed by a processor, the steps of the video stream-based high-speed multi-face recognition method as described in any one of the above items are implemented.

[0047] The present invention discloses a method, system and medium for high-speed recognition of multiple faces based on video streams, which creatively combine face restoration and face recognition, utilize a multi-task learning method, and combine several loss functions to perform two-stage training on the network, specifically including separate training and joint training, thereby achieving accurate, efficient and universal face recognition for natural scene video streams, retaining identity information, and solving the difficult problems and pain points of existing natural scene face recognition algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a high-speed multi-face recognition method based on video stream of the present invention is shown;

[0049] Figure 2 A block diagram of a high-speed multi-face recognition system based on video streams of the present invention is shown. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0052] In view of the shortcomings of the existing technology, a commonly used solution is: before face recognition, a face restoration algorithm is used to restore a low-quality face image to a high-quality face image, and then face recognition is performed on the restored face. However, some face restoration algorithms proposed with this approach currently have the following problems: 1. Imprecision. They rely on specific mathematical modeling to characterize the relationship between low-quality images and original images. Moreover, since the degradation process of face images is unknown, this mathematical relationship is assumed, approximate, and non-unique. Therefore, the performance of the algorithm is unstable. It performs better on face images in some scenes, but worse on other scenes. 2. Limitations. They only perform face restoration on specific faces with a single low-quality factor, such as face super-resolution, face de-illumination, face denoising, etc., and are powerless for face images affected by multiple low-quality factors at the same time. 3. Identity inconsistency. The face restoration model and the face recognition model are trained separately. Therefore, although the face image processed by the face restoration model overcomes the low-quality factor and is restored to a high-quality face, due to the lack of identity consistency constraints, the face identity attributes may also be lost, which will cause subsequent face recognition errors. Therefore, it is of great practical significance to study a stable, accurate, universal, efficient and identity-retaining face restoration algorithm to improve the performance of face recognition. Therefore, a multi-face high-speed recognition method based on video stream is proposed in the present invention, wherein a multi-task restoration and recognition neural network model (Multi-Task Restoration and Recognition Integrated Network, MTRI-Net) is constructed, and the network is trained in two stages. In the first stage, the restoration generative adversarial network (RGAN) and the recognition network (RN) of MTRI-Net are trained separately. In the second stage, the face restoration network and the face recognition network are integrated into a whole and jointly trained through the multi-task learning method. Multiple loss functions need to be trained in each stage, which ensures the performance of restoration and recognition and the consistency of the identities before and after.

[0053] Specifically, Figure 1 A flowchart of a high-speed multi-face recognition method based on video stream is shown in the present application.

[0054] like Figure 1 As shown, the present application discloses a multi-face high-speed recognition method based on video stream, comprising the following steps:

[0055] S102, acquiring a test image based on a test video stream;

[0056] S104, inputting the test image into the trained multi-task restoration and recognition neural network model to obtain a model output;

[0057] S106, extracting features of the target image output by the model to obtain identity features and confidence.

[0058] It should be noted that, in this embodiment, it is specifically explained how to perform multi-face restoration and recognition, and specifically use a trained multi-task restoration and recognition neural network model for recognition, wherein the training steps are described later, and in specific applications, a test image is obtained through a test video stream, and then the test image is pre-processed and input into the trained multi-task restoration and recognition neural network model for processing, so as to obtain a corresponding target image based on the model output, wherein the target image is a high-quality face image, and then feature extraction is performed on the high-quality face image to identify the identity features and confidence of the high-quality face image, wherein the preprocessing method correspondingly includes face detection, face alignment and image enhancement operations.

[0059] According to an embodiment of the present invention, the training steps of the multi-task restoration recognition neural network model specifically include:

[0060] Obtaining a surveillance video stream of a face image to be restored to construct a standard face image dataset;

[0061] Performing image processing based on the standard face image data set to obtain an image pair, wherein the processing method at least includes face detection and face alignment, and a preset image enhancement operation;

[0062] Based on the image pairs after image processing, the initial neural network model is trained in two stages to obtain the multi-task restoration and recognition neural network model, wherein:

[0063] In the first stage, the image is used to separately train the restoration generative adversarial network and the recognition network in the model;

[0064] The second stage uses the image pairs to jointly train the restoration generative adversarial network and the recognition network.

[0065] It should be noted that, in this embodiment, the training steps of the multi-task restoration and recognition neural network model are specifically described, wherein, specifically, a face image dataset is first constructed, wherein a surveillance video stream containing face images to be restored that are affected by low-quality factors is collected, a face recognition video dataset in a natural scene is constructed, and a standard high-quality face image of the corresponding face is obtained, thereby constructing a standard face image dataset.

[0066] Furthermore, for each frame of the surveillance video in the training set, face detection and face alignment operations are performed, and the low-quality faces after alignment are cropped and image enhancement operations are performed on the low-quality faces to increase the diversity of the data set and the generalization ability and robustness of the network. A large number of low-quality faces and their corresponding identity standard faces form image pairs, and the initial neural network model is trained in two stages. In the first stage, the restoration generative adversarial network and the recognition network in the model are trained separately, and in the second stage, the restoration generative adversarial network and the recognition network are jointly trained to obtain the trained multi-task restoration and recognition neural network model.

[0067] According to an embodiment of the present invention, image processing is performed using face detection and face alignment, specifically including:

[0068] Face detection and face alignment are performed based on the preset algorithm, where:

[0069] Detect the scale change of the face to obtain the key points of the face;

[0070] The rotation angle is calculated according to the detected facial key points, and the face alignment operation is performed based on the calculated rotation angle.

[0071] It should be noted that, in this embodiment, the preset algorithm is the MTCNN algorithm, wherein the MTCNN (Multi-task Cascaded Convolutional Networks) algorithm is specifically a face detection algorithm based on a cascaded convolutional neural network. When in use, it can detect faces of multiple scales at one time, is suitable for monitoring the large scale changes of faces, and has high detection accuracy. According to the detected facial key points, the rotation angle is calculated, the tilted face is aligned, and then the aligned face is cropped from the original image.

[0072] According to an embodiment of the present invention, the preset image enhancement operation is used, specifically including:

[0073] The image is randomly flipped horizontally with a preset probability to obtain a left-right symmetrical processed image;

[0074] Performing random Gaussian noise processing on the processed image, and setting a disturbance range to adjust image parameters;

[0075] The image size is unified and the pixel values ​​are normalized. Bicubic interpolation is used for the unified size processing.

[0076] It should be noted that, in this embodiment, the preset probability is taken as p=0.1, and random horizontal flipping is performed to enable the model to better process left-right symmetrical facial images, and random Gaussian noise with a mean of "0" and a variance of "1" is set to increase the facial images affected by noise; and random salt and pepper noise with white noise and black noise probabilities of "0.01" is set to simulate extreme transmission conditions; and the disturbance range is set to "0.1-0.3" to adjust the image parameters, specifically adjusting the random brightness, contrast and saturation within the disturbance range to enhance the model's generalization ability for color and lighting; finally, the image size is uniformly adjusted to "768x768", and bicubic interpolation is selected as the interpolation method. Selecting a larger size can retain more facial details, which is more suitable for low-quality face restoration and recognition, and the pixel value is normalized to convert the value of each pixel point of the image from "0-255" to "0-1".

[0077] According to an embodiment of the present invention, in the first stage, the loss functions used when the restoration generative adversarial network and the recognition network in the separate training model using the image pair include MSE loss function, perceptual loss function, identity loss function and adversarial loss function.

[0078] It should be noted that, in this embodiment, the restoration generative adversarial network corresponds to (Restoration Generative Adversarial Network, RGAN), and the recognition network corresponds to (Recognition Network, RN), wherein a specific training strategy is adopted to train RGAN and RN separately. By optimizing several loss functions, RGAN can preliminarily learn the basic image restoration transformation function from face images.

[0079] In this embodiment, RGAN draws on the idea of ​​generative adversarial networks (GAN) and includes two parts: a generator and a discriminator. The generator attempts to generate high-quality faces based on low-quality faces, hoping to trick the discriminator into thinking that the generated faces are real faces, while the discriminator hopes to distinguish between generated faces and real faces. In the process of the game between the two, the generation ability of the generator and the discrimination ability of the discriminator are both improved.

[0080] Among them, the generator adopts an encoder-decoder structure, and the specific CNN structure selected is U-Net. U-Net introduces a skip connection mechanism, which directly connects the feature map output of each layer of the encoder to the output of the corresponding decoder, and then inputs it into the next layer after being spliced ​​with the decoder feature map. This method not only makes full use of the features extracted by the encoder, maintains the consistency of the encoder and decoder, but also alleviates problems such as gradient disappearance and gradient explosion.

[0081] The loss functions of the separate training process of RGAN include MSE loss function, perceptual loss function, identity loss function and adversarial loss function.

[0082] For the MSE loss function, the direct pixel difference between the generated high-quality image and the real high-quality image is measured, and its calculation formula is:

[0083]

[0084] in, The restored high-quality face image generated by the RGAN generator, I h is a given real high-quality face image, and N is the total number of pixels in the image.

[0085] In order to make the generated image as close to the real image as possible, in addition to the MSE loss of the bottom pixels of the image, this embodiment introduces perceptual loss, so that the features of each level of the generated image and the features of each level of the real image are as close as possible. This helps to a certain extent that the generated image will not be alienated from the real image at a larger scale feature level, such as the overall structure, shape and identity of the face. It should be noted that the extraction process of the features at each level of perceptual loss is not completed by the encoder of the generator, but by an independent pre-trained CNN. In this embodiment, considering that a complex feature extractor is needed to extract multi-layer features for sufficient feature comparison and hiding, a pre-trained ResNet is used as an independent feature extractor.

[0086] In order to maintain the consistency of the generated faces and the real faces at all levels of features, perceptual loss is introduced Perceived loss In fact, the restored high-quality face and the real high-quality face are sent to independent feature extractors to obtain the feature map representation of each layer of the two respectively. The difference between each corresponding feature point of the feature map of the two is taken and then squared and summed to measure the difference between the feature maps of each layer of the two. The calculation formula is:

[0087]

[0088] Among them, Φ l (*) is the feature extraction function of the lth layer of the feature extractor, and L is the total number of layers. In order to maintain the consistency of the identity information of the generated face and the real face, the identity loss is introduced The pre-trained VGG-Face model is used to extract the identity features of the generated face and the real face. The calculation formula is:

[0089]

[0090] Among them, F(*) is the "1024"-dimensional output identity feature vector of the last fully connected layer of the VGG-Face model.

[0091] Furthermore, adversarial loss is the core loss function of RGAN, which is divided into two parts: generation loss and discrimination loss. The discriminator determines whether the input face is a generated face or a real face. If the output result is closer to "1", the discriminator believes that the input face is a real face (true sample); if it is closer to "0", the discriminator believes that the input face is a generated face (fake sample). The generator hopes to generate faces as close to real faces as possible to confuse the discriminator, that is, to minimize the difference between the generated face discrimination result and "1". The discriminator hopes to minimize the difference between the generated face discrimination result and "0" and minimize the difference between the real face discrimination result and "1" at the same time.

[0092] For the generation loss, the calculation formula is:

[0093]

[0094] Among them, D is the discriminator of RGAN.

[0095] Furthermore, for the discriminative loss, the calculation formula is:

[0096]

[0097] Therefore, in the single training stage, the total loss function of the generator of RGAN is for:

[0098]

[0099] Among them, λ MSE , p , id and λ G are the weight parameters for the MSE loss function, perceptual loss function, identity loss function, and generation loss function, respectively, to measure the priority of the importance of different loss functions.

[0100] Specifically, consider all training samples, let The alternating optimization process of RGAN is obtained. First, the discriminator D is optimized, and the generator G is fixed at this time. The optimization target is:

[0101]

[0102] Among them, I l is the low-quality face image to be restored, P h is the distribution of real high-quality faces in the training set, P lis the distribution of low-quality faces to be restored in the training set. Next, optimize the generator G, and at this time fix the discriminator D. The optimization goal is:

[0103]

[0104] Among them, P hl For I l and I h The joint probability distribution of .

[0105] Furthermore, the discriminator D and the generator G are trained alternately and repeatedly. In the present embodiment, "epochs=300, batch_size=16" are set, the optimizer is the Adam function, and the learning rate is "0.001". Considering that the loss function of the generator G is complex and its learning ability is strong, while the learning ability of the discriminator D is relatively weak, for each mini-batch of each epoch, the number of iterations of the discriminator D is "20" times, and the number of iterations of the generator G is "4" times, giving the discriminator D more learning opportunities.

[0106] In one embodiment of the present invention, the recognition network uses ArcFace, which improves the classic Softmax function and introduces angle cosine boundary loss, and has the characteristics of high precision and robustness. In the single training stage, the improved Softmax loss function (including all training samples) used is:

[0107]

[0108]

[0109] Among them, y (i) is the true label of the i-th high-quality face, W j is the weight vector of the jth category, N is the batch_size, C is the number of identity categories, s is the scaling factor, m is the angle boundary, and the discriminative ability of the enhanced feature is enhanced. j is the angle of the jth category. In order to speed up the training progress, it is first pre-trained on the large-scale face dataset LFW, and then fine-tuned with the self-built dataset. In this embodiment, the training hyperparameters of the recognition network are set to "epochs = 20, batch_size = 16", the optimizer is the Adam function, the learning rate is "0.001", "Weight_Decay = 0.0001, m = 0.4, s = 30".

[0110] According to an embodiment of the present invention, in the second stage, the loss function used for jointly training the restoration generative adversarial network and the recognition network using the image pairs includes a joint loss function.

[0111] It should be noted that, in this embodiment, RGAN and RN are jointly trained to obtain low-quality face images. and the corresponding true identity label y (i) As the input of the entire network to train MTRI-Net, that is:

[0112]

[0113] By training the joint loss function, the restoration quality and recognition quality of RGAN and RN can be improved simultaneously, and the consistency of identity features can be always maintained. Accordingly, the hyperparameters of the joint loss function training are set to "epochs = 100, batch_size = 16", the optimizer is the Adam function, the learning rate is "0.001", and "Weight_Decay = 0.0001, m = 0.4, s = 30".

[0114] Figure 2 A block diagram of a high-speed multi-face recognition system based on video streams of the present invention is shown.

[0115] like Figure 2 As shown, the present invention discloses a high-speed multi-face recognition system based on video stream, including a memory and a processor, wherein the memory includes a high-speed multi-face recognition method program based on video stream, and when the high-speed multi-face recognition method program based on video stream is executed by the processor, the following steps are implemented:

[0116] Acquire a test image based on a test video stream;

[0117] Input the test image into the trained multi-task restoration and recognition neural network model to obtain the model output;

[0118] Feature extraction is performed on the target image output by the model to obtain identity features and confidence.

[0119] It should be noted that, in this embodiment, it is specifically explained how to perform multi-face restoration and recognition, and specifically use a trained multi-task restoration and recognition neural network model for recognition, wherein the training steps are described later, and in specific applications, a test image is obtained through a test video stream, and then the test image is pre-processed and input into the trained multi-task restoration and recognition neural network model for processing, so as to obtain a corresponding target image based on the model output, wherein the target image is a high-quality face image, and then feature extraction is performed on the high-quality face image to identify the identity features and confidence of the high-quality face image, wherein the preprocessing method correspondingly includes face detection, face alignment and image enhancement operations.

[0120] According to an embodiment of the present invention, the training steps of the multi-task restoration recognition neural network model specifically include:

[0121] Obtaining a surveillance video stream of a face image to be restored to construct a standard face image dataset;

[0122] Performing image processing based on the standard face image data set to obtain an image pair, wherein the processing method at least includes face detection and face alignment, and a preset image enhancement operation;

[0123] Based on the image pairs after image processing, the initial neural network model is trained in two stages to obtain the multi-task restoration and recognition neural network model, wherein:

[0124] In the first stage, the image is used to separately train the restoration generative adversarial network and the recognition network in the model;

[0125] The second stage uses the image pairs to jointly train the restoration generative adversarial network and the recognition network.

[0126] It should be noted that, in this embodiment, the training steps of the multi-task restoration and recognition neural network model are specifically described, wherein, specifically, a face image dataset is first constructed, wherein a surveillance video stream containing face images to be restored that are affected by low-quality factors is collected, a face recognition video dataset in a natural scene is constructed, and a standard high-quality face image of the corresponding face is obtained, thereby constructing a standard face image dataset.

[0127] Furthermore, for each frame of the surveillance video in the training set, face detection and face alignment operations are performed, and the low-quality faces after alignment are cropped and image enhancement operations are performed on the low-quality faces to increase the diversity of the data set and the generalization ability and robustness of the network. A large number of low-quality faces and their corresponding identity standard faces form image pairs, and the initial neural network model is trained in two stages. In the first stage, the restoration generative adversarial network and the recognition network in the model are trained separately, and in the second stage, the restoration generative adversarial network and the recognition network are jointly trained to obtain the trained multi-task restoration and recognition neural network model.

[0128] According to an embodiment of the present invention, image processing is performed using face detection and face alignment, specifically including:

[0129] Face detection and face alignment are performed based on the preset algorithm, where:

[0130] Detect the scale change of the face to obtain the key points of the face;

[0131] The rotation angle is calculated according to the detected facial key points, and the face alignment operation is performed based on the calculated rotation angle.

[0132] It should be noted that, in this embodiment, the preset algorithm is the MTCNN algorithm, wherein the MTCNN (Multi-task Cascaded Convolutional Networks) algorithm is specifically a face detection algorithm based on a cascaded convolutional neural network. When in use, it can detect faces of multiple scales at one time, is suitable for monitoring the large scale changes of faces, and has high detection accuracy. According to the detected facial key points, the rotation angle is calculated, the tilted face is aligned, and then the aligned face is cropped from the original image.

[0133] According to an embodiment of the present invention, the preset image enhancement operation is used, specifically including:

[0134] The image is randomly flipped horizontally with a preset probability to obtain a left-right symmetrical processed image;

[0135] Performing random Gaussian noise processing on the processed image, and setting a disturbance range to adjust image parameters;

[0136] The image size is unified and the pixel values ​​are normalized. Bicubic interpolation is used for the unified size processing.

[0137] It should be noted that, in this embodiment, the preset probability is taken as p=0.1, and random horizontal flipping is performed to enable the model to better process left-right symmetrical facial images, and random Gaussian noise with a mean of "0" and a variance of "1" is set to increase the facial images affected by noise; and random salt and pepper noise with white noise and black noise probabilities of "0.01" is set to simulate extreme transmission conditions; and the disturbance range is set to "0.1-0.3" to adjust the image parameters, specifically adjusting the random brightness, contrast and saturation within the disturbance range to enhance the model's generalization ability for color and lighting; finally, the image size is uniformly adjusted to "768x768", and bicubic interpolation is selected as the interpolation method. Selecting a larger size can retain more facial details, which is more suitable for low-quality face restoration and recognition, and the pixel value is normalized to convert the value of each pixel point of the image from "0-255" to "0-1".

[0138] According to an embodiment of the present invention, in the first stage, the loss functions used when the restoration generative adversarial network and the recognition network in the separate training model using the image pair include MSE loss function, perceptual loss function, identity loss function and adversarial loss function.

[0139] It should be noted that, in this embodiment, the restoration generative adversarial network corresponds to (Restoration Generative Adversarial Network, RGAN), and the recognition network corresponds to (Recognition Network, RN), wherein a specific training strategy is adopted to train RGAN and RN separately. By optimizing several loss functions, RGAN can preliminarily learn the basic image restoration transformation function from face images.

[0140] In this embodiment, RGAN draws on the idea of ​​generative adversarial networks (GAN) and includes two parts: a generator and a discriminator. The generator attempts to generate high-quality faces based on low-quality faces, hoping to trick the discriminator into thinking that the generated faces are real faces, while the discriminator hopes to distinguish between generated faces and real faces. In the process of the game between the two, the generation ability of the generator and the discrimination ability of the discriminator are both improved.

[0141] Among them, the generator adopts an encoder-decoder structure, and the specific CNN structure selected is U-Net. U-Net introduces a skip connection mechanism, which directly connects the feature map output of each layer of the encoder to the output of the corresponding decoder, and then inputs it into the next layer after being spliced ​​with the decoder feature map. This method not only makes full use of the features extracted by the encoder, maintains the consistency of the encoder and decoder, but also alleviates problems such as gradient disappearance and gradient explosion.

[0142] The loss functions of the separate training process of RGAN include MSE loss function, perceptual loss function, identity loss function and adversarial loss function.

[0143] For the MSE loss function, the direct pixel difference between the generated high-quality image and the real high-quality image is measured, and its calculation formula is:

[0144]

[0145] in, The restored high-quality face image generated by the RGAN generator, I h is a given real high-quality face image, and N is the total number of pixels in the image.

[0146] In order to make the generated image as close to the real image as possible, in addition to the MSE loss of the bottom pixels of the image, this embodiment introduces perceptual loss, so that the features of each level of the generated image and the features of each level of the real image are as close as possible. This helps to a certain extent that the generated image will not be alienated from the real image at a larger scale feature level, such as the overall structure, shape and identity of the face. It should be noted that the extraction process of the features at each level of perceptual loss is not completed by the encoder of the generator, but by an independent pre-trained CNN. In this embodiment, considering that a complex feature extractor is needed to extract multi-layer features for sufficient feature comparison and hiding, a pre-trained ResNet is used as an independent feature extractor.

[0147] In order to maintain the consistency of the generated faces and the real faces at all levels of features, perceptual loss is introduced Perceived loss In fact, the restored high-quality face and the real high-quality face are sent to independent feature extractors to obtain the feature map representation of each layer of the two respectively. The difference between each corresponding feature point of the feature map of the two is taken and then squared and summed to measure the difference between the feature maps of each layer of the two. The calculation formula is:

[0148]

[0149] Among them, Φ l (*) is the feature extraction function of the lth layer of the feature extractor, and L is the total number of layers. In order to maintain the consistency of the identity information of the generated face and the real face, the identity loss is introduced The pre-trained VGG-Face model is used to extract the identity features of the generated face and the real face. The calculation formula is:

[0150]

[0151] Among them, F(*) is the "1024"-dimensional output identity feature vector of the last fully connected layer of the VGG-Face model.

[0152] Furthermore, adversarial loss is the core loss function of RGAN, which is divided into two parts: generation loss and discrimination loss. The discriminator determines whether the input face is a generated face or a real face. If the output result is closer to "1", the discriminator believes that the input face is a real face (true sample); if it is closer to "0", the discriminator believes that the input face is a generated face (fake sample). The generator hopes to generate faces as close to real faces as possible to confuse the discriminator, that is, to minimize the difference between the generated face discrimination result and "1". The discriminator hopes to minimize the difference between the generated face discrimination result and "0" and minimize the difference between the real face discrimination result and "1" at the same time.

[0153] For the generation loss, the calculation formula is:

[0154]

[0155] Among them, D is the discriminator of RGAN.

[0156] Furthermore, for the discriminative loss, the calculation formula is:

[0157]

[0158] Therefore, in the single training stage, the total loss function of the generator of RGAN is for:

[0159]

[0160] Among them, λ MSE , p , id and λ G are the weight parameters for the MSE loss function, perceptual loss function, identity loss function, and generation loss function, respectively, to measure the priority of the importance of different loss functions.

[0161] Specifically, consider all training samples, let The alternating optimization process of RGAN is obtained. First, the discriminator D is optimized, and the generator G is fixed at this time. The optimization target is:

[0162]

[0163] Among them, I l is the low-quality face image to be restored, P h is the distribution of real high-quality faces in the training set, P l is the distribution of low-quality faces to be restored in the training set. Next, optimize the generator G, and at this time fix the discriminator D. The optimization goal is:

[0164]

[0165] Among them, P hl For I l and I h The joint probability distribution of .

[0166] Furthermore, the discriminator D and the generator G are trained alternately and repeatedly. In the present embodiment, "epochs=300, batch_size=16" are set, the optimizer is the Adam function, and the learning rate is "0.001". Considering that the loss function of the generator G is complex and its learning ability is strong, while the learning ability of the discriminator D is relatively weak, for each mini-batch of each epoch, the number of iterations of the discriminator D is "20" times, and the number of iterations of the generator G is "4" times, giving the discriminator D more learning opportunities.

[0167] In one embodiment of the present invention, the recognition network uses ArcFace, which improves the classic Softmax function and introduces angle cosine boundary loss, and has the characteristics of high precision and robustness. In the single training stage, the improved Softmax loss function (including all training samples) used is:

[0168]

[0169]

[0170] Among them, y (i) is the true label of the i-th high-quality face, W j is the weight vector of the jth category, N is the batch_size, C is the number of identity categories, s is the scaling factor, m is the angle boundary, and the discriminative ability of the enhanced feature is enhanced. j is the angle of the jth category. In order to speed up the training progress, it is first pre-trained on the large-scale face dataset LFW, and then fine-tuned with the self-built dataset. In this embodiment, the training hyperparameters of the recognition network are set to "epochs = 20, batch_size = 16", the optimizer is the Adam function, the learning rate is "0.001", "Weight_Decay = 0.0001, m = 0.4, s = 30".

[0171] According to an embodiment of the present invention, in the second stage, the loss function used for jointly training the restoration generative adversarial network and the recognition network using the image pairs includes a joint loss function.

[0172] It should be noted that, in this embodiment, RGAN and RN are jointly trained to obtain low-quality face images. and the corresponding true identity label y (i) As the input of the entire network to train MTRI-Net, that is:

[0173]

[0174] By training the joint loss function, the restoration quality and recognition quality of RGAN and RN can be improved simultaneously, and the consistency of identity features can be always maintained. Accordingly, the hyperparameters of the joint loss function training are set to "epochs = 100, batch_size = 16", the optimizer is the Adam function, the learning rate is "0.001", and "Weight_Decay = 0.0001, m = 0.4, s = 30".

[0175] The third aspect of the present invention provides a computer-readable storage medium, which includes a method program for high-speed recognition of multiple faces based on video stream. When the method program for high-speed recognition of multiple faces based on video stream is executed by a processor, the steps of a method for high-speed recognition of multiple faces based on video stream as described in any one of the above items are implemented.

[0176] The present invention discloses a method, system and medium for high-speed recognition of multiple faces based on video streams, which creatively combine face restoration and face recognition, utilize a multi-task learning method, and combine several loss functions to perform two-stage training on the network, specifically including separate training and joint training, thereby achieving accurate, efficient and universal face recognition for natural scene video streams, retaining identity information, and solving the difficult problems and pain points of existing natural scene face recognition algorithms.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0179] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0180] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0181] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A high-speed multi-face recognition method based on video stream, characterized in that: The following steps are involved: Acquire a test image based on a test video stream; Input the test image into the trained multi-task restoration and recognition neural network model to obtain the model output; Feature extraction is performed on the target image output by the model to obtain identity features and confidence.

2. The method for high-speed multi-face recognition based on video stream according to claim 1, characterized in that: The training steps of the multi-task restoration and recognition neural network model specifically include: obtaining a surveillance video stream of a face image to be restored to construct a standard face image data set; Performing image processing based on the standard face image data set to obtain an image pair, wherein the processing method at least includes face detection and face alignment, and a preset image enhancement operation; Based on the image pairs after image processing, the initial neural network model is trained in two stages to obtain the multi-task restoration and recognition neural network model, wherein: In the first stage, the image is used to separately train the restoration generative adversarial network and the recognition network in the model; The second stage uses the image pairs to jointly train the restoration generative adversarial network and the recognition network.

3. The method for high-speed multi-face recognition based on video stream according to claim 2, characterized in that: Image processing using face detection and face alignment, including: Face detection and face alignment are performed based on the preset algorithm, where: Detect the scale change of the face to obtain the key points of the face; The rotation angle is calculated according to the detected facial key points, and the face alignment operation is performed based on the calculated rotation angle.

4. The method for high-speed multi-face recognition based on video stream according to claim 3, characterized in that: Take advantage of preset image enhancement operations, including: The image is randomly flipped horizontally with a preset probability to obtain a left-right symmetrical processed image; Performing random Gaussian noise processing on the processed image, and setting a disturbance range to adjust image parameters; The image size is unified and the pixel values ​​are normalized. Bicubic interpolation is used for the unified size processing.

5. The method for high-speed multi-face recognition based on video stream according to claim 4, characterized in that: In the first stage, the loss functions used in the restoration generation adversarial network and the recognition network in the individual training model using the image pair include MSE loss function, perceptual loss function, identity loss function and adversarial loss function.

6. The method for high-speed multi-face recognition based on video stream according to claim 5, characterized in that: In the second stage, the image pairs are used to jointly train the restoration generative adversarial network and the loss function adopted by the recognition network includes a joint loss function.

7. A high-speed multi-face recognition system based on video stream, characterized in that: The invention comprises a memory and a processor, wherein the memory comprises a multi-face high-speed recognition method program based on a video stream, and the multi-face high-speed recognition method program based on a video stream is executed by the processor to implement the following steps: Acquire a test image based on a test video stream; Input the test image into the trained multi-task restoration and recognition neural network model to obtain the model output; Feature extraction is performed on the target image output by the model to obtain identity features and confidence.

8. The high-speed multi-face recognition system based on video stream according to claim 7, characterized in that: The training steps of the multi-task restoration and recognition neural network model specifically include: obtaining a surveillance video stream of a face image to be restored to construct a standard face image data set; Performing image processing based on the standard face image data set to obtain an image pair, wherein the processing method at least includes face detection and face alignment, and a preset image enhancement operation; Based on the image pairs after image processing, the initial neural network model is trained in two stages to obtain the multi-task restoration and recognition neural network model, wherein: In the first stage, the image is used to separately train the restoration generative adversarial network and the recognition network in the model; The second stage uses the image pairs to jointly train the restoration generative adversarial network and the recognition network.

9. The high-speed multi-face recognition system based on video stream according to claim 8, characterized in that: Image processing using face detection and face alignment, including: Face detection and face alignment are performed based on the preset algorithm, where: Detect the scale change of the face to obtain the key points of the face; The rotation angle is calculated according to the detected facial key points, and the face alignment operation is performed based on the calculated rotation angle.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a method program for high-speed recognition of multiple faces based on video streams. When the method program for high-speed recognition of multiple faces based on video streams is executed by a processor, the steps of a method for high-speed recognition of multiple faces based on video streams as described in any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Training method, recognition method and system based on multi-task deep learning network

    CN111666873A

  • Abnormal body temperature monitoring method and device in epidemic situation environment and medium

    CN112329743A

  • Face image restoration method, system and device and storage medium

    CN112801912A

  • Generative adversarial network-based face restoration method and device, and storage medium

    CN114862699A

  • Method and device for generating face recognition model

    WO2020186886A1