Face recognition model training method, face recognition method and device

Abstract

The embodiment of the invention provides a face recognition model training method, a face recognition method and a face recognition device. The method comprises the following steps: acquiring an image sample including a face region, and performing sample enhancement processing on the image sample through a trained generative adversarial network model to obtain an enhanced image sample; the image sample has identity information corresponding to the face region; obtaining a first face feature of a face region in the enhanced image sample and a second face feature of the face region in the image sample; performing feature fusion on the first face feature and the second face feature to obtain a fused feature; the identity information is determined as the label, the face recognition model is trained through the label and the fusion features, and the trained face recognition model is obtained, the face recognition model can accurately recognize faces of different poses in the image, and the face recognition accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a face recognition model training method, a face recognition method and a face recognition device. Background Art

[0002] Facial recognition, a contactless biometric identification technology, is widely used in public safety, intelligent security, and other fields. It is specifically used to determine the identity information corresponding to the facial area in an image.

[0003] In the related art, a facial image including the front face can be obtained through an image acquisition device, and then the image can be directly subjected to facial recognition through an image recognition model to obtain a facial recognition result.

[0004] However, due to the limited shooting angles of image acquisition devices, faces in captured images often exhibit pose deviations, such as only showing a side profile. Related face recognition methods are unable to accurately perform face recognition on such images with pose deviations. Summary of the Invention

[0005] The present invention provides a face recognition model training method, a face recognition method and a face recognition device, so as to solve the technical problem in the related art that when face recognition is performed on a face image with posture deviation, the recognition result has poor accuracy.

[0006] In a first aspect, the present invention provides a face recognition model training method, comprising:

[0007] Obtaining an image sample including a face region, and performing sample enhancement processing on the image sample using a trained generative adversarial network model to obtain an enhanced image sample; the image sample has identity information corresponding to the face region in the image sample;

[0008] Acquire a first facial feature of a facial region in the enhanced image sample and a second facial feature of the facial region in the image sample;

[0009] Performing feature fusion on the first facial feature and the second facial feature to obtain a fused feature;

[0010] The identity information is determined as a label, and a face recognition model is trained using the label and the fusion feature to obtain a trained face recognition model.

[0011] In a second aspect, the present invention provides a face recognition method, comprising:

[0012] Acquire an image to be processed including a face area;

[0013] Inputting the image to be processed into the trained face recognition model to obtain identity information corresponding to the face area in the image to be processed;

[0014] The trained face recognition model is obtained by training according to the method described in the first aspect.

[0015] In a third aspect, the present invention provides a face recognition model training device, comprising:

[0016] A first acquisition module is configured to acquire an image sample including a face region and perform sample enhancement processing on the image sample using a trained generative adversarial network model to obtain an enhanced image sample; the image sample has identity information corresponding to the face region in the image sample;

[0017] A second acquisition module, configured to acquire a first facial feature of the face region in the enhanced image sample and a second facial feature of the face region in the image sample;

[0018] a feature fusion module, configured to fuse the first facial feature and the second facial feature to obtain a fused feature;

[0019] The recognition module is used to determine the identity information as a label, train a face recognition model using the label and the fusion feature, and obtain a trained face recognition model.

[0020] In a fourth aspect, the present invention provides a face recognition device, comprising:

[0021] A third acquisition module is used to acquire an image to be processed including a face area;

[0022] The fourth acquisition module is used to input the image to be processed into the trained face recognition model to obtain identity information corresponding to the face area in the image to be processed; wherein, the trained face recognition model is trained according to the method described in the first aspect.

[0023] In a fifth aspect, the present invention provides an electronic device, characterized in that it includes:

[0024] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to the first aspect is implemented.

[0025] In a sixth aspect, the present invention provides a readable storage medium, characterized in that when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect.

[0026] In an embodiment of the present invention, an image sample is enhanced by a generative adversarial network model to obtain an enhanced image sample. A face recognition model is trained based on the enhanced image sample, thereby improving the generalization and robustness of the face recognition model. Feature fusion is performed on the first face feature of the face area in the enhanced image sample and the second face feature of the face area in the image sample. The face recognition model is trained based on the fused feature, thereby improving the accuracy of the face recognition model in recognizing face features in a small area and accurately recognizing face images in various postures, such as frontal and side faces. In other words, the face recognition model obtained based on the method of this embodiment can not only accurately recognize faces in images including frontal face areas, but also in images including non-frontal face areas such as side faces, thereby solving the problem that the face recognition method in the related art can only perform frontal face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 This is a flowchart of the steps of a face recognition model training method provided by an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of the steps of a face recognition model training method provided by an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of the steps of a face recognition method provided by an embodiment of the present invention;

[0031] Figure 4 This is a flowchart of another face recognition model training method provided by an embodiment of the present invention;

[0032] Figure 5 This is a structural diagram of a face recognition model training device provided by an embodiment of the present invention;

[0033] Figure 6 is a structural diagram of another face recognition device provided by an embodiment of the present invention;

[0034] Figure 7 This is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] It should be noted that the data samples involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0037] Figure 1 This is a flowchart of a face recognition model training method provided by an embodiment of the present invention. Figure 1 As shown, the method may include:

[0038] Step 101: obtain an image sample including a face area, and perform sample enhancement processing on the image sample through the trained generative adversarial network model to obtain an enhanced image sample.

[0039] The image sample has identity information corresponding to the face area in the image sample.

[0040] Generative Adversarial Networks (GANs) are deep learning network models consisting of a generator and a discriminator. These models can generate image data that differs minimally from real-world image samples. They are used to generate images that include facial regions. Image samples are augmented based on the GAN-generated images to produce enhanced image samples.

[0041] Furthermore, the enhanced image samples include image samples generated by GANs and original image samples. Furthermore, the image samples include a first image sample in which the face region is a profile face region, and a second image sample in which another face region (e.g., a frontal face, a half-profile face, etc.) is included; a third image sample in which the face region is a profile face region is generated by GANs, and the first image sample and the third image sample are determined as enhanced image samples.

[0042] For example, a video sample is obtained, and an image sample including a facial region is obtained from the video sample. Specifically, this includes: performing denoising, contrast enhancement, and frame extraction on the video frame images in the video sample to obtain a high-quality image sample. For example, methods such as median filtering and Gaussian filtering can be used to remove noise from the video frame; image contrast can be enhanced through methods such as histogram equalization and automatic contrast adjustment; and key frames are uniformly extracted from the video, and the extracted key frames are determined as the image samples in this step. Key frame extraction can reduce computational complexity and improve data processing efficiency.

[0043] Step 102: Acquire a first facial feature of a facial region in the enhanced image sample and a second facial feature of the facial region in the image sample.

[0044] For example, the facial region in the enhanced image sample can be a profile region. It should be noted that the profile region in this embodiment refers to the facial region in a facial image captured in a profile pose, rather than the profile portion of a frontal facial image. The enhanced image sample includes a first image sample in which the facial region in the image sample is a profile region, and a third image sample generated by GANs in which the facial region is a profile region.

[0045] For example, the first facial feature of the first image sample in the enhanced image sample is obtained by the following method: obtaining an image sample from a video sample, performing side face region detection on the image sample, and performing side face feature extraction on the side face region.

[0046] Among them, the method of side face region detection combines a multi-scale feature fusion algorithm and a target detection algorithm to improve the detection ability of side face regions of different scales. Specifically, a side face region detection model with a multi-scale feature pyramid structure is constructed through the Feature Pyramid Networks (FPN) model to fuse feature information of different scales, thereby improving the detection ability of small-scale side face regions. FPN is a network structure used for target detection and is used for side face region detection. Then, side face image samples are obtained, and the side face image samples are enhanced by operations such as rotation, scaling, and flipping to obtain enhanced side face image samples to improve the generalization ability of the FPN model. Among them, the side face region detection model uses a Max-out classification network as the activation function. Specifically, the Max-out activation function is used in the classification layer of the side face region detection model to improve the classification ability of the side face region detection model for side face regions of different scales.

[0047] The method for extracting profile features includes: obtaining a profile feature extraction network model based on deep learning, which is capable of automatically learning the key features of the profile, including the shape and position information of key parts such as the outline, nose, and mouth, to provide effective feature representation for subsequent recognition. Furthermore, the profile feature extraction network model is a network model based on a convolutional neural network (CNN), which includes multiple convolutional layers, pooling layers, and fully connected layers for automatically learning the above-mentioned key features of the profile. Specifically, the image feature vector of the first image sample is extracted by the profile feature extraction network model, and then the most representative features are selected from the extracted features through a feature selection algorithm as the basis for recognition. The feature selection algorithm includes: selecting profile features from the extracted features for representing the shape and position information of key parts such as the outline, nose, and mouth.

[0048] Furthermore, a side face feature extraction network model is trained by labeling side face image samples with the shape and position information of key parts such as the contour, nose, and mouth, and obtaining a trained side face feature extraction network model. The first image sample is input into the trained side face feature extraction network model to obtain the side face features in the first image sample for characterizing the shape and position information of key parts such as the contour, nose, and mouth.

[0049] Furthermore, the third image sample can be input into the trained profile feature extraction network model to obtain profile features in the third image sample that are used to characterize the shape and position information of key parts such as the contour, nose, and mouth.

[0050] Step 103: Fusing the first facial feature and the second facial feature to obtain a fused feature.

[0051] In this example, the first facial feature and the second facial feature are vector representations of facial features; the first facial feature and the second facial feature are vector-concatenated to obtain a fusion feature.

[0052] For example, the trained generative adversarial network model performs sample enhancement processing on the first image sample in which the face area is the side face area in the image sample to obtain an enhanced image sample in which the face area is the side face area, and extracts the first face feature in the enhanced image sample, where the first face feature is the side face feature; extracts the second face feature of the second image sample in which the face area is other face areas (for example, the front face area) in the image sample; and performs vector splicing on the first face feature and the second face feature to obtain a fusion feature.

[0053] Step 104 : Determine the identity information as a label, train a face recognition model using the label and the fusion feature, and obtain a trained face recognition model.

[0054] Specifically, the fusion features are input into the face recognition model to obtain the face recognition result output by the face recognition model, which includes identity information; according to the loss function of the face recognition model, as well as the identity information output by the face recognition model and the identity information used as a label, the loss function value is calculated, and the model parameters of the face recognition model are optimized through the loss function value to obtain a trained face recognition model.

[0055] With the widespread adoption of video surveillance systems, facial recognition technology has become a crucial technology in areas such as public safety and intelligent security. However, in practical applications, due to the limited viewing angles of the video capture devices in video surveillance systems, only non-frontal facial features (e.g., profiles) can be captured. This makes facial recognition based on non-frontal information difficult. Facial recognition algorithms in related technologies typically rely on frontal facial features for recognition, but their performance is poor for non-frontal features such as profiles, making them difficult to meet practical application requirements.

[0056] In this embodiment, an image sample including a face region is obtained, and sample enhancement processing is performed on the image sample using a trained generative adversarial network model to obtain an enhanced image sample; a first facial feature of the face region in the enhanced image sample and a second facial feature of the face region in the image sample are obtained; feature fusion is performed on the first facial feature and the second facial feature to obtain a fused feature; identity information is determined as a label, and a face recognition model is trained using the label and the fused feature to obtain a trained face recognition model. By performing sample enhancement processing on the image sample using the generative adversarial network model to obtain an enhanced image sample, and training the face recognition model based on the enhanced image sample, the generalization and robustness of the face recognition model can be improved. By performing feature fusion on the first facial feature of the face region in the enhanced image sample and the second facial feature of the face region in the image sample, and training the face recognition model based on the fused feature, the face recognition model can improve the accuracy of the face recognition model in recognizing facial features in a small region, and can accurately recognize facial images in various postures, such as frontal and profile faces. In other words, the face recognition model obtained based on the method of this embodiment can not only accurately identify faces in images including the frontal face area, but also can accurately identify faces in images including non-frontal face areas such as side faces, solving the problem that the face recognition method in the related art can only perform frontal face recognition.

[0057] Figure 2 This is another face recognition model training method provided by the embodiment of the present application, referring to Figure 2 , the method may include the following steps:

[0058] Step 201: Acquire an image sample including a face area.

[0059] For example, the image sample includes a first image sample in which the face area is a side face area, and a second image sample in which the face area is another area.

[0060] For example, collect raw image samples of faces in various poses (especially profiles). For example, the raw image samples can be obtained from an existing face recognition dataset or obtained by self-photographing and annotation. The face recognition dataset can be the Labeled Faces in the Wild (LFW) database, the CASIA face database, etc. In order to increase the diversity of image samples, face images of different genders, ages, races, lighting conditions, and occlusion conditions can be obtained, and raw image samples can be constructed based on these face images. The raw image samples are preprocessed to obtain the image samples in this step. The preprocessing methods may include face detection, cropping, scaling, and normalization.

[0061] Step 202: Perform sample enhancement processing on the first image sample by using a generative adversarial network model to obtain an enhanced image sample.

[0062] The image sample has identity information corresponding to the face region, and the identity information is user identity information of the user corresponding to the face.

[0063] For example, step 202 may include sub-steps A1 to A2:

[0064] Sub-step A1: Generate a third image sample including a side face area through a generative adversarial network model.

[0065] Specifically, to generate more high-quality image samples to enhance the dataset, a GAN model consisting of a generator and a discriminator is constructed. This model is then trained using measured image samples, resulting in a trained GAN model. The generator in the GANs model receives random noise or latent space vectors as input to generate realistic side-face images, while the discriminator distinguishes whether the image input to the discriminator is real or generated by the generator. The generator and discriminator are alternately trained based on side-face image samples, continuously optimizing the generator's generation capabilities and the discriminator's discrimination capabilities until a Nash equilibrium is reached, resulting in a trained GAN model.

[0066] Furthermore, the input images to the discriminator include real profile images and profile images generated by the generator; real profile images can be obtained from existing datasets. The third image sample generated by the GANs is ultimately input into the profile recognition model as augmented data, but the GANs training process itself does not involve input to the profile recognition model.

[0067] GANs are not part of the profile face recognition model, but rather an independent data augmentation tool. Their role is to provide richer and higher-quality input data for the profile face recognition model by generating and enhancing profile face images.

[0068] Sub-step A2: determining the third image sample and the first image sample as enhanced image samples.

[0069] The enhanced image sample includes: a third image sample whose face area is a side face area generated by the GANs model, and a first image sample whose face area is a side face area in the image sample.

[0070] Step 203: Acquire a first facial feature of the facial region in the enhanced image sample.

[0071] For example, step 203 may include sub-steps B1 to B3:

[0072] Sub-step B1: detecting the side face area in the enhanced image sample using a side face area detection model.

[0073] For example, the side face images selected from the image samples are used as a training set to train the side face region detection model. The side face region detection model can adopt a network model based on a deep learning target detection algorithm. The side face region detection model is trained with the training set, and the network structure and parameters are adjusted during the training process to obtain a side face region detection model that can accurately detect the side face region in the image. The side face region detection model can be a single shot multibox detector (SSD), an image recognition model (You Only Look Once, YOLO), a faster region convolutional neural network (Faster Region-based Convolutional Neural Network, Faster R-CNN), or other network models.

[0074] Sub-step B2: extracting the side face features of the side face area using a side face feature extraction model.

[0075] For example, based on the profile face detection model, a profile face feature extraction model can be further trained. This profile face feature extraction model can use a CNN architecture, which automatically learns key features in profile face images through multi-layer convolution and pooling operations. To improve the model's generalization ability, transfer learning can be used: pre-training on a large face dataset and then fine-tuning on the profile face dataset.

[0076] For example, the image sample further includes a second image sample having other face regions.

[0077] Sub-step B3: determining the profile feature as the first facial feature of the face area in the enhanced image sample.

[0078] The profile features include the shape and position features of preset parts of the profile, which are key parts used to characterize the profile identity, such as the profile outline, nose, mouth, etc.

[0079] For example, the image sample further includes a second image sample in which the face region is another face region. The another face region may be a frontal face region, a half-face face region, etc.

[0080] Step 204: Acquire facial features of the face area in the second image sample.

[0081] For example, the method of step 203 may be referred to to obtain the facial features of the face area in the second image sample.

[0082] For example, Haar features and Adaptive Boosting (AdaBoost), Histogram of Oriented Gradients Support Vector Machine (HOG SVM) model can be used to automatically locate the face area in the image sample.

[0083] Before performing facial region detection, the image can be cropped, scaled to a uniform size, and normalized to eliminate the effects of varying lighting, contrast, and other factors on image quality. Profile face images are selected from the processed dataset and manually annotated. The annotations include the face's bounding box, the locations of key points (such as eyes, nose, and mouth) in the image sample, and identity tags. This annotation information is used to train a profile face feature extraction model for extracting profile features.

[0084] Step 205: Determine the facial features of the face area in the second image sample as second facial features.

[0085] The facial region in the second image sample may be a facial region other than a profile face, for example, a frontal face region, a half-profile face region, etc. Correspondingly, the facial features of the facial region in the second image sample may be frontal face features, half-profile face features, etc. Furthermore, the second facial features may be frontal face features, half-profile face features, etc.

[0086] Step 206: Fusing the first facial feature and the second facial feature to obtain a fused feature.

[0087] Furthermore, the accuracy of identity recognition can be further improved through multi-pose fusion processing. Specifically, profile face recognition can be fused with the recognition results of other poses, such as frontal face recognition and half-profile face recognition. By fusing the recognition results under different poses, a more robust and accurate identity judgment can be obtained. For example, the first facial feature of an enhanced image sample with a profile face region and the second facial feature of a second image sample with a frontal face region, half-profile face region, etc. can be concatenated to form feature vectors. The concatenated fused features can then be input into a face recognition model to obtain a trained face recognition model.

[0088] For example, obtain a high-quality multi-pose face dataset. For example, obtain raw image samples from surveillance videos, public databases (such as LFW, CASIA-WebFace), or other sources. These raw image samples must cover different poses, lighting conditions, and occlusions. Each raw image sample must be annotated with "identity" and "pose" labels, such as frontal, half-profile, and profile.

[0089] After obtaining a multi-pose face dataset including original image samples, the original image samples need to be preprocessed, for example, the original image samples need to be standardized. Specifically, the images are aligned through facial key point detection (such as the position of the eyes and the tip of the nose), and the face area of uniform size is cropped. Subsequently, illumination normalization and contrast adjustment are performed to eliminate the interference of environmental factors on recognition. To further enhance data diversity, the original image samples can be rotated, translated or noise perturbation processed. The standardized multi-pose face dataset is obtained, and the pattern samples in the processed face dataset contain dual labels of identity and posture.

[0090] Based on the preprocessed dataset, a feature extraction model is trained separately for each posture. During the training of the feature extraction model, the input data is a single-pose image (e.g., only a front face image or only a side face image) labeled with identity and posture labels. The feature extraction model can be a convolutional neural network, such as a residual neural network (Residual Neural Network, ResNet), a visual geometry group network (Visual Geometry Group Network, VGG), etc., and a transfer learning strategy can be used to fine-tune the feature extraction model: first, the model is trained on a general face dataset (e.g., the VGGFace2 dataset) to learn common facial features such as texture and irrelevant distribution; then, the network parameters are fine-tuned based on the single-pose data to adapt to the feature expression of a specific posture by adjusting the network parameters.

[0091] Furthermore, transfer learning is used to train a feature extraction model based on large-scale general face datasets (such as LFW and CASIA) to learn common facial features such as texture and structure, and to fine-tune the feature extraction model during model training. The fine-tuning method includes: further training the trained feature extraction model using a side face dataset to adjust the network parameters to adapt to the side face features. In the process of fine-tuning through the transfer learning algorithm, backpropagation is used to update the weights in combination with the optimizer to adjust the learning rate (such as cosine annealing) and prevent overfitting. The optimizer can be Adam, Stochastic Gradient Descent (SGD), etc. Among them, only the fully connected layer or part of the convolutional layer (such as the high-level network) is fine-tuned to retain the underlying general feature extraction capabilities. Among them, fine-tuning is a specific technical means of transfer learning, not an independent algorithm.

[0092] Furthermore, by training a lightweight pose classifier, a classifier is obtained for determining the pose type of input image samples. When training pose classification, the input is a multi-pose image, and the output is the corresponding pose label. For example, the classifier can use an efficient model such as MobileNet or combine it with traditional methods such as Support Vector Machine (SVM).

[0093] In one embodiment, the above method can be used to train three single-pose feature extraction models for recognizing the front face, half-profile face, and profile face, respectively, as well as a pose classifier.

[0094] In real-world recognition scenarios, faces to be recognized may present a single pose (such as a profile) or an unknown pose. The input consists of the image to be recognized and a trained single-pose model and classifier. The pose classifier first determines the pose type, then calls the corresponding pose feature extraction model to generate a feature vector. If the input image is in an unknown pose, the output of multiple models must be combined.

[0095] A separately trained feature extraction model is used to extract the first facial features in the enhanced image and the second facial features in the second image, and then multi-pose feature fusion is performed on the first facial features and the second facial features.

[0096] Furthermore, the core logic of feature fusion lies in integrating feature information from different postures. For example, feature fusion can be performed by directly concatenating the feature vectors of different postures (including the feature vectors of the first facial feature and the feature vectors of the second facial feature) to form a high-dimensional comprehensive feature. For example, if the first facial feature is a profile feature and the second facial feature is a frontal feature, the profile and frontal features are combined into a long vector.

[0097] Another example is a weighted fusion method, which assigns weights based on the confidence level of each posture recognition and fuses the first and second facial features based on the weights. For example, if the confidence level of profile recognition is high, the weight of the first facial feature is set to 0.7, and the weight of the second facial feature is set to 0.3. The weighted sum of these factors yields the fused feature.

[0098] For example, feature fusion processing can also be performed through dimensionality reduction optimization methods. Specifically, if the dimension of the fused features is high, the principal component analysis (PCA) method or autoencoder can be used to compress the fused features into a low-dimensional space to retain key discriminant information.

[0099] The fused features are vector representations of comprehensive features, which may include complementary information of multiple postures (e.g., frontal face, side face, and half-side face).

[0100] Step 207: Determine the identity information as a label, train a face recognition model using the label and the fusion feature, and obtain a trained face recognition model.

[0101] For example, the fused features can be compared with multi-pose templates in a database to obtain face recognition results, and the face recognition model can be optimized based on the face recognition results. Specifically, the fused feature vector and the multi-pose features of each identity stored in the database (such as the front and side face features of identity A) are input into the face recognition model. The degree of match between the fused features and the multi-pose template is determined by calculating the similarity. The identity information corresponding to the fused features is then determined based on the degree of match. The similarity can include, for example, cosine similarity and Euclidean distance.

[0102] To improve the robustness of face recognition models, a multi-strategy fusion approach can be used to fuse the decision results. For example, a weighted voting method can be used to fused the decision results. If there are differences in the independent recognition results for different poses, a weight is assigned to each pose based on its confidence level. The weighted weights are then used to calculate the total decision score, which represents the similarity between the fused features and the multi-pose template. For example, if the score for profile recognition is 0.8 and the score for frontal recognition is 0.6, and the weights for profile and frontal recognition are 0.7 and 0.3, respectively, the total decision score is 0.8 × 0.7 + 0.6 × 0.3 = 0.74.

[0103] The facial recognition result is determined by a preset score threshold, for example, 0.75. If the total decision score exceeds the threshold, the identities are considered a match. Otherwise, the identity is marked as unknown or requires manual review. The identity recognition model outputs the final identity determination, for example, identity A or unknown. After obtaining the identity recognition result, it can be fed back to the system for subsequent operations based on actual needs.

[0104] In this embodiment, trained GANs are used to enhance the original profile image sample (i.e., the first image sample), such as by adjusting lighting and posture, to improve the quality and recognition rate of the profile image sample. The enhanced profile image is then input into a face recognition model for profile recognition to perform identity recognition. The face recognition model for measurement and recognition is based on deep learning technology and is trained using a large number of image samples including profile data. It is able to learn the mapping relationship between profile and identity information to accurately perform profile recognition.

[0105] In this embodiment, GANs have been used to enhance the profile image, so even slight differences in profile angles can be better captured and recognized. For example, the enhanced profile image can also be compared with profile templates in a preset database to obtain profile recognition results.

[0106] Furthermore, methods such as cosine similarity and Euclidean distance can be used to calculate the similarity between the first facial feature of the measured image and the facial feature of the side face model in the database. Based on the similarity and a preset similarity threshold, it is determined whether the facial identity of the side face to be identified is the facial identity of the side face template in the database.

[0107] For example, the generative adversarial network model includes a generator and an adversary; the generator includes a deconvolution layer and a residual connection layer. Before step 202, the face recognition model training method of this embodiment also includes steps 208 to 210:

[0108] Step 208: Acquire first data.

[0109] The first data includes one of the following: random noise, latent space vector;

[0110] In step 209, the first data is input into the generator, the data features of the first data are sampled and processed through the deconvolution layer in the generator, and the sampling processing results and the data features are residually connected through the residual connection layer in the generator to obtain a first image.

[0111] In order to make up for the deficiency of side face dataset, GANs is used to generate realistic side face images and enhance the original side face images.

[0112] Step 210, calculating the loss function value between the first image and the fourth image sample through the adversary, optimizing the generative adversarial network model according to the loss function value, and obtaining a trained generative adversarial network model; the fourth image sample has a face area.

[0113] For example, a generative adversarial network model includes a generator and an adversary; the generator includes deconvolution layers and residual connection layers. Specifically, the GANs model receives random noise or latent space vectors and transforms these inputs through a series of network layers in the GANs model to generate a third image sample that resembles a real side face. For example, the generator network uses a structure such as deconvolution layers and residual connections, based on which it can generate high-quality side face images.

[0114] Furthermore, the discriminator network is used to distinguish between the profile images generated by the generator and real profile images. The goal of the discriminator is to classify the authenticity of the input data as accurately as possible. The discriminator network adopts a convolutional neural network structure, which contains multiple convolutional layers, activation layers, and fully connected layers. It is used to distinguish whether the input image is a real profile face or a fake profile face generated by the generator. The training process of the GANs model includes adversarial training of the generator and the discriminator. Through the adversarial training of the generator and the discriminator, the generation ability of the generator and the identification ability of the discriminator are continuously optimized. During the training process, the generator attempts to generate more and more realistic profile images to deceive the discriminator, while the discriminator strives to improve its identification ability to distinguish between the real and the fake.

[0115] In the GANs model, the generator and discriminator are trained using an adversarial loss function. To further improve the quality of generated images, content loss (such as L1 or L2 loss) and perceptual loss (based on the high-level feature loss of the trained network) can be added to the generator to ensure that the generated profile images are not only close to the real images at the pixel level, but also maintain consistency in semantics and texture. L1 loss is an absolute loss, and L2 is a squared loss.

[0116] Compared to GANs models in related technologies, the generator of the GANs model in this embodiment uses a transposed convolution layer and a residual connection structure, while the discriminator uses a convolutional neural network combined with an adversarial training strategy. Furthermore, the loss function design incorporates content loss (L1 / L2) and perceptual loss (based on high-level features of the trained network).

[0117] The GANs model includes a generator with deconvolution layers and residual connections, and a loss function that introduces content loss and perceptual loss. When a third image sample is generated by the trained GANs model, and the third image sample is a side face image, the side face image can be optimized, thereby improving the quality of the side face image generated based on GAN and ensuring the semantic consistency of the side face image.

[0118] This approach generates realistic profile images by inputting random noise or latent space vectors into the generator, rather than directly converting real images. The discriminator distinguishes fake images from real images. Through adversarial training between the two, the generator can gradually improve its generation quality, ultimately producing images that resemble realistic profile images.

[0119] Furthermore, in the early stages of GANs model training, the generator may not be able to produce high-quality profile images, making it easy for the discriminator to distinguish between the generator-generated profile images and real profile images. However, as training progresses, the generator gradually learns how to generate more realistic profile images, and the discriminator also needs to continuously improve its discrimination ability. This process is achieved by alternatingly updating the parameters of the generator and discriminator until a Nash equilibrium is reached. When this state is reached, the generator can generate profile images that are sufficient to deceive the discriminator, while the discriminator cannot accurately distinguish between the generator-generated profile images and real profile images.

[0120] In this embodiment, image samples obtained by methods such as surveillance video, as well as a GANs model and a side face recognition algorithm can be combined to solve the problem of low recognition rate of traditional face recognition technology in complex surveillance environments, especially when the face appears in side face form. By introducing the GANs algorithm, the accuracy and robustness of side face recognition can be improved.

[0121] Furthermore, in this embodiment, by introducing GANs technology to generate and enhance profile images, the deficiencies of the profile dataset are addressed, and the quality and recognition rate of profile images are improved. The profile recognition model obtained based on the method of this embodiment is robust to interference factors such as changes in lighting and posture, and can achieve accurate profile recognition in complex monitoring environments. The method of this embodiment is not only applicable to traditional fields such as public safety and intelligent security, but can also be expanded to multiple fields such as human-computer interaction, providing users with more convenient and secure identity authentication services.

[0122] The present application also provides a face recognition method, referring to Figure 3 , the method may include the following steps:

[0123] Step 301: Acquire an image to be processed including a face area.

[0124] For example, video data to be processed may be obtained, pre-processed such as denoising may be performed on the video data, and video frames to be processed may be extracted at preset time intervals, and the extracted video frames to be processed may be determined as images to be processed.

[0125] Step 302: Input the image to be processed into the trained face recognition model to obtain identity information corresponding to the face area in the image to be processed.

[0126] The trained face recognition model is obtained by training according to the method of any of the above embodiments.

[0127] In this embodiment, an image sample is enhanced using a generative adversarial network model to obtain an enhanced image sample. A face recognition model is trained based on the enhanced image sample, thereby improving the generalization and robustness of the face recognition model. Feature fusion is performed on the first facial feature of the face region in the enhanced image sample and the second facial feature of the face region in the image sample. The face recognition model is trained based on the fused features. The face recognition model thus obtained is used to perform face recognition on the face region in the processed image with high recognition accuracy. The model can also accurately recognize faces in various poses, such as frontal and side faces. In other words, the face recognition model obtained based on the method of this embodiment can not only accurately recognize faces in images including frontal face regions, but also in images including non-frontal face regions, such as side faces, thereby resolving the problem that face recognition methods in related technologies can only perform frontal face recognition.

[0128] Reference Figure 4 The following is a further exemplary description of the training method of the face recognition model of the present application, taking the face recognition model used for side face recognition as an example:

[0129] Step S1: obtaining original image samples from a video, and preprocessing the original image samples to obtain preprocessed image samples.

[0130] Specifically, preprocessing such as denoising is performed on the original image samples to improve the image quality of the original image samples and provide clear original image samples for subsequent processing.

[0131] Step S2: extracting the side face area from the preprocessed image sample using a side face detection algorithm.

[0132] For example, a video data sample can be obtained, and a video frame sample can be obtained from the video data sample, and the video frame sample can be determined as the original image sample in this embodiment. The video frame sample is preprocessed, and then the trained side face region detection model is used to accurately extract the side face region from the preprocessed video frame sample.

[0133] Step S3: extracting the first side face feature of the side face area using the side face feature extraction model.

[0134] Specifically, a profile feature extraction model based on a deep learning network is used to extract the first profile feature, wherein the first profile feature is a part of the first facial feature in the aforementioned embodiment.

[0135] Step S4: Generate a side face image through the GANs model and extract the second side face feature of the side face image.

[0136] The core modules of the profile face recognition model are video preprocessing, profile detection, feature extraction, and recognition and comparison. GANs are only used for training data augmentation and are not involved in the testing phase.

[0137] The side face image generated by the GANs model in this step is the third image sample in the aforementioned embodiment, and the second side face feature in this step is part of the first facial feature in the aforementioned embodiment.

[0138] Step S5: input the first side face feature and the second side face feature into a side face recognition model, train the side face recognition model, and obtain a trained side face recognition model.

[0139] The first facial feature of the aforementioned embodiment includes a first side face feature and a second side face feature. The trained side face recognition model is a facial recognition model used for side face recognition.

[0140] For example, the method of the aforementioned embodiment may also be referred to, and a frontal face image may be used to train a face recognition model to obtain a frontal face recognition model for frontal face recognition.

[0141] Among them, the side face recognition result includes the identity information corresponding to the side face area.

[0142] The profile recognition model uses deep learning technology to learn the mapping relationship between profile features and identities through training. This model establishes an association model from profile images to identity labels. This is achieved through a feature extraction network (such as a CNN). The model learns how to extract key identity-related features (such as outline and facial features) from profile images. Learning this mapping relationship ensures the accuracy and consistency of feature extraction, ensuring that the comparison results reflect the true identity association.

[0143] In the recognition stage, the similarity between the feature vector of the enhanced profile face image and the feature template pre-stored in the database can be calculated. According to the calculated similarity, the identity information corresponding to the profile face area is obtained to complete the profile face recognition.

[0144] Furthermore, a set of profile face images that were not used in the training can be selected as a test set. The image samples in the test set contain different lighting conditions, posture changes, and occlusion situations to comprehensively evaluate the robustness of the profile face recognition model.

[0145] The test set images are fed into the trained profile face recognition model to test the profile face recognition results. Specifically, the profile face detection model detects the profile face region in the test set image samples. The profile face feature extraction model then extracts the profile face features. Finally, the extracted measured features are compared with the templates in the database to determine the identity of the profile face to be recognized, thereby obtaining the profile face recognition results.

[0146] Furthermore, the profile face recognition model's accuracy, recall, and harmonic mean (F1 score) are calculated based on the recognition results and the true labels of the test set image samples to evaluate the algorithm's recognition performance. The model's performance under different lighting conditions, posture changes, and occlusions can also be analyzed to identify potential areas for improvement.

[0147] Figure 5 This is a structural diagram of a face recognition model training device provided by an embodiment of the present invention, referring to Figure 5 , the device 40 comprises:

[0148] The first acquisition module 401 is used to obtain an image sample including a face area, and perform sample enhancement processing on the image sample through a trained generative adversarial network model to obtain an enhanced image sample; the image sample has identity information corresponding to the face area in the image sample; the second acquisition module 402 is used to obtain a first face feature of the face area in the enhanced image sample, and a second face feature of the face area in the image sample; the feature fusion module 403 is used to perform feature fusion on the first face feature and the second face feature to obtain a fused feature; the recognition module 404 is used to determine the identity information as a label, train a face recognition model through the label and the fused feature, and obtain a trained face recognition model.

[0149] Optionally, the image sample includes a first image sample whose facial area is a side face area; the first acquisition module 401 may include: a first acquisition submodule, used to perform sample enhancement processing on the first image sample through a generative adversarial network model to obtain an enhanced image sample.

[0150] Optionally, the image sample also includes a second image sample in which the face area is another face area; the second acquisition module 402 may include: a second acquisition submodule, used to obtain facial features of the face area in the second image sample; and a first determination submodule, used to determine the facial features of the face area in the second image sample as second facial features.

[0151] Optionally, the first acquisition module 401 includes: an image generation submodule, used to generate a third image sample including a side face area through a generative adversarial network model; and a second determination submodule, used to determine the third image sample and the first image sample as enhanced image samples.

[0152] Optionally, the second acquisition module includes: a first extraction submodule, used to detect the side face area in the enhanced image sample through a side face area detection model; a second extraction submodule, used to extract the side face features of the side face area through a side face feature extraction model; and a third determination submodule, used to determine the side face features as the first facial features of the facial area in the enhanced image sample.

[0153] Optionally, the generative adversarial network model includes a generator and an adversary; the generator includes a deconvolution layer and a residual connection layer; the device 40 also includes: a fifth acquisition module for acquiring first data; the first data includes one of the following: random noise, latent space vector; a sixth acquisition module for inputting the first data into the generator, sampling the data features of the first data through the deconvolution layer, and performing residual connection processing on the sampling processing results and the data features through the residual connection layer to obtain a first image; a seventh acquisition module for calculating the loss function value between the first image and the fourth image samples through the adversary, optimizing the adversarial network model according to the loss function value, and obtaining a trained generative adversarial network model; the fourth image sample has a face area.

[0154] In this embodiment, an image sample is enhanced by a generative adversarial network model to obtain an enhanced image sample. The face recognition model is trained based on the enhanced image sample, which can improve the generalization and robustness of the face recognition model. The first facial feature of the face area in the enhanced image sample and the second facial feature of the face area in the image sample are fused. The face recognition model is trained based on the fused features, which can improve the accuracy of the face recognition model in recognizing facial features in small areas and accurately recognize facial images in various postures, such as frontal and side faces. In other words, the face recognition model obtained based on the method of this embodiment can not only accurately recognize faces in images including frontal face areas, but also in images including non-frontal face areas such as side faces, thus solving the problem that the face recognition methods in the related art can only perform frontal face recognition.

[0155] Figure 6 This is a structural diagram of a face recognition model training device provided by an embodiment of the present invention, referring to Figure 6The device 50 includes: a third acquisition module 501, used to obtain an image to be processed including a face area; a fourth acquisition module 502, used to input the image to be processed into a trained face recognition model to obtain identity information corresponding to the face area in the image to be processed; wherein the trained face recognition model is trained according to any of the above methods.

[0156] In this embodiment, an image sample is enhanced using a generative adversarial network model to obtain an enhanced image sample. A face recognition model is trained based on the enhanced image sample, thereby improving the generalization and robustness of the face recognition model. Feature fusion is performed on the first facial feature of the face region in the enhanced image sample and the second facial feature of the face region in the image sample. The face recognition model is trained based on the fused features. The face recognition model thus obtained is used to perform face recognition on the face region in the processed image with high recognition accuracy. The model can also accurately recognize faces in various poses, such as frontal and side faces. In other words, the face recognition model obtained based on the method of this embodiment can not only accurately recognize faces in images including frontal face regions, but also in images including non-frontal face regions, such as side faces, thereby resolving the problem that face recognition methods in related technologies can only perform frontal face recognition.

[0157] The present invention also provides an electronic device, see Figure 7 , including: a processor 701, a memory 702, and a computer program 7021 stored in the memory and capable of running on the processor. When the processor executes the program, the face recognition model training method or face recognition method of the aforementioned embodiment is implemented.

[0158] The present invention also provides a readable storage medium, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the face recognition model training method or face recognition method of the aforementioned embodiment.

[0159] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] It should be noted that the various information and data obtained in the embodiments of the present invention are all obtained with the authorization of the information / data holder.

[0161] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the above description of specific languages is for the purpose of disclosing the best mode of the present invention.

[0162] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0163] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention.

[0164] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0165] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of the sorting device according to the present invention. The present invention may also be implemented as an apparatus or device program for performing a portion or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0166] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0167] The user information (including but not limited to the user's device information, user personal information, etc.) and related data involved in the present invention are all information authorized by the user or authorized by all parties.

[0168] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0170] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A face recognition model training method, characterized in that: include: Obtain an image sample including a face area, and perform sample enhancement processing on the image sample using the trained generative adversarial network model to obtain an enhanced image sample; The image sample has identity information corresponding to a face region in the image sample; Acquire a first facial feature of a facial region in the enhanced image sample and a second facial feature of the facial region in the image sample; Performing feature fusion on the first facial feature and the second facial feature to obtain a fused feature; The identity information is determined as a label, and a face recognition model is trained using the label and the fusion feature to obtain a trained face recognition model.

2. The method according to claim 1, characterized in that The image sample includes a first image sample whose face area is a side face area; the image sample is subjected to sample enhancement processing by a trained generative adversarial network model to obtain an enhanced image sample, including: The first image sample is subjected to sample enhancement processing by the generative adversarial network model to obtain the enhanced image sample.

3. The method according to claim 2, characterized in that The image sample further includes a second image sample whose face region is another face region; and obtaining a second face feature of the face region in the image sample includes: Acquire facial features of the face area in the second image sample; The facial features of the face area in the second image sample are determined as the second facial features.

4. The method according to claim 2, characterized in that The trained generative adversarial network model performs sample enhancement processing on the image sample to obtain an enhanced image sample, including: Generate a third image sample including a side face area by using the generative adversarial network model; The third image sample and the first image sample are determined as the enhanced image samples.

5. The method according to claim 2, characterized in that The obtaining of a first facial feature of a facial region in the enhanced image sample includes: Detecting the side face region in the enhanced image sample using a side face region detection model; Extracting the side face features of the side face area using a side face feature extraction model; The side face feature is determined as the first face feature of the face area in the enhanced image sample.

6. The method according to any one of claims 1 to 5, characterized in that The generative adversarial network model includes a generator and an adversary; the method further includes: Acquire first data; the first data includes at least one of the following: random noise, latent space vector; Inputting the first data into the generator, performing sampling processing on data features of the first data through a deconvolution layer in the generator, and performing residual connection processing on the sampling processing results and the data features through a residual connection layer in the generator to obtain a first image; The adversary calculates the loss function value between the first image and the fourth image samples, optimizes the generative adversarial network model according to the loss function value, and obtains a trained generative adversarial network model; the fourth image sample has the face area.

7. A face recognition method, characterized in that: include: Acquire an image to be processed including a face area; Inputting the image to be processed into a trained face recognition model to obtain identity information corresponding to the face area in the image to be processed; The trained face recognition model is obtained by training according to the method according to any one of claims 1 to 6.

8. A face recognition model training device, characterized in that: include: A first acquisition module is used to acquire an image sample including a face area, and perform sample enhancement processing on the image sample using the trained generative adversarial network model to obtain an enhanced image sample; The image sample has identity information corresponding to a face region in the image sample; A second acquisition module, configured to acquire a first facial feature of the face region in the enhanced image sample and a second facial feature of the face region in the image sample; a feature fusion module, configured to fuse the first facial feature and the second facial feature to obtain a fused feature; The recognition module is used to determine the identity information as a label, train a face recognition model using the label and the fusion feature, and obtain a trained face recognition model.

9. A face recognition device, characterized in that: include: A third acquisition module is used to acquire an image to be processed including a face area; The fourth acquisition module is used to input the image to be processed into a trained face recognition model to obtain identity information corresponding to the face area in the image to be processed; wherein the trained face recognition model is trained according to the method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

11. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods of claims 1-7.

Images