Face feature encoder training method, face verification method and related device
By training a face alignment network model to generate a frontal image of a side view and initializing a face feature encoder, the reliability problem of side face image verification is solved, and accurate recognition and security verification of side images are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU WANGDAO HLDG CO LTD
- Filing Date
- 2023-04-27
- Publication Date
- 2026-07-14
Smart Images
Figure CN116740483B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and in particular to a face feature encoder training method, a face verification method, and related equipment. Background Technology
[0002] With the development of mobile internet, compared with fingerprint recognition, iris recognition and voice recognition, facial recognition, as a mature, convenient and non-invasive recognition technology, has been widely used in various verification scenarios, such as facial verification for online transactions and facial verification for community access control.
[0003] Currently, face verification mainly relies on facial landmark detection and alignment to extract facial features for recognition and comparison. However, this two-dimensional face alignment-based recognition and verification method struggles to complete face verification when faced with images containing only side-view facial information, thus reducing the reliability of face verification. Summary of the Invention
[0004] In view of the above problems, this disclosure provides a face feature encoder training method, face verification method, and related equipment that overcome or at least partially solve the above problems. The technical solution is as follows:
[0005] A method for training a facial feature encoder includes:
[0006] A pre-trained face alignment network model is obtained, wherein the face alignment network model includes an image encoder and a face image generator;
[0007] The first side profile image of the face is input into the face alignment network model. The image encoder and the face image generator are used to generate the first frontal image of the face. The first side profile image and the first frontal image of the face are used as a first pair of data.
[0008] The face feature encoder is initialized using the network weights of the image encoder;
[0009] The face feature encoder is trained using the first pair of data to obtain the trained face feature encoder, which is used to perform face verification on the face image to be verified.
[0010] Optionally, obtaining the pre-trained face alignment network model includes:
[0011] Obtain a pre-trained face image generator;
[0012] In the face alignment network model, the face image generator is fixed, and the face alignment network model is initialized and trained using a first face image pair dataset. The network weights of the image encoder in the face alignment network model are adjusted. The first face image pair dataset includes at least one set of second paired data, and the second paired data includes two identical face images.
[0013] The face alignment network model is trained using a second face image pair dataset, and the network weights of the image encoder in the face alignment network model are adjusted to obtain the trained face alignment network model. The second face image pair dataset includes at least one set of third paired data, and the third paired data includes a second face side image and its corresponding second face front image.
[0014] Optionally, the step of inputting the first side-view image of the face into the face alignment network model, and generating a first frontal image of the face using the image encoder and the face image generator, includes:
[0015] The first side profile image of a face is input into the face alignment network model, and the image encoder is used to encode the first side profile image of the face to obtain the first face feature code corresponding to the first side profile image of the face.
[0016] The first facial feature code is input into the face image generator to obtain the first frontal face image output by the face image generator.
[0017] Optionally, training the face feature encoder using the first pair of data to obtain the trained face feature encoder, used for face verification of the face image to be verified, includes:
[0018] The first face profile image from the first pair of data is input into the face feature encoder to obtain the second face feature code corresponding to the first face profile image;
[0019] The first frontal face image from the first pair of data is input into the face feature encoder to obtain the third face feature code corresponding to the first frontal face image;
[0020] Using the second and third face feature codes, the network weights of the face feature encoder are adjusted to obtain the trained face feature encoder, which is used to perform face verification on the face image to be verified.
[0021] Optionally, before training the face alignment network model using the second face image dataset, the method further includes:
[0022] Random geometric transformations are performed on each of the second face profile images in the dataset.
[0023] A face verification method includes:
[0024] Obtain the face image to be verified;
[0025] The face image to be verified is input into a pre-trained face feature encoder to obtain a face feature code corresponding to the face image to be verified, wherein the face feature encoder is trained using any of the face feature encoder training methods described above;
[0026] The face feature encoding is used to perform face verification on the face image to be verified, and the face verification result is obtained.
[0027] A face feature encoder training device includes: a face alignment network model acquisition unit, a pairwise data acquisition unit, a face feature encoder initialization unit, and a face feature encoder training unit.
[0028] The face alignment network model obtaining unit is used to obtain a pre-trained face alignment network model, wherein the face alignment network model includes an image encoder and a face image generator;
[0029] The paired data acquisition unit is used to input the first face side image into the face alignment network model, use the image encoder and the face image generator to generate the first face front image, and take the first face side image and the first face front image as a first pair of data.
[0030] The face feature encoder initialization unit is used to initialize the face feature encoder using the network weights of the image encoder;
[0031] The face feature encoder training unit is used to train the face feature encoder using the first pair of data to obtain the trained face feature encoder, which is used to perform face verification on the face image to be verified.
[0032] A face verification device includes: a face image acquisition unit, a face feature encoding acquisition unit, and a verification result acquisition unit.
[0033] The face image acquisition unit is used to acquire the face image to be verified.
[0034] The face feature encoding acquisition unit is used to input the face image to be verified into a pre-trained face feature encoder to obtain the face feature encoding corresponding to the face image to be verified, wherein the face feature encoder is trained using the face feature encoder training device described above.
[0035] The verification result obtaining unit is used to perform face verification on the face image to be verified using the face feature encoding, and obtain the face verification result.
[0036] A computer-readable storage medium storing a program that, when executed by a processor, implements the face feature encoder training method described above; and / or implements the face verification method described above.
[0037] An electronic device includes at least one processor, at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the face feature encoder training method described above; and / or to execute the face verification method described above.
[0038] By employing the above technical solutions, the face feature encoder training method, face verification method, and related equipment provided in this disclosure can obtain a pre-trained face alignment network model. The face alignment network model includes an image encoder and a face image generator. A first side-view image of a face is input into the face alignment network model. Using the image encoder and the face image generator, a first frontal image of a face is generated. The first side-view image and the first frontal image of a face are used as a first pair of data. The face feature encoder is initialized using the network weights of the image encoder. The face feature encoder is trained using the first pair of data to obtain a trained face feature encoder, which is used for face verification of the face image to be verified. This disclosure generates a frontal image of a face corresponding to a side-view image through the face alignment network model, avoiding the loss of face features. This enables the face feature encoder trained in this way to accurately recognize side-view images of faces, thereby improving the reliability of face verification.
[0039] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description
[0040] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0041] Figure 1 A flowchart illustrating one embodiment of the face feature encoder training method provided in this disclosure is shown.
[0042] Figure 2 A flowchart illustrating another implementation of the face feature encoder training method provided in this disclosure is shown.
[0043] Figure 3 A flowchart illustrating another implementation of the face feature encoder training method provided in this disclosure is shown.
[0044] Figure 4 A flowchart illustrating another implementation of the face feature encoder training method provided in this disclosure is shown.
[0045] Figure 5 A flowchart illustrating one implementation of the face verification method provided in this disclosure is shown.
[0046] Figure 6 A schematic diagram of a face feature encoder training device provided in an embodiment of this disclosure is shown;
[0047] Figure 7 A schematic diagram of a face verification device provided in an embodiment of this disclosure is shown. Detailed Implementation
[0048] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0049] Face verification typically includes four basic steps: face detection, face alignment, face encoding, and face matching. First, a face detection algorithm is used to detect faces in the acquired image. Then, the detected faces are cropped and rotated and aligned based on the detected facial landmarks. Next, an encoder is used to extract facial features, and face verification is completed through feature lookup and comparison.
[0050] However, 2D rotation alignment of faces cannot solve the problem of lateral alignment of faces in the 3D direction; that is, rotation alignment of facial key points cannot achieve the recognition and verification of lateral facial images. Current mainstream face recognition detection uses face images of fixed size and pose as input. The drawback of this is that various guidance frames are needed during face verification. If the pose of the person being photographed or the position between the person and the camera is incorrect, the face will be judged as unsuccessful and needs to be restarted. This operation is extremely inefficient and provides a poor user experience. Furthermore, such fixed-angle frontal images are easily compromised by techniques such as DeepFake, reducing the reliability of face verification.
[0051] Although existing adversarial networks can detect side faces by detecting face yaw angle, the face verification process is time-consuming and difficult to apply in scenarios with high real-time requirements. At the same time, directly using adversarial networks to generate frontal images from side face images will lose a lot of feature information, meaning that the generated frontal images are not like the original side face images, resulting in low accuracy of face verification.
[0052] To address the limitations of current face verification methods, this embodiment trains a face feature encoder and uses the trained encoder for face verification. During face verification, face alignment is not required; the encoder representing facial features can be directly extracted from the side profile image for verification, thereby improving the reliability and security of face verification.
[0053] like Figure 1 The diagram shows a flowchart of one embodiment of the face feature encoder training method provided in this disclosure. The method may include:
[0054] S100. Obtain a pre-trained face alignment network model, wherein the face alignment network model includes an image encoder and a face image generator.
[0055] Among them, the face alignment network model can be an end-to-end network model based on StyleGAN (A Style-Based Generator Architecture for Generative Adversarial Networks).
[0056] The image encoder can be a PSP (Pixel2Style2Pixel) based encoder. The face image generator can be a StyleGAN based face generator. In the face alignment network model, the image encoder performs the encoding function, and the face image generator performs the decoding function.
[0057] The image encoder is implemented based on Feature Pyramid Networks (FPN). In the face alignment network model, the image encoder extracts features at three different levels according to the proportion of each layer in the Feature Pyramid Network, and then uses a fully connected module (map2style) to generate 18 codes, i.e., 18 512-dimensional vectors.
[0058] Because convolution operations are independent of image size, the input of a fully convolutional architecture can support images of different sizes. This embodiment removes the fixed limitation on input image size imposed by the StyleGAN-based face generator and adds an image encoder to support different image sizes, resulting in a face alignment network model employing a fully convolutional "image encoder-face image generator" architecture.
[0059] The image encoder downsamples the input image of size W×H and extracts its W / 8×H / 8 features through downsampled convolutional layers and ResBlocks. These features replace the original fixed-size input features of StyleGAN at layer 8. Therefore, the face alignment network model supports image inputs where sizes W and H are divisible by 8.
[0060] The embodiments disclosed herein can input the 18 codes output by the image encoder into the face image generator layer by layer through affine transformation, thereby realizing a graph-to-graph end-to-end network model.
[0061] Optional, based on Figure 1 The method shown is as follows: Figure 2 As shown, this is a flowchart illustrating another implementation of the face feature encoder training method provided in this disclosure. Step S100 may include:
[0062] S110, Obtain a pre-trained face image generator.
[0063] This disclosure embodiment can use real human face images as training data to train a face image generator so that the trained face image generator can generate frontal human face images.
[0064] S120. Fix the face image generator in the face alignment network model, use the first face image pair dataset to initialize the face alignment network model for training, and adjust the network weights of the image encoder in the face alignment network model. The first face image pair dataset includes at least one set of second paired data, and the second paired data includes the same two face images.
[0065] In this dataset, the face images can be either side-view or front-view images. Front-view images primarily refer to aligned images where the facial features are symmetrically displayed. Side-view images, in contrast to front-view images, mainly refer to misaligned images where the facial features are not symmetrically displayed, or where some facial features are not visible.
[0066] This embodiment of the disclosure fixes the face image generator in the face alignment network model, so that the network parameters of the face image generator are not changed during the training process of the face alignment network model.
[0067] This disclosure embodiment can use the same face image as the input and output of the face alignment network model to train the face alignment network model, and adjust the network weights of the image encoder in the face alignment network model during the training process to achieve the initialization of the image encoder.
[0068] S130. The face alignment network model is trained using the second face image pair dataset. The network weights of the image encoder in the face alignment network model are adjusted to obtain the trained face alignment network model. The second face image pair dataset includes at least one set of third paired data, which includes a second face side image and its corresponding second face front image.
[0069] In this embodiment, after initial training of the face alignment network model using a first face image dataset, the face alignment network model can be further trained using a second face image dataset based on the current network weights of the image encoder in the face alignment network model.
[0070] The second side view image and the second front view image can be real face images.
[0071] In this embodiment, a second side view image of a face can be used as the input to a face alignment network model, and a frontal image of a second face can be used as the output of the face alignment network model. A loss function is used to supervise the training process of the face alignment network model, and the network weights of the image encoder in the face alignment network model are adjusted during the training process to achieve further optimization of the image encoder.
[0072] Optionally, the loss function provided in this disclosure embodiment can be any one of Mean Squared Error Loss (MSE loss), Learned Perceptual Image Patch Similarity Loss (LPIPS loss), Regularization Loss, and Identity Loss (ID loss).
[0073] This embodiment of the present disclosure uses a loss function to supervise the training process of the face alignment network model, which can retain the facial feature information of the second face side image and the second face front image, so that the face front image generated by the trained face alignment network model can retain the facial feature information of the corresponding face side image, thereby improving the realism of the generated face front image.
[0074] Optionally, in order to enable the face alignment network model to learn to process face images of multiple angles and sizes, embodiments of this disclosure may perform random geometric transformations on each second face side image in the second face image pair dataset before S130.
[0075] Random geometric transformations can include flipping, scaling, translation, and rotation.
[0076] This embodiment of the disclosure enhances the face alignment network model by performing random geometric transformations on each second face profile image in the dataset. This enables the trained face alignment network model to process face profile images from multiple angles and sizes, and to output corresponding frontal face images for face profile images in different scenarios.
[0077] This embodiment of the disclosure trains a face alignment network model and adjusts the network weights of the image encoder so that the trained face alignment network model can generate a more realistic frontal face image based on the input side face image, thereby providing rich training data for the face feature encoder.
[0078] Optionally, embodiments of this disclosure may use the same frontal face image as input and output to initialize and reconstruct the image encoder.
[0079] S200. Input the first face side image into the face alignment network model, use the image encoder and face image generator to generate the first face front image, and take the first face side image and the first face front image as a first pair of data.
[0080] The first profile image of the face can be a real face image. The first frontal image of the face is a face image generated by inference through the image encoder and face image generator of the face alignment network model.
[0081] Optional, based on Figure 1 The method shown is as follows: Figure 3 As shown, this is a flowchart illustrating another implementation of the face feature encoder training method provided in this disclosure. Step S200 may include:
[0082] S210. Input the first face side image into the face alignment network model, and use the image encoder to encode the first face side image to obtain the first face feature code corresponding to the first face side image.
[0083] In this context, facial feature encoding refers to the data information corresponding to a facial image in the latent space. The distance between different facial feature encodings can represent the similarity between different faces.
[0084] S220. Input the first facial feature code into the face image generator to obtain the first frontal face image output by the face image generator.
[0085] Specifically, in this embodiment, the first face side image can be extracted according to the proportion of each layer in the feature pyramid network to extract three different levels of features. Then, a first face feature code is generated using a fully connected module. The first face feature code is then input into the face image generator layer by layer through affine transformation to obtain the first face frontal image generated by the face image generator based on the first face feature code.
[0086] This embodiment of the disclosure uses an image encoder and a face image generator in a pre-trained face alignment network model to generate corresponding frontal face images from real side-view images. In the absence of paired data of real side-view and frontal face images, it generates corresponding frontal face images from collected side-view images, providing rich training data for the face feature encoder.
[0087] S300: Initialize the face feature encoder using the network weights of the image encoder.
[0088] The structure of the face feature encoder is the same as that of the image encoder, and it can be an encoder based on PSP (Pixel2Style2Pixel).
[0089] This embodiment of the disclosure can initialize the face feature encoder based on the network weights of the image encoder in the pre-trained face alignment network model, so that the network parameters of the face feature encoder are consistent with those of the image encoder in the pre-trained face alignment network model.
[0090] S400. The face feature encoder is trained using the first pair of data to obtain a trained face feature encoder, which is used to perform face verification on the face image to be verified.
[0091] This embodiment of the disclosure can use a frontal face image and its corresponding side face image generated by a face alignment network model, and supervise the training process of the face feature encoder after initialization through a classification task, adjusting the network weights of the face feature encoder during the training process.
[0092] It is understood that the embodiments of this disclosure can utilize a face alignment network model to generate a large number of frontal face images, thereby using multiple sets of first paired data to train the face feature encoder in multiple rounds, improving the size robustness and pose robustness of the face recognition of the face feature encoder, and thus improving the real-time performance and stability of face verification.
[0093] Optionally, when training the face feature encoder using the first pair of data, embodiments of this disclosure can also use a positive sample face image pair dataset and a negative sample face image pair dataset to supervise the training process of the initialized face feature encoder through a classification task. During training, the network weights of the face feature encoder are adjusted. The positive sample face image pair dataset includes at least one fourth pair of data with positive labels, indicating that the face images in the fourth pair belong to the same person; that is, the fourth pair includes two face images of the same person. The negative sample face image pair dataset includes at least one fifth pair of data with negative labels, indicating that the face images in the fifth pair do not belong to the same person; that is, the fifth pair includes a face image of one person and a face image of another person. The face images can be frontal or side-view images. This embodiment of the disclosure continues to train the face feature encoder by using positive sample face image pairs datasets and negative sample face image pairs datasets. This enables the face feature encoder to not only identify whether a side view image and a front view image are of the same person, but also to identify face images of different people, thereby better completing the face verification task.
[0094] Optional, based on Figure 1 The method shown is as follows: Figure 4 As shown, this is a flowchart illustrating another implementation of the face feature encoder training method provided in this disclosure. Step S400 may include:
[0095] S410. Input the first face profile image from the first pair of data into the face feature encoder to obtain the second face feature code corresponding to the first face profile image.
[0096] S420. Input the first frontal face image from the first pair of data into the face feature encoder to obtain the third face feature code corresponding to the first frontal face image.
[0097] S430. Using the second and third face feature codes, adjust the network weights of the face feature encoder to obtain a trained face feature encoder, which is used to perform face verification on the face image to be verified.
[0098] Specifically, in this embodiment, the training process of the face feature encoder after initialization can be supervised by a loss function with the goal of reducing the distance between the second face feature code and the third face feature code. The network weights of the face feature encoder can be adjusted so that the face feature code obtained by the trained face feature encoder from the side view image is closer to the face feature code from the front view image, which helps to perform face verification through the side view image.
[0099] This embodiment uses first pairwise data to distill a lightweight face feature encoder through supervised training. This encoder supports the extraction and conversion of face features from both side-view and front-view images. It supports variable-size and misaligned face image inputs, making it suitable not only for common face verification scenarios using front-view images but also for side-view images. This embodiment obtains the face feature encoding of a user's side-view image through the face feature encoder and uses this encoding for face verification. This avoids the attack risks of DeepFake and other forgery techniques faced when using front-view images for face verification, thus improving the authenticity and reliability of face verification.
[0100] The face feature encoder training method disclosed herein can obtain a pre-trained face alignment network model, wherein the face alignment network model includes an image encoder and a face image generator. A first side-view image of a face is input into the face alignment network model, and a first frontal image of the face is generated using the image encoder and the face image generator. The first side-view image and the first frontal image of the face are used as a first pair of data. The face feature encoder is initialized using the network weights of the image encoder. The face feature encoder is trained using the first pair of data to obtain a trained face feature encoder, which is used for face verification of the face image to be verified. This disclosure generates a frontal image of the face corresponding to the side-view image of the face through the face alignment network model, which can avoid the loss of face features, enabling the face feature encoder trained thereby to accurately recognize the side-view image of the face, thus improving the reliability of face verification.
[0101] like Figure 5 The diagram shows a flowchart of one embodiment of the face verification method provided in this disclosure. The method may include:
[0102] S500: Obtain the face image to be verified.
[0103] The face image to be verified can be a side view image or a front view image.
[0104] Specifically, in this embodiment of the disclosure, a face detection algorithm can be used to identify the face image to be verified.
[0105] Optionally, embodiments of this disclosure use a lightweight, single-step face detection network, RetinaFace, to detect whether a face exists in the input image. If a face exists, the face portion of the image is cropped to obtain the face image to be verified.
[0106] S600. Input the face image to be verified into the pre-trained face feature encoder to obtain the face feature code corresponding to the face image to be verified.
[0107] The face feature encoder can downsample the face image to be verified, extract the face feature information of the face image to be verified through sampling convolutional layers and ResBlocks, and then use a fully connected module to generate a face feature code corresponding to the face image to be verified based on the face feature information.
[0108] The face feature encoder is trained using any of the face feature encoder training methods provided in this disclosure.
[0109] S700: Using facial feature encoding, perform facial verification on the facial image to be verified and obtain the facial verification result.
[0110] Optionally, in this embodiment, the distance between the facial feature codes of each known face image in the target database and the facial feature codes of the face image to be verified can be calculated respectively. This distance is used as a similarity metric for face verification; the smaller the distance, the more similar the faces are. If the distance between the facial feature codes of any known face image in the target database and the facial feature codes of the face image to be verified is less than a first preset threshold, the obtained face verification result is that the known face image and the face image to be verified belong to the same person. The facial feature codes of each known face image in the target database are generated by a facial feature encoder trained using any of the facial feature encoder training methods provided in this embodiment. This embodiment of the present disclosure compares the facial feature codes of the face image to be verified with the facial feature codes of each known face image in the target database to determine whether the face image to be verified belongs to any known face image in the target database, thus achieving face verification.
[0111] Optionally, the face verification method provided in this disclosure can be applied to access control machines. A face feature encoder is trained using any of the face feature encoder training methods provided in this disclosure. Face feature codes are pre-generated for the face images of residents or company employees, and these face feature codes are stored in a target database. When the access control machine collects a user's face image to be verified, the face feature encoder obtains the face feature code corresponding to the face image to be verified. This face feature code is then compared with the face feature codes of the face images of residents or company employees in the target database to verify whether the user is a resident or company employee.
[0112] Optionally, embodiments of this disclosure may obtain a user identifier corresponding to the face image to be verified, and obtain a face feature code of a reserved face image corresponding to the user identifier. The face feature code of the reserved face image is the code generated by a face feature encoder trained using any of the face feature encoder training methods provided in embodiments of this disclosure. When the face feature encoder is trained using any of the face feature encoder training methods provided in embodiments of this disclosure to generate the face feature code for the face image to be verified, the distance between the face feature code corresponding to the face image to be verified and the face feature code of the reserved face image is calculated. This distance is used as a similarity metric for face verification. If the distance is less than a second preset threshold, the face verification result is that the known face image and the face image to be verified belong to the same person, and the face verification passes.
[0113] Optionally, the face verification method provided in this disclosure can be applied to online transaction scenarios. A face feature encoder, trained using any of the face feature encoder training methods provided in this disclosure, generates a face feature code for a user's pre-reserved face image and saves this face feature code in correspondence with the user's user identifier. When a user uses the online transaction service, the system obtains the user's face image to be verified and the user identifier. Based on the transaction system, it obtains the face feature code of the pre-reserved face image corresponding to the user identifier. The face feature encoder then obtains the face feature code corresponding to the face image to be verified. Finally, the system compares the face feature code corresponding to the face image to be verified with the face feature code of the pre-reserved face image to verify whether the face image to be verified and the pre-reserved face image belong to the same user.
[0114] The face verification method provided in this disclosure can obtain a face image to be verified, input the face image to be verified into a pre-trained face feature encoder, and obtain a face feature code corresponding to the face image to be verified. The face feature encoder is trained using any of the face feature encoder training methods provided in the embodiments of this disclosure. Face verification is performed on the face image to be verified using the face feature code to obtain the face verification result. This disclosure implements face verification of the face image to be verified through a face feature encoder, and can be applied to face verification scenarios including access control and online transactions. It also supports face verification using side-view images of the face, effectively reducing the attack risk of face spoofing technology, increasing the security and reliability of face verification, and improving the stability, convenience, and user-friendliness of face verification.
[0115] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.
[0116] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0117] Corresponding to the above-described embodiments of the face feature encoder training method, this disclosure also provides a face feature encoder training device, the structure of which is as follows: Figure 6 As shown, it includes: a face alignment network model acquisition unit 100, a pairwise data acquisition unit 200, a face feature encoder initialization unit 300, and a face feature encoder training unit 400.
[0118] The face alignment network model acquisition unit 100 is used to obtain a pre-trained face alignment network model, wherein the face alignment network model includes an image encoder and a face image generator.
[0119] The paired data acquisition unit 200 is used to input the first face side image into the face alignment network model, generate the first face front image using the image encoder and face image generator, and take the first face side image and the first face front image as a first paired data set.
[0120] The face feature encoder initialization unit 300 is used to initialize the face feature encoder using the network weights of the image encoder.
[0121] The face feature encoder training unit 400 is used to train the face feature encoder using the first pair of data to obtain a trained face feature encoder, which is used to perform face verification on the face image to be verified.
[0122] Optionally, the face alignment network model obtaining unit 100 may include: a face image generator obtaining subunit, a first training subunit, and a second training subunit.
[0123] The face image generator acquisition subunit is used to obtain a pre-trained face image generator.
[0124] The first training subunit is used to fix the face image generator in the face alignment network model, initialize the face alignment network model using the first face image pair dataset, and adjust the network weights of the image encoder in the face alignment network model. The first face image pair dataset includes at least one set of second paired data, and the second paired data includes the same two face images.
[0125] The second training subunit is used to train the face alignment network model using the second face image pair dataset, adjust the network weights of the image encoder in the face alignment network model, and obtain the trained face alignment network model. The second face image pair dataset includes at least one set of third paired data, which includes a second face side image and its corresponding second face front image.
[0126] Optionally, the paired data acquisition unit 200 may include: a first face feature encoding acquisition subunit and a first face frontal image acquisition subunit.
[0127] The first face feature encoding subunit is used to input the first face side image into the face alignment network model, and use an image encoder to encode the first face side image to obtain the first face feature code corresponding to the first face side image.
[0128] The first face frontal image acquisition subunit is used to input the first face feature code into the face image generator to obtain the first face frontal image output by the face image generator.
[0129] Optionally, the face feature encoder training unit 400 may include: a second face feature encoding acquisition subunit, a third face feature encoding acquisition subunit, and a face feature encoder acquisition subunit.
[0130] The second face feature encoding subunit is used to input the first face side image from the first pair of data into the face feature encoder to obtain the second face feature code corresponding to the first face side image.
[0131] The third face feature encoding subunit is used to input the first face frontal image from the first pair of data into the face feature encoder to obtain the third face feature code corresponding to the first face frontal image.
[0132] The face feature encoder obtains a subunit, which is used to adjust the network weights of the face feature encoder using the second and third face feature codes to obtain a trained face feature encoder, which is used to perform face verification on the face image to be verified.
[0133] Optionally, the face alignment network model obtaining unit 100 may further include: a random geometric transformation subunit.
[0134] The random geometric transformation subunit is used to perform random geometric transformations on each second face profile image in the second face image dataset before the second training subunit trains the face alignment network model using the second face image dataset.
[0135] The face feature encoder training device disclosed herein can obtain a pre-trained face alignment network model, wherein the face alignment network model includes an image encoder and a face image generator. A first side-view image of a face is input into the face alignment network model, and a first frontal image of a face is generated using the image encoder and the face image generator. The first side-view image and the first frontal image of the face are used as a first pair of data. The face feature encoder is initialized using the network weights of the image encoder. The face feature encoder is trained using the first pair of data to obtain a trained face feature encoder, which is used for face verification of the face image to be verified. This disclosure generates a frontal image of a face corresponding to a side-view image through the face alignment network model, which can avoid the loss of face features, enabling the face feature encoder trained thereby to accurately recognize the side-view image of the face, thus improving the reliability of face verification.
[0136] Corresponding to the above-described face verification method, this disclosure also provides a face verification device, the structure of which is as follows: Figure 7 As shown, it may include: a face image acquisition unit 500 to be verified, a face feature encoding acquisition unit 600, and a verification result acquisition unit 700.
[0137] The face image acquisition unit 500 is used to acquire the face image to be verified.
[0138] The face feature encoding acquisition unit 600 is used to input the face image to be verified into a pre-trained face feature encoder to obtain the face feature code corresponding to the face image to be verified.
[0139] The face feature encoder is trained using any of the face feature encoder training devices provided in this disclosure.
[0140] The verification result acquisition unit 700 is used to perform face verification on the face image to be verified using face feature encoding, and obtain the face verification result.
[0141] The face verification device provided in this disclosure can obtain a face image to be verified, input the face image to be verified into a pre-trained face feature encoder, and obtain a face feature code corresponding to the face image to be verified. The face feature encoder is trained using any of the face feature encoder training methods provided in the embodiments of this disclosure. Face verification is performed on the face image to be verified using the face feature code to obtain the face verification result. This disclosure implements face verification of the face image to be verified through a face feature encoder, and can be applied to face verification scenarios including access control and online transactions. It also supports face verification using side-view images of the face, effectively reducing the attack risk of face spoofing technology, increasing the security and reliability of face verification, and improving the stability, convenience, and user-friendliness of face verification.
[0142] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0143] The face feature encoder training device includes a processor and a memory. The face alignment network model acquisition unit 100, the paired data acquisition unit 200, the face feature encoder initialization unit 300, and the face feature encoder training unit 400 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.
[0144] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured. By adjusting kernel parameters, a face alignment network model can generate a frontal face image corresponding to a side profile image, avoiding the loss of facial features. This allows the face feature encoder trained in this way to accurately recognize side profile images, improving the reliability of face verification.
[0145] This disclosure provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the face feature encoder training method.
[0146] This disclosure provides a processor for running a program, wherein the program executes the face feature encoder training method during runtime.
[0147] This disclosure provides an electronic device, which includes at least one processor, at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the above-described face feature encoder training method.
[0148] This disclosure also provides a computer program product that, when executed on an electronic device, is adapted to perform the steps of an initialization method for training a facial feature encoder.
[0149] The face verification device includes a processor and a memory. The face image acquisition unit 500, the face feature encoding acquisition unit 600, and the verification result acquisition unit 700 are all stored as program units in the memory. The processor executes the program units stored in the memory to realize the corresponding functions.
[0150] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, face verification of the face image to be verified can be achieved through a face feature encoder. This is applicable to face verification scenarios including access control and online transactions, and supports face verification using side-view images. This effectively reduces the attack risk of face spoofing techniques, increases the security and reliability of face verification, and improves its stability, convenience, and user-friendliness.
[0151] This disclosure provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the face verification method.
[0152] This disclosure provides a processor for running a program, wherein the program executes the face verification method during runtime.
[0153] This disclosure provides an electronic device, which includes at least one processor, at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the above-described face verification method.
[0154] This disclosure also provides a computer program product that, when executed on an electronic device, is adapted to perform steps for initializing a face verification method.
[0155] The electronic devices mentioned in this article can be servers, PCs, tablets, mobile phones, and access control machines, etc.
[0156] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, electronic devices (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0157] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.
[0158] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.
[0159] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0160] In the description of this disclosure, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure.
[0161] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0162] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0163] The above are merely embodiments of this disclosure and are not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.
Claims
1. A method for training a facial feature encoder, characterized in that, include: A pre-trained face alignment network model is obtained, wherein the face alignment network model includes an image encoder and a face image generator; The first side profile image of the face is input into the face alignment network model. The image encoder and the face image generator are used to generate the first frontal image of the face. The first side profile image and the first frontal image of the face are used as a first pair of data. The face feature encoder is initialized using the network weights of the image encoder; The face feature encoder is trained using the first pair of data to obtain a trained face feature encoder, which is used to perform face verification on the face image to be verified, so that the face feature encoding obtained by the trained face feature encoder for the side face image is close to the face feature encoding for the front face image.
2. The method according to claim 1, characterized in that, The process of obtaining the pre-trained face alignment network model includes: Obtain a pre-trained face image generator; In the face alignment network model, the face image generator is fixed, and the face alignment network model is initialized and trained using a first face image pair dataset. The network weights of the image encoder in the face alignment network model are adjusted. The first face image pair dataset includes at least one set of second paired data, and the second paired data includes two identical face images. The face alignment network model is trained using a second face image pair dataset, and the network weights of the image encoder in the face alignment network model are adjusted to obtain the trained face alignment network model. The second face image pair dataset includes at least one set of third paired data, and the third paired data includes a second face side image and its corresponding second face front image.
3. The method according to claim 1, characterized in that, The step of inputting the first side profile image of a face into the face alignment network model, and generating a first frontal image of a face using the image encoder and the face image generator, includes: The first side profile image of a face is input into the face alignment network model, and the image encoder is used to encode the first side profile image of the face to obtain the first face feature code corresponding to the first side profile image of the face. The first facial feature code is input into the face image generator to obtain the first frontal face image output by the face image generator.
4. The method according to claim 1, characterized in that, The step of training the face feature encoder using the first pairwise data to obtain the trained face feature encoder, which is used to perform face verification on the face image to be verified, includes: The first face profile image from the first pair of data is input into the face feature encoder to obtain the second face feature code corresponding to the first face profile image; The first frontal face image from the first pair of data is input into the face feature encoder to obtain the third face feature code corresponding to the first frontal face image; Using the second and third face feature codes, the network weights of the face feature encoder are adjusted to obtain the trained face feature encoder, which is used to perform face verification on the face image to be verified.
5. The method according to claim 2, characterized in that, Before training the face alignment network model using the second face image dataset, the method further includes: Random geometric transformations are performed on each of the second face profile images in the dataset.
6. A face verification method, characterized in that, include: Obtain the face image to be verified; The face image to be verified is input into a pre-trained face feature encoder to obtain a face feature code corresponding to the face image to be verified, wherein the face feature encoder is trained using the method described in any one of claims 1 to 5; The face feature encoding is used to perform face verification on the face image to be verified, and the face verification result is obtained.
7. A facial feature encoder training device, characterized in that, include: The system includes a face alignment network model acquisition unit, a pairwise data acquisition unit, a face feature encoder initialization unit, and a face feature encoder training unit. The face alignment network model obtaining unit is used to obtain a pre-trained face alignment network model, wherein the face alignment network model includes an image encoder and a face image generator; The paired data acquisition unit is used to input the first face side image into the face alignment network model, use the image encoder and the face image generator to generate the first face front image, and take the first face side image and the first face front image as a first pair of data. The face feature encoder initialization unit is used to initialize the face feature encoder using the network weights of the image encoder; The face feature encoder training unit is used to train the face feature encoder using the first pair of data to obtain the trained face feature encoder, which is used to perform face verification on the face image to be verified, so that the face feature encoding obtained by the trained face feature encoder for the side face image is close to the face feature encoding for the front face image.
8. A face verification device, characterized in that, include: The system comprises a face image acquisition unit, a face feature encoding acquisition unit, and a verification result acquisition unit. The face image acquisition unit is used to acquire the face image to be verified. The face feature encoding acquisition unit is used to input the face image to be verified into a pre-trained face feature encoder to obtain the face feature encoding corresponding to the face image to be verified, wherein the face feature encoder is trained using the device described in claim 7. The verification result obtaining unit is used to perform face verification on the face image to be verified using the face feature encoding, and obtain the face verification result.
9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the face feature encoder training method as described in any one of claims 1 to 5; and / or implements the face verification method as described in claim 6.
10. An electronic device, the electronic device comprising at least one processor, and at least one memory and a bus connected to the processor; wherein, The processor and the memory communicate with each other via the bus; The processor is used to call program instructions in the memory to execute the face feature encoder training method as described in any one of claims 1 to 5; and / or to execute the face verification method as described in claim 6.