Face recognition method, device, processing equipment and storage medium

Abstract

The embodiments of the present application provide a face recognition method, device, processing equipment and storage medium, which relate to the field of computer vision technology. By acquiring an initial acquisition image, the initial acquisition image includes: facial image information of a target face; positioning the initial acquisition image to obtain feature point information of the target face; determining the yaw angle of the target face according to the feature point information of the target face; if the yaw angle of the target face is greater than a preset angle threshold, determining that the angle of the target face is a side face; if the angle of the target face is a side face, generating a frontal face image corresponding to the initial acquisition image based on a pre-trained generative adversarial network; determining the identity information of the target face according to the frontal face image and a face image library, wherein the face image library stores multiple face images and the identity information of each face image. The influence of changes in face posture and deformation on identity recognition is weakened, and the efficiency and accuracy of face recognition are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a face recognition method, apparatus, processing device and storage medium. Background Art

[0002] Facial recognition technology is a biometric recognition technology. It is a method of identity recognition based on a person's facial feature information. It is widely used in scenarios such as access management, door access control and attendance, and intelligent alarm.

[0003] Existing face recognition technology generally detects and aligns the captured image taken by the camera to determine the face area, then extracts the feature vector of the face area, matches the captured image with the images stored in the target library, and identifies whether they are the same identity information.

[0004] However, since the existing target library is generally built based on frontal samples of the face, and the face may deform to varying degrees during movement, the captured image may only contain side face information, which leads to reduced accuracy of face recognition. Summary of the invention

[0005] The purpose of the present application includes, for example, providing a face recognition method, apparatus, processing equipment and storage medium, which can reconstruct a frontal face image based on the side face of the target face, so as to further determine the identity information of the target face, weaken the influence of changes in face posture and face deformation on face identity recognition, and improve the efficiency and accuracy of face recognition.

[0006] The embodiments of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a face recognition method, the method comprising:

[0008] Acquire an initial acquisition image, wherein the initial acquisition image includes: facial image information of a target face;

[0009] Performing positioning processing on the initial collected image to obtain feature point information of the target face;

[0010] Determining the yaw angle of the target face according to the feature point information of the target face;

[0011] If the yaw angle of the target face is greater than a preset angle threshold, determining that the angle of the target face is a side face;

[0012] If the angle of the target face is a side face, a frontal face image corresponding to the initial collected image is generated based on a pre-trained generative adversarial network;

[0013] The identity information of the target face is determined according to the frontal face image and a face image library, wherein the face image library stores a plurality of face images and the identity information of each face image.

[0014] In an optional implementation, the adversarial network includes: a target generation network, wherein the target generation network includes: an encoder and a decoder;

[0015] The generating a frontal face image corresponding to the initial collected image based on a pre-trained generative adversarial network comprises:

[0016] Cropping the initial collected image to obtain a face cropped image;

[0017] Inputting the cropped face image into the encoder to obtain an encoding vector;

[0018] generating random noise information and posture code information;

[0019] The coding vector, random noise information and posture code information are input into the decoder, and the decoder decodes the coding vector according to the random noise information and the posture code information to obtain the frontal face image, wherein the random noise information is used to characterize the change information of the face appearance, and the posture code information is used to highlight the facial identity information of the face and weaken the facial posture information.

[0020] In an optional embodiment, the method further includes:

[0021] Constructing a generative adversarial initial network, wherein the generative adversarial initial network includes: an initial discriminant network and an initial generative network;

[0022] Acquire a plurality of training images, wherein the plurality of training images include: image information of a plurality of training faces;

[0023] The generative adversarial initial network is trained using the multiple training images, and the initial discriminant network and the initial generative network are modified according to the training results to obtain the generative adversarial network.

[0024] In an optional implementation, the using the plurality of training images to train the generative adversarial initial network, and modifying the initial discriminant network and the initial generative network according to the training results to obtain the generative adversarial network, comprises:

[0025] generating random noise information and posture code information;

[0026] Inputting a first training image, posture code information, and random noise information into the initial generation network to obtain a first generated image;

[0027] Inputting the first training image and the first generated image into the initial discriminant network to obtain an image evaluation result of the first generated image;

[0028] According to the image evaluation result, the initial generation network and the initial discrimination network are modified to obtain a new generation adversarial initial network;

[0029] The above process is repeated until the image evaluation result reaches a preset threshold, and the new generative adversarial initial network is used as the generative adversarial network.

[0030] In an optional implementation, determining the identity information of the target face according to the frontal face image and the face image library includes:

[0031] Performing feature extraction on the frontal face image to obtain target features of the target face;

[0032] The target features are compared with the features of each face image in the face image library, and the identity information of the target face is determined according to the comparison result.

[0033] In an optional implementation, comparing the target feature with features of each face image in the face image library and determining the identity information of the target face according to the comparison result includes:

[0034] Calculating the similarity between the target feature and the features of each face image respectively to obtain multiple similarity scores;

[0035] Compare the multiple similarity scores with a preset score threshold respectively, and take the face image corresponding to the similarity score greater than the preset score threshold and with the largest value as the target comparison image;

[0036] The identity information of the target comparison image is used as the identity information of the target face.

[0037] In an optional implementation, positioning processing is performed on the initial collected image to obtain feature point information of the target face, including:

[0038] Inputting the initial collected image into a pre-trained convolutional neural network to obtain a plurality of three-dimensional feature points of the target face;

[0039] The feature point information of the target face is generated according to the multiple three-dimensional feature points.

[0040] In a second aspect, an embodiment of the present application provides a face recognition device, comprising:

[0041] An acquisition module is used to acquire an initial acquisition image, wherein the initial acquisition image includes: facial image information of a target face;

[0042] A positioning module, used to perform positioning processing on the initial collected image to obtain feature point information of the target face;

[0043] A processing module, which determines the yaw angle of the target face according to the feature point information of the target face; if the yaw angle of the target face is greater than a preset angle threshold, determines that the angle of the target face is a side face; if the angle of the target face is a side face, generates a frontal face image corresponding to the initial collected image based on a pre-trained generative adversarial network;

[0044] The determination module is used to determine the identity information of the target face according to the frontal face image and the face image library, wherein the face image library stores multiple face images and the identity information of each face image.

[0045] The processing module is also specifically used for: the adversarial network includes: a target generation network, the target generation network includes: an encoder and a decoder; cropping the initial captured image to obtain a cropped face image; inputting the cropped face image into the encoder to obtain a coding vector; generating random noise information and posture code information; inputting the coding vector, random noise information and posture code information into the decoder, and the decoder decodes the coding vector according to the random noise information and the posture code information to obtain the frontal face image, wherein the random noise information is used to characterize the change information of the face appearance, and the posture code information is used to highlight the facial identity information of the face and weaken the facial posture information.

[0046] A training module is used to construct a generative adversarial initial network, wherein the generative adversarial initial network includes: an initial discriminant network and an initial generative network; obtain multiple training images, wherein the multiple training images include: image information of multiple training faces; use the multiple training images to train the generative adversarial initial network, and modify the initial discriminant network and the initial generative network according to the training results to obtain the generative adversarial network.

[0047] The training module is also specifically used to generate random noise information and posture code information; input the first training image, posture code information, and random noise information into the initial generation network to obtain a first generated image; input the first training image and the first generated image into the initial discriminant network to obtain an image evaluation result of the first generated image; according to the image evaluation result, modify the initial generation network and the initial discriminant network to obtain a new generation adversarial initial network; repeat the above process until the image evaluation result reaches a preset threshold, and use the new generation adversarial initial network as the generation adversarial network.

[0048] The determination module is also specifically used to extract features from the frontal face image to obtain target features of the target face; compare the target features with features of each face image in the face image library, and determine the identity information of the target face based on the comparison results.

[0049] The determination module is also specifically used to calculate the similarity between the target features and the features of each facial image, respectively, to obtain multiple similarity scores; compare the multiple similarity scores with preset score thresholds, respectively, and use the facial image corresponding to the similarity score that is greater than the preset score threshold and has the largest value as the target comparison image; and use the identity information of the target comparison image as the identity information of the target face.

[0050] The positioning module is also specifically used to input the initial collected image into a pre-trained convolutional neural network to obtain multiple three-dimensional feature points of the target face; and generate feature point information of the target face based on the multiple three-dimensional feature points.

[0051] In a third aspect, an embodiment of the present application provides a processing device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, and when the processing device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the face recognition method as described in any one of the first aspects.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the face recognition method as described in any one of the first aspects are implemented.

[0053] The beneficial effects of the embodiments of the present application include:

[0054] By using the face recognition method, device, processing equipment and storage medium provided in the present application, it is possible to generate a corresponding frontal face image from an image of the target face with the angle of the face being side face based on a generative adversarial network, and then further determine the identity information of the target face based on this. In this way, through the step of generating the side face conversion, the relevant features of the face are supplemented, avoiding the inability to recognize the identity of the face due to the lack of facial features in the side face image, and improving the accuracy of identity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 This is a schematic diagram of the structure of the existing generative adversarial network;

[0057] Figure 2 A schematic diagram of a face recognition method according to an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the steps of the face recognition method provided in the embodiment of the present application;

[0059] Figure 4 A schematic diagram of the steps of generating feature point information in a face recognition method provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of three-dimensional facial feature points of a face recognition method provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the structure of a generative adversarial network for a face recognition method provided in an embodiment of the present application;

[0062] Figure 7 A schematic diagram of another step flow of the face recognition method provided in an embodiment of the present application;

[0063] Figure 8 A schematic diagram of another step flow of the face recognition method provided in an embodiment of the present application;

[0064] Fig. 9 A schematic diagram of another step flow of the face recognition method provided in an embodiment of the present application;

[0065] Fig.10 A schematic diagram of another step flow of the face recognition method provided in an embodiment of the present application;

[0066] Fig.11 A schematic diagram of another step flow of the face recognition method provided in an embodiment of the present application;

[0067] Fig.12 A schematic diagram of the structure of a face recognition device provided in an embodiment of the present application;

[0068] Fig.13 A schematic diagram of the structure of a processing device provided in an embodiment of the present application.

[0069] Icon: 101-generative model; 102-discriminative model; 401-initial acquired image; 402-cropped face image; 403-encoder; 404-random noise information; 405-encoded vector; 406-posture code information; 407-decoder; 408-frontal face image; 409-discriminator; 100-face recognition device; 1001-acquisition module; 1002-positioning module; 1003-processing module; 1004-determination module; 1005-training module; 2001-processor; 2002-memory. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0071] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0072] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0073] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0074] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0075] Face recognition technology is a biometric technology that identifies people based on their facial features. Face recognition-based identity recognition algorithms generally include face image detection, face image feature extraction, and matching and recognition steps.

[0076] First, after the collected facial image, usually an image containing a frontal face, is input, the algorithm will detect whether there is a face in the image. If so, the facial information such as the position and size of the face will be marked.

[0077] Then, image features are extracted from the image with the face, and feature modeling is performed on the face based on one or more of the pixel statistical features, algebraic features or visual features in the face image to extract feature data of the face.

[0078] Finally, the extracted facial feature data is searched and matched with the feature templates stored in the database, and the identity information of the collected face is confirmed based on the degree of similarity.

[0079] In the above process, the feature templates stored in the database are generally established based on frontal samples of the face. When the acquisition device captures and collects images, the face will undergo different degrees of deformation with changes in posture, so that the collected images may only contain large-pose side face images with a large rotation angle. This may cause the extracted facial feature data to be partially missing, affecting the accuracy of face recognition.

[0080] Based on this, the applicant has proposed a face recognition method, device, processing equipment and storage medium after research, which can reconstruct a large-gesture side face image into a frontal face image, determine the identity information of the target face based on the comparison results with the face image library, supplement the missing facial feature information, and improve the accuracy of face recognition.

[0081] Generative Adversarial Network (GAN) is a deep learning network. Figure 1 As shown, it generally includes a generative model 101 (Generative Model) and a discriminative model 102 (Discriminative Model). The generative model 101 is used to generate a generated set that looks natural and real and similar to the training set based on the input random noise, and the discriminative model 102 is used to determine whether a given instance is a natural real instance from the training set or a generated set artificially forged from the generative model 101. If it is determined to be data from the training set, the judgment result is true, otherwise, if it is determined to be data from the generated set, the judgment result is false.

[0082] In other words, the training process of the generative adversarial network can be regarded as a zero-sum game process. The objective function of the GAN network is as follows:

[0083]

[0084] In the above formula, D is the discriminant model 102, and G is the generative model 101. D(x) is the output of the discriminant model 102, and G(z) is the output of the generative model 101. From the above formula, it can be deduced that the loss function LOSS of the generative model 101 is: log(1-D(G(z))), and the loss function LOSS of the discriminant model 102 is -log D(x)+log(1-D(G(z))).

[0085] Then, during the training process, the training goal of the generation model 101 is to make D(G(z)) approach 1, so that the loss of G is reduced, and the generated data gradually approaches the real data. When the pseudo data of the generation set and the real data of the training set are difficult to distinguish in the discriminant model 102, the training process is completed.

[0086] It can be seen that the theoretical basis of GAN is generative adversarial network. Compared with the previous convolutional neural network, GAN has a stronger remodeling ability, especially in processing unstructured data, such as face images, and can infinitely approach the global optimal solution through continuous iteration of the generative model 101 and the discriminative model 102. Therefore, in this embodiment, the side face rotation problem of large postures is solved with the help of GAN.

[0087] like Figure 2 As shown, the initial captured image of the target face with a side face angle is positioned and processed, and the direction of the yaw angle is determined according to the feature point information, and then input into the target generation network in the pre-trained generative adversarial network. Then, the encoder of the target generation network encodes the initial captured image and represents it as a vector containing a low dimension, which is connected with random noise information and posture code information, and is input into the decoder as a fully connected layer. Through the deconvolution processing of the decoder, a frontal face image is reconstructed. Finally, the above-mentioned frontal face image is compared with the face images in the face image library in turn, and the face image corresponding to the similarity score greater than the preset score threshold and the largest similarity score is used as the target comparison image, and the identity information of the target comparison image is read to determine the identity information of the target face.

[0088] The following is an explanation of a face recognition method, apparatus, processing device, and storage medium provided in an embodiment of the present application in combination with a number of specific application examples.

[0089] Figure 3FIG. 1 is a schematic diagram of a step flow of a face recognition method provided by an embodiment of the present application. The execution subject of the method may be a processing device with computing processing capabilities, such as Figure 3 As shown, the method comprises the following steps:

[0090] S201, obtaining an initial acquisition image.

[0091] The initially collected image includes: facial image information of the target face.

[0092] The initial captured image may be a static image with the target face captured by a camera, or a video frame with the target face captured from a video captured by a camera.

[0093] The angle of the target face in the initial captured image can be a frontal face or a side face, and can include one target face or multiple people. The image including the target face is obtained by cropping, and this application does not limit this.

[0094] S202, performing positioning processing on the initial collected image to obtain feature point information of the target face.

[0095] Positioning processing is the process of preprocessing the initial collected image. Based on the detection of the human face in the initial collected image, it further determines the feature points of the facial features, such as the eyes, eyebrows, nose, mouth, and facial contour, through the texture features of the face and the position constraints between the various feature points.

[0096] S203, determining the yaw angle of the target face according to the feature point information of the target face.

[0097] Optionally, based on the coordinates corresponding to the feature point information of the target face, the posture angle of the target face, that is, the direction and angle of the face relative to the camera, can be calculated, thereby determining whether the target face is facing the camera sideways.

[0098] The posture of the face can be described by a combination of pitch angle, yaw angle and roll angle, where the pitch angle refers to the angle of rotation of the face around the X-axis, which is expressed as the pitch or tilt of the head; the yaw angle refers to the angle of rotation of the face around the Y-axis, which is expressed as the horizontal rotation of the head to the left or right; and the roll angle refers to the angle of rotation of the face around the Z-axis, which is expressed as the rotation of the head, that is, the horizontal inclination angle.

[0099] Optionally, a standard, forward three-dimensional model may be firstly established, and then rotated about the Y axis until it coincides with the feature point information of the target face, and the current rotation angle is used as the yaw angle of the target face.

[0100] S204: If the yaw angle of the target face is greater than a preset angle threshold, determine that the angle of the target face is a side face.

[0101] From the above description, it can be seen that the yaw angle is used to determine the horizontal distance that the target face rotates to the left or right. Therefore, when the yaw angle of the target face is greater than the preset angle threshold, the angle of the target face can be considered to be a side face. For example, the preset angle threshold can be 50 degrees.

[0102] It should be noted that when the angle of the target face to the camera exceeds the preset threshold, it is a large pose image. Conversely, when the angle is less than the preset threshold, it can be understood as a small pose image. The large pose image can be regarded as a side face image, and the small pose image as a front face image.

[0103] In addition, based on the feature point information of the target face, the processing device can also determine whether the target face has a horizontal inclination. Since the horizontal inclination is greater than a preset inclination threshold, such as 5 degrees, the feature point positioning error will increase due to the mismatch between the face and the template of the feature extraction model described below. Therefore, for the initial collected image with a large horizontal inclination, it is also necessary to rotate it to horizontal according to the feature point information of the target face.

[0104] Optionally, the horizontal inclination of the initial acquisition image can be calculated by the following formula:

[0105]

[0106] Among them, (x leye ,y leye ) is the coordinate of the feature point of the left outer corner of the target face, (x reye ,y reye ) is the feature point coordinate of the right outer corner of the target face. If the horizontal inclination angle θ is greater than the preset inclination angle threshold, the feature point coordinate of the right outer corner of the eye can be used as the rotation center to rotate the initial acquisition image so that the initial acquisition image reaches the horizontal level.

[0107] Optionally, the step of calculating the horizontal tilt angle θ may be performed before or after the above-mentioned determination of the angle of the target face. In addition, for the initial collected image whose horizontal tilt angle θ is less than a preset tilt angle threshold, no rotation processing may be performed on it.

[0108] S205: If the angle of the target face is a side face, a frontal face image corresponding to the initial collected image is generated based on a pre-trained generative adversarial network.

[0109] If, based on the feature point information of the target face, it is determined that the initial captured image is a side face image in a large posture, and the horizontal inclination angle θ is too large, the initial captured image is first rotated to the horizontal, and then a frontal face image corresponding to the initial captured image is generated based on the pre-trained generative adversarial network.

[0110] It is understandable that when the target face is in profile, the frontal face area cannot be recognized due to self-occlusion. Therefore, based on the non-rigid characteristics of the face and the mapping relationship between the profile and the frontal face, the profile image can be input into the generative adversarial network to convert it into a frontal face image and complete the relevant features of the missing face.

[0111] S206, determining the identity information of the target face according to the frontal face image and the face image library, wherein the face image library stores a plurality of face images and the identity information of each face image.

[0112] A frontal face image that has been rotated or transformed, or only one of the processes or no process is performed, and the frontal face image containing the frontal face is compared with multiple face images in the face image library in turn to determine whether the identity information of the target face is one of the identity information of each face image in the face image library.

[0113] In this embodiment, the initial collected image of the target face with a side face angle is converted into a frontal face image, and the identity information of the target face is further determined based on the face image library. The facial feature information missing due to self-occlusion is supplemented by the conversion, thereby improving the accuracy of face recognition.

[0114] Alternatively, if Figure 4 As shown, in the above step S202, the initial collected image is positioned and processed to obtain feature point information of the target face, which can be achieved by the following steps S301 to S302.

[0115] S301, inputting the initial collected image into a pre-trained convolutional neural network to obtain multiple three-dimensional feature points of the target face.

[0116] Optionally, the structure of the convolutional neural network may be a network formed by stacking four HourGlass hourglass networks, wherein the convolutional neural network is obtained by pre-training with multiple 2D annotated images.

[0117] Optionally, in order to increase the accuracy of the convolutional neural network in generating feature points for large-pose faces, image flipping, rotation, color jittering, scale noise, random occlusion and other procedures may be added during the training process. Of course, this is not limited to this.

[0118] The convolutional neural network can perform three-dimensional annotation on the input initial collected image to generate three-dimensional feature points of multiple faces. Each three-dimensional feature point can be used to describe the spatial position information of the facial features of the target person.

[0119] Figure 5The figure shows a schematic diagram of a plurality of three-dimensional feature points when the angle of the target face is a slightly upward side face. In addition, the number of three-dimensional feature points can be set according to the accuracy, and can be 68 for example.

[0120] S302, generating feature point information of a target face according to the multiple three-dimensional feature points.

[0121] Optionally, a set of multiple three-dimensional feature points may be used as feature point information of the target face to calculate information such as the yaw angle of the target face in the three-dimensional space.

[0122] In this embodiment, the feature point information of the target face is determined through the pre-trained convolutional neural network, which paves the way for accurately determining the yaw angle in the following embodiments.

[0123] Alternatively, if Figure 6 As shown, the above-mentioned adversarial network includes: a target generation network, and the target generation network includes: an encoder 403 and a decoder 407.

[0124] Among them, the goal of encoder 403 is to encode the input image into a low-dimensional short vector representation, which can be composed of a convolutional layer, a pooling layer and a BatchNormalization layer of a convolutional neural network. The convolutional layer is responsible for obtaining the local features of the image, the pooling layer downsamples the image and transmits the scale-invariant features to the next layer, and the BatchNormalization layer mainly normalizes the distribution of the image to accelerate learning.

[0125] The goal of decoder 407 is to use the input short vector to decode the structural information about the graph. The decoder upsamples the short vector and then performs convolution on the upsampled image to perfect the complete geometric shape of the features in the short vector, thereby compensating for the loss of details caused by the encoder shortening the image to a short vector.

[0126] like Figure 6 , Figure 7 As shown, in the above step S304, the frontal face image 408 corresponding to the initial collected image 401 is generated based on the pre-trained generative adversarial network, which can be implemented by the following steps S501 to S504.

[0127] S501 , cropping the initial collected image 401 to obtain a cropped face image 402 .

[0128] Optionally, the initial captured image 401 may contain some useless background information. To prevent this information from interfering with the subsequent recognition process, face recognition and frame selection can be performed on the initial captured image 401, the background information can be cropped, and the part with the target face can be retained as foreground information to obtain a face cropped image 402.

[0129] S502 , input the cropped face image 402 into the encoder 403 to obtain the encoding vector 405 .

[0130] Then, the face cropped image 402 is input into the encoder 403 to obtain a one-dimensional encoding vector 405. The one-dimensional encoding vector 405 includes the feature information of the main facial features of the target face in the face cropped image 402. For example, the elements of the encoding vector 405 may represent facial skin color, eyebrow position, eye size, posture information, etc.

[0131] S503, generating random noise information 404 and gesture code information 406.

[0132] The random noise information 404 is a one-dimensional vector randomly generated by a processing device and is used to model the variance of other quantities except identity information and posture information.

[0133] The posture code information 406 may be a preset one-dimensional ONE-HOT vector, which encodes N states by using an N-bit state register, each state has an independent register bit, and only one bit is valid at any time, wherein the valid bit is recorded as 1. In the embodiment of the present application, the posture code information 406 may be used to indicate that the posture information is a positive vector.

[0134] S504, input the coding vector 405, random noise information 404 and posture code information 406 into the decoder 407, and the decoder 407 decodes the coding vector 405 according to the random noise information 404 and the posture code information 406 to obtain a frontal face image 408.

[0135] The random noise information 404 is used to characterize the change information of the face appearance, and the posture code information 406 is used to highlight the facial identity information of the face and weaken the facial posture information.

[0136] The three one-dimensional vectors of the encoding vector 405, random noise information 404, and posture code information 406 obtained above are connected. Optionally, they can also be input into the decoder 407 after passing through the fully connected layer and the transposition process. The decoder 407 performs deconvolution processing. According to the feature information of the target face in the encoding vector 405 and the posture code information 406, the posture code in the encoding vector 405 is modified to frontal, and then the random noise information 404 is added to avoid the local optimum in the decoding mapping process, thereby generating a frontal face image 408 with a frontal posture.

[0137] In addition, after the generative adversarial network training is completed, the discriminator 409 does not judge the generated frontal face image 408.

[0138] In this embodiment, the generative adversarial network converts the side face in the initial captured image into a front face by replacing the posture code according to the posture code information, the encoding vector, and the random noise information. While the encoder retains the main features of the target face, it also changes the posture of the target face, and according to the content of the encoding vector, it supplements the feature information required for converting the side face into the front face, thereby improving the matching degree between the front face and the side face after the conversion.

[0139] Alternatively, if Figure 8 As shown, the face recognition method provided in the embodiment of the present application also includes the following steps, which are used to describe the generation process of the generative adversarial network.

[0140] S601, construct a generative adversarial initial network.

[0141] Among them, the generative adversarial initial network includes: an initial discriminant network and an initial generative network.

[0142] First, according to the goal of face posture conversion, a generative adversarial initial network including an initial generative network and an initial discriminative network can be constructed. The parameters of the generative adversarial initial network are all initial parameters. After training, the initial parameters are adapted and modified during the training process to obtain a generative adversarial network.

[0143] It is understandable that the initial discriminant network and the initial generative network are trained synchronously and progress together. If the image generated by the initial generative network passes the identification of the initial discriminant network, it will be rewarded. Otherwise, if it fails to pass the initial discriminant network, it will be punished. During the training process, if the initial discriminant network is updated too quickly, the initial generative network may not pass the identification. If the initial generative network is updated too quickly, the initial generative network may be rewarded for continuously generating data of poor quality.

[0144] S602, obtaining multiple training images.

[0145] The multiple training images include: image information of multiple training faces.

[0146] Optionally, the multiple training images may include frontal face image information of the training face, and may also include side face image information of the training face, and the frontal face image information and side face image information may be multiple groups of images, each group of images corresponding to the same face. Among them, the frontal face image information can be used as a real image sample to update the initial discriminant network, and the side face image information can be used as the input of the initial generation network. After the initial generation network generates an image based on it, the initial generation network is updated based on it.

[0147] S603, using multiple training images to train the generative adversarial initial network, and modifying the initial discriminant network and the initial generative network according to the training results to obtain a generative adversarial network.

[0148] Finally, using the above-mentioned multiple training images, according to the loss of the generated frontal face images and the real frontal face images, the initial parameters of the initial discriminant network and the initial generative network in the generative adversarial initial network are respectively corrected. When the generative adversarial initial network converges, the generative adversarial initial network is used as the generative adversarial network.

[0149] In this embodiment, the initial discriminant network and the initial generative network are corrected based on multiple training images, and supervised learning is used to avoid overfitting of the generative adversarial network, thereby achieving a better frontal face generation effect.

[0150] Alternatively, if Fig. 9 As shown, in the above step S603, the generative adversarial initial network is trained using multiple training images, and the initial discriminant network and the initial generative network are modified according to the training results to obtain a generative adversarial network, which can be implemented by the following steps S701 to S706.

[0151] S701, generate random noise information and posture code information.

[0152] The principle of generative adversarial networks is: first, during the training process, the underlying data distribution behind the faces in the original data set is learned; then, the initial generative network approximates this distribution to generate face images of new postures; the initial discriminative network corrects the initial generative network again by identifying the errors in the face images; in this game, the two are improved alternately.

[0153] In order to enable the initial generation network to generate different face images each time, random noise information randomly generated by the processing device needs to be added each time an image is generated to increase the randomness of the generated face image so that it can be closer to the distribution of real frontal faces during multiple training processes.

[0154] As described in the above embodiments, the posture code information can be a one-dimensional ONE-HOT vector, including multiple binary bits, used to represent different postures of the face, with only the position of the target posture being 1, and the other positions being 0. In the embodiment of the present application, the position where the facial posture information is positive can be set as the valid bit 1. It can be understood that if the image to be generated by the target face is a positive face, the posture code information can be used all the time. If the posture of the target face to generate the image is modified, new posture code information needs to be generated, and the position corresponding to the target posture is set to 1.

[0155] S702, input the first training image, posture code information, and random noise information into the initial generation network to obtain a first generated image.

[0156] Optionally, the first training image can be an image of a face in the face training set, which is obtained after horizontal rotation and cropping. After the first training image passes through the initial encoder of the initial generation network, an initial coding vector is obtained, which is input into the initial decoder in the initial generation network together with the above-mentioned posture code information and random noise information. According to the features in the initial coding vector, the side face image in the first training image is converted into a frontal face image as the first generated image.

[0157] S703: Input the first training image and the first generated image into an initial discriminant network to obtain an image evaluation result of the first generated image.

[0158] Next, the initial discriminant network calculates the difference between the first training image and the first generated image, and obtains the image evaluation result of the first generated image.

[0159] Optionally, the similarity between the first training image and the first generated image can be determined by using methods such as Euclidean distance and cosine distance of the faces of the two, thereby determining the image evaluation result. The specific calculation method is not limited in this application.

[0160] S704: According to the image evaluation result, the initial generation network and the initial discrimination network are modified to obtain a new generative adversarial initial network.

[0161] Optionally, based on the image evaluation results, the loss functions of the initial generation network and the initial discriminant network are calculated, and the parameters of the initial generation network and the initial discriminant network are updated in turn according to the losses to obtain a new initial generation network and a new initial discriminant network as a new generative adversarial initial network, wherein the specific loss function is not limited here.

[0162] It should be noted that in order to improve the stability of the training process, the initial generation network and the initial discrimination network may not be updated in the same sub-training. Optionally, the initial discrimination network may be updated once after the initial generation network is updated one or two times. The initial generation network and the initial discrimination network may also be updated synchronously or asynchronously in other ways, which is not limited in this application.

[0163] S705, repeat the above process until the image evaluation result reaches a preset threshold.

[0164] If the image evaluation result reaches a preset threshold, that is, the similarity between the first training image and the first generated image reaches the preset threshold, it can be considered that the current initial parameters have reached the target.

[0165] S706: Use the new generative adversarial initial network as a generative adversarial network.

[0166] The generative adversarial initial network whose initial parameters reach the target is used as the generative adversarial network to generate frontal face images.

[0167] It should be noted that, in the present embodiment, in the generative adversarial network, the frontal face image is generated and outputted only by the encoder and decoder in the trained generative network, without the need for the discriminator to discriminate again.

[0168] In this embodiment, the initial generative network and the initial discriminative network are cyclically updated by supervised learning, which improves the stability and efficiency of the training process and also improves the accuracy of the frontal faces generated by the trained generative adversarial network.

[0169] Alternatively, if Fig.10 As shown, in the above step S206, determining the identity information of the target face according to the frontal face image and the face image library may include the following steps.

[0170] S801, extracting features from the frontal face image to obtain target features of the target face.

[0171] A frontal face image is an image containing a high-dimensional feature vector. In order to improve the comparison efficiency in subsequent steps, the target features of the target face can be extracted to obtain a short vector.

[0172] The feature extraction method may be based on a histogram of oriented grids (HOG) method, may utilize a dlib library, or may be a convolutional neural network method, but is not limited thereto.

[0173] S802, comparing the target features with the features of each face image in the face image library, and determining the identity information of the target face according to the comparison result.

[0174] Optionally, the above-mentioned feature extraction method may be used to extract and store the features of each face image in the face image library in advance.

[0175] In this way, the target features extracted above are compared with the features of each pre-selected and stored face image in turn, and the identity information of the target face is determined based on the matching results with the target features of the target face and further based on the data in the face image library.

[0176] In this embodiment, the features of the target face are extracted and then cyclically compared with the features of each face image in the face image library, thereby improving the efficiency of face recognition.

[0177] Alternatively, if Fig.11As shown, in the above step S802, the target features are compared with the features of each face image in the face image library, and the identity information of the target face is determined according to the comparison result, which can be achieved by the following steps S901 to S903.

[0178] S901, respectively calculating the similarity between the target feature and the feature of each face image to obtain a plurality of similarity scores.

[0179] In order to determine the similarity between the target face and each face in the face image library, the distance between the target feature and the feature of each face image can be calculated as multiple similarity scores to evaluate the similarity. It should be noted that the distance and the similarity score are inversely proportional. The closer the distance, the more similar the two are, and the higher the similarity score. In this embodiment, the specific mapping formula of the distance and the similarity score is not specifically limited.

[0180] Optionally, the distance calculated above may be one or more of cosine distance, Minkowski distance, Mahalanobis distance, and Euclidean distance, but is certainly not limited thereto.

[0181] S902, comparing the multiple similarity scores with the preset score threshold respectively, and taking the face image corresponding to the similarity score greater than the preset score threshold and with the largest value as the target comparison image.

[0182] Furthermore, the multiple similarity scores calculated in the above steps are compared with a preset score threshold. Exemplarily, if only one similarity score is greater than the preset score threshold, the face image corresponding to the similarity score is used as the target comparison image, and if multiple similarity scores are all greater than the preset score threshold, the face image corresponding to the largest similarity score is used as the target comparison image. Conversely, if there is no similarity score that exceeds the preset score threshold, the identity information of the target face cannot be determined.

[0183] It is understandable that if different distance calculation methods are used, the preset scoring thresholds may also be different.

[0184] S903: Use the identity information of the target comparison image as the identity information of the target face.

[0185] The face image library can store the identity information of each face in a one-to-one correspondence with the target comparison image. Therefore, after determining one of the images in the face image library as the target comparison image, the processing device can read its corresponding identity information and output the identity information as the identity information of the target face.

[0186] In this embodiment, the identity information of the target face is determined efficiently and accurately by cyclically comparing the target face with the face image library.

[0187] See also Fig.12The embodiment of the present application further provides a face recognition device 100, comprising:

[0188] The acquisition module 1001 is used to acquire an initial acquisition image, where the initial acquisition image includes: facial image information of a target face;

[0189] A positioning module 1002 is used to perform positioning processing on the initial collected image to obtain feature point information of the target face;

[0190] The processing module 1003 is used to determine the yaw angle of the target face according to the feature point information of the target face; if the yaw angle of the target face is greater than a preset angle threshold, it is determined that the angle of the target face is a side face; if the angle of the target face is a side face, a frontal face image corresponding to the initial collected image is generated based on the pre-trained generative adversarial network;

[0191] The determination module 1004 is used to determine the identity information of the target face according to the frontal face image and the face image library, in which a plurality of face images and the identity information of each face image are stored.

[0192] The processing module 1003 is also specifically used for: the adversarial network includes: a target generation network, the target generation network includes: an encoder and a decoder; cropping the initial captured image to obtain a cropped face image; inputting the cropped face image into the encoder to obtain a coding vector; generating random noise information and posture code information; inputting the coding vector, the random noise information and the posture code information into the decoder, and the decoder decodes the coding vector according to the random noise information and the posture code information to obtain a frontal face image, wherein the random noise information is used to characterize the change information of the face appearance, and the posture code information is used to highlight the facial identity information of the face and weaken the facial posture information.

[0193] The training module 1005 is used to construct a generative adversarial initial network, which includes: an initial discriminant network and an initial generative network; obtain multiple training images, which include: image information of multiple training faces; use the multiple training images to train the generative adversarial initial network, and modify the initial discriminant network and the initial generative network according to the training results to obtain a generative adversarial network.

[0194] The training module 1005 is also specifically used to generate random noise information and posture code information; input the first training image, posture code information, and random noise information into the initial generation network to obtain a first generated image; input the first training image and the first generated image into the initial discriminant network to obtain an image evaluation result of the first generated image; according to the image evaluation result, correct the initial generation network and the initial discriminant network to obtain a new generative adversarial initial network; repeat the above process until the image evaluation result reaches a preset threshold, and use the new generative adversarial initial network as the generative adversarial network.

[0195] The determination module 1004 is also specifically used to extract features from the frontal face image to obtain target features of the target face; compare the target features with features of each face image in the face image library, and determine the identity information of the target face based on the comparison results.

[0196] Determination module 1004 is also specifically used to calculate the similarity between the target features and the features of each facial image, respectively, to obtain multiple similarity scores; compare the multiple similarity scores with preset score thresholds, respectively, and use the facial image corresponding to the similarity score that is greater than the preset score threshold and has the largest value as the target comparison image; and use the identity information of the target comparison image as the identity information of the target face.

[0197] The positioning module 1002 is also specifically used to input the initial collected image into a pre-trained convolutional neural network to obtain multiple three-dimensional feature points of the target face; and generate feature point information of the target face based on the multiple three-dimensional feature points.

[0198] See also Fig.13 This embodiment also provides a processing device, which includes: a processor 2001, a memory 2002 and a bus. The memory 2002 stores machine-readable instructions executable by the processor 2001. When the processing device is running, the above machine-readable instructions are executed. The processor 2001 communicates with the memory 2002 through the bus. The processor 2001 is used to execute the steps of the face recognition method in the above embodiment.

[0199] The memory 2002, the processor 2001 and the bus components are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The data processing device of the face recognition system includes at least one software function module that can be stored in the memory 2002 in the form of software or firmware or solidified in the operating system (OS) of the processing device. The processor 2001 is used to execute the executable modules stored in the memory 2002, such as the software function modules and computer programs included in the data processing device of the face recognition system.

[0200] Among them, the memory 2002 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0201] Optionally, the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method embodiment are executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0202] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0203] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0204] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A face recognition method, characterized in that: The method comprises: Acquire an initial acquisition image, wherein the initial acquisition image includes: facial image information of a target face; Performing positioning processing on the initial collected image to obtain feature point information of the target face; Determine the yaw angle of the target face according to the feature point information of the target face; wherein the yaw angle is used to determine the distance that the target face rotates horizontally to the left or right; If the yaw angle of the target face is greater than a preset angle threshold, determining that the angle of the target face is a side face; If the angle of the target face is a side face, a frontal face image corresponding to the initial collected image is generated based on a pre-trained generative adversarial network; the adversarial network includes: a target generation network, and the target generation network includes: an encoder and a decoder; Determining the identity information of the target face according to the frontal face image and a face image library, wherein the face image library stores a plurality of face images and the identity information of each face image; The generating a frontal face image corresponding to the initial collected image based on a pre-trained generative adversarial network comprises: Cropping the initial collected image to obtain a face cropped image; Inputting the cropped face image into the encoder to obtain an encoding vector; Generate random noise information and posture code information; The coding vector, random noise information and posture code information are input into the decoder, and the decoder decodes the coding vector according to the random noise information and the posture code information to obtain the frontal face image, wherein the random noise information is used to characterize the change information of the face appearance, and the posture code information is used to highlight the facial identity information of the face and weaken the facial posture information.

2. The face recognition method according to claim 1, characterized in that: The method further comprises: Constructing a generative adversarial initial network, wherein the generative adversarial initial network includes: an initial discriminant network and an initial generative network; Acquire a plurality of training images, wherein the plurality of training images include: image information of a plurality of training faces; The generative adversarial initial network is trained using the multiple training images, and the initial discriminant network and the initial generative network are modified according to the training results to obtain the generative adversarial network.

3. The face recognition method according to claim 2, characterized in that: The method of training the generative adversarial initial network using the plurality of training images and correcting the initial discriminant network and the initial generative network according to the training results to obtain the generative adversarial network includes: Generate random noise information and posture code information; Inputting a first training image, posture code information, and random noise information into the initial generation network to obtain a first generated image; Inputting the first training image and the first generated image into the initial discriminant network to obtain an image evaluation result of the first generated image; According to the image evaluation result, the initial generation network and the initial discrimination network are modified to obtain a new generation adversarial initial network; The above process is repeated until the image evaluation result reaches a preset threshold, and the new generative adversarial initial network is used as the generative adversarial network.

4. The face recognition method according to claim 1, characterized in that: The step of determining the identity information of the target face according to the frontal face image and the face image library includes: Performing feature extraction on the frontal face image to obtain target features of the target face; The target features are compared with the features of each face image in the face image library, and the identity information of the target face is determined according to the comparison result.

5. The face recognition method according to claim 4, characterized in that: The step of comparing the target feature with the features of each face image in the face image library and determining the identity information of the target face according to the comparison result includes: Calculating the similarity between the target feature and the features of each face image respectively to obtain multiple similarity scores; Compare the multiple similarity scores with a preset score threshold respectively, and take the face image corresponding to the similarity score greater than the preset score threshold and with the largest value as the target comparison image; The identity information of the target comparison image is used as the identity information of the target face.

6. The face recognition method according to claim 1, characterized in that: The performing positioning processing on the initial collected image to obtain feature point information of the target face includes: Inputting the initial collected image into a pre-trained convolutional neural network to obtain a plurality of three-dimensional feature points of the target face; The feature point information of the target face is generated according to the multiple three-dimensional feature points.

7. A face recognition device, characterized in that: include: An acquisition module is used to acquire an initial acquisition image, wherein the initial acquisition image includes: facial image information of a target face; A positioning module, used to perform positioning processing on the initial collected image to obtain feature point information of the target face; A processing module, used to determine the yaw angle of the target face according to the feature point information of the target face; wherein the yaw angle is used to determine the distance that the target face rotates horizontally to the left or right; if the yaw angle of the target face is greater than a preset angle threshold, it is determined that the angle of the target face is a side face; if the angle of the target face is a side face, a frontal face image corresponding to the initial collected image is generated based on a pre-trained generative adversarial network; the adversarial network includes: a target generation network, and the target generation network includes: an encoder and a decoder; A determination module, used to determine the identity information of the target face according to the frontal face image and a face image library, wherein the face image library stores a plurality of face images and the identity information of each face image; The processing module is further used to crop the initial collected image to obtain a cropped face image; input the cropped face image into the encoder to obtain a coding vector; generate random noise information and posture code information; input the coding vector, random noise information and posture code information into the decoder, and the decoder decodes the coding vector according to the random noise information and the posture code information to obtain the frontal face image, wherein the random noise information is used to characterize the change information of the face appearance, and the posture code information is used to highlight the facial identity information of the face and weaken the facial posture information.

8. A processing device, characterized in that: The processing device includes: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, and when the processing device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the face recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the face recognition method according to any one of claims 1 to 6 are implemented.

Images