Face recognition-based identity verification processing method and device

Abstract

The application provides a face recognition-based identity authentication processing method and device, relates to the technical field of deep learning, and can be applied to the financial field or other technical fields.The method comprises the following steps: obtaining a face image to be verified, performing feature extraction on the face image to be verified, and fine-tuning a face detection frame according to the extracted features; the face image to be verified is an image containing a face of a user who has applied makeup; makeup features are extracted according to the fine-tuned face detection frame and the extracted features, and the makeup features are separated to obtain a bare-faced image; the bare-faced image is compared with a reference face image in terms of image similarity to obtain a face recognition-based identity authentication result.The device executes the above method.The face recognition-based identity authentication processing method and device provided in the embodiment of the application can improve the accuracy of makeup face recognition, and thus improve the efficiency of face-based identity authentication.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and specifically to an identity verification processing method and apparatus based on face recognition. Background Technology

[0002] Identity verification is a necessary part of the banking process. For ordinary transactions, customers typically verify their identity by entering a password. However, complex transactions or transactions involving large sums of money often require facial recognition to verify the customer's identity. Facial recognition involves comparing the facial features in two images to determine the degree of similarity and generating a facial similarity score. If the score exceeds a certain threshold, it is determined that the two people are the same person, and the transaction can proceed. If the score does not exceed the threshold, it is determined that they are not the same person, and the transaction cannot proceed.

[0003] In real life, when customers go to the bank to conduct business, they usually wear makeup, accessories, or change their hairstyles when taking photos. Makeup, accessories, and hairstyles can obscure the face to varying degrees, resulting in a lower similarity score compared to photos used as a comparison benchmark, which does not match the actual situation and affects identity verification.

[0004] The bank's current identity verification process compares a customer's on-site photo with their ID card photo. If the similarity is 90% or higher, the customer is considered the same person and verification is successful, allowing the bank to continue processing the transaction. If the similarity does not meet the standard, verification fails, and the customer needs to provide other supporting documents (such as a medical insurance card, driver's license, etc.) to prove their identity. If the customer fails the facial comparison and does not have other supporting identification documents, identity verification is difficult to complete, and the bank cannot continue processing the transaction.

[0005] Existing generative face comparison and face recognition methods mostly focus on generating clear faces from a frontal angle for face images at extreme angles or blurry face images, and then feeding the generated face images into a discriminator for judgment. Although generative networks are also applicable to faces with makeup, they do not specifically consider the impact of makeup on face comparison and recognition, and the recognition effect on faces with makeup is not good. Summary of the Invention

[0006] To address the problems in the prior art, embodiments of the present invention provide a face recognition-based identity verification processing method and apparatus, which can at least partially solve the problems existing in the prior art.

[0007] On one hand, this invention proposes an identity verification processing method based on face recognition, comprising:

[0008] A face image to be verified is obtained; features are extracted from the face image; and the face detection box is fine-tuned based on the extracted features. The face image to be verified is an image containing the face of a user who has applied makeup.

[0009] Based on the finely adjusted face detection box and the extracted features, makeup features are extracted and separated to obtain a bare face image;

[0010] The image similarity of the unedited face image is compared with that of a reference face image to obtain the identity verification result based on face recognition.

[0011] The feature extraction based on the face image to be verified includes:

[0012] Extract the shallow and deep facial features of each layer of the face image to be verified;

[0013] The deep facial features are added to the shallow facial features of each high-level face to obtain the high-level fusion features; wherein, the shallow facial features of each high-level face are the shallow facial features of each layer retained after removing the shallow facial features of the first layer.

[0014] The highest-level fusion feature among the high-level fusion features is extracted again to obtain the deep fusion features of each high-level feature.

[0015] The step of fine-tuning the face detection box based on the extracted features includes:

[0016] Based on the aforementioned high-level deep fusion features and the preset face detection box fine-tuning model, the coordinates of the four endpoints of the face detection box are numerically adjusted to obtain the coordinate adjustment values ​​corresponding to the four endpoints respectively.

[0017] The preset face detection box fine-tuning model includes three convolutional layers and one fully connected layer connected in series.

[0018] The face detection box is fine-tuned based on the coordinate adjustment values ​​to obtain the fine-tuned face detection box.

[0019] The step of adjusting the coordinates of the four endpoints of the face detection box based on the deep fusion features of each high-level layer and the preset face detection box fine-tuning model includes:

[0020] The face detection box is used to extract corresponding feature segments from each high-level deep fusion feature;

[0021] The coordinate adjustment values ​​of each output by the fine-tuning model based on the preset face detection box are determined.

[0022] The step of extracting makeup features based on the fine-tuned face detection bounding box and the extracted features, and separating the makeup features to obtain a bare-faced face image, includes:

[0023] The finely tuned face detection bounding box is used to extract target feature fragments from the extracted features; the target feature fragments include each high-level deep fusion feature and each layer of shallow face features;

[0024] Based on the preset makeup separation model, the image resolution of each high-level deep fusion feature and each layer of face shallow features is adjusted, and the two types of features after resolution adjustment are stitched together.

[0025] The feature size and number of channels of the spliced ​​features are adjusted to obtain intermediate features. Two depthwise separable convolution operations are performed on the intermediate features to obtain attention weights. The attention weights are multiplied by the intermediate features to obtain the makeup layer features.

[0026] Subtract the makeup layer features from the intermediate features, and perform feature resolution enhancement and channel dimension reduction operations on the subtraction result to obtain a bare face image;

[0027] The preset makeup separation model includes an upsampling model unit and a makeup separation model unit.

[0028] The upsampling model unit includes a first channel and a second channel connected in parallel;

[0029] The first channel includes two cascaded transposed convolutional layers, used to adjust the resolution of the deep fusion feature images of each high layer;

[0030] The second channel includes a transposed convolutional layer, which is used to adjust the image resolution of the shallow facial features in each layer;

[0031] Two cascaded transposed convolutional layers are included at the ends of the first and second channels to adjust the feature size and number of channels of the spliced ​​features to obtain the intermediate features;

[0032] The makeup separation model unit includes two cascaded depthwise separable convolutional layers used to obtain makeup layer features;

[0033] After obtaining the subtraction result, the upsampling model unit also includes three cascaded transposed convolutional layers to improve feature resolution and reduce channel dimension, thereby obtaining a bare face image.

[0034] The face recognition-based identity verification method further includes:

[0035] The preset face detection box fine-tuning model is pre-trained using a first loss function;

[0036] The first loss function is the smoothing loss function L1.

[0037] The face recognition-based identity verification method further includes:

[0038] The preset makeup separation model is pre-trained using a second loss function;

[0039] The second loss function is obtained by adding the perceptual loss function and the mean squared error function.

[0040] On one hand, the present invention proposes an identity verification processing device based on face recognition, comprising:

[0041] The acquisition unit is used to acquire a face image to be verified, extract features from the face image to be verified, and fine-tune the face detection box based on the extracted features; the face image to be verified is an image containing the face of a user who has applied makeup.

[0042] The separation unit is used to extract makeup features based on the fine-tuned face detection box and the extracted features, and to separate the makeup features to obtain a bare face image;

[0043] The verification unit is used to compare the unedited face image with the reference face image to obtain the identity verification result based on face recognition.

[0044] In another aspect, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a bus, wherein,

[0045] The processor and the memory communicate with each other via the bus;

[0046] The memory stores program instructions that can be executed by the processor, and the processor can execute the following methods by calling the program instructions:

[0047] A face image to be verified is obtained; features are extracted from the face image; and the face detection box is fine-tuned based on the extracted features. The face image to be verified is an image containing the face of a user who has applied makeup.

[0048] Based on the finely adjusted face detection box and the extracted features, makeup features are extracted and separated to obtain a bare face image;

[0049] The image similarity of the unedited face image is compared with that of a reference face image to obtain the identity verification result based on face recognition.

[0050] This invention provides a non-transitory computer-readable storage medium, comprising:

[0051] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the following methods:

[0052] A face image to be verified is obtained; features are extracted from the face image; and the face detection box is fine-tuned based on the extracted features. The face image to be verified is an image containing the face of a user who has applied makeup.

[0053] Based on the finely adjusted face detection box and the extracted features, makeup features are extracted and separated to obtain a bare face image;

[0054] The image similarity of the unedited face image is compared with that of a reference face image to obtain the identity verification result based on face recognition.

[0055] The present invention provides a face recognition-based identity verification processing method and apparatus, which acquires a face image to be verified, extracts features from the face image, and fine-tunes a face detection box based on the extracted features. The face image to be verified is an image containing the face of a user with makeup. Makeup features are extracted based on the fine-tuned face detection box and the extracted features, and the makeup features are separated to obtain a bare-faced face image. The bare-faced face image is compared with a reference face image to obtain an identity verification result based on face recognition. This method can improve the accuracy of face recognition with makeup, thereby improving the efficiency of face-based identity verification. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0057] Figure 1 This is a flowchart illustrating an identity verification processing method based on face recognition provided in an embodiment of the present invention.

[0058] Figure 2 This is a flowchart illustrating an identity verification method based on face recognition provided in another embodiment of the present invention.

[0059] Figure 3 This is an overall structural diagram of the prediction stage model provided in an embodiment of the present invention.

[0060] Figure 4 This is a structural diagram of the prediction stage Adjust provided in an embodiment of the present invention.

[0061] Figure 5 This is a structural diagram of the prediction stage Fack photo generator provided in an embodiment of the present invention.

[0062] Figure 6 This is a structural diagram of the Discriminator in the prediction stage provided in an embodiment of the present invention.

[0063] Figure 7 This is a diagram showing the overall structure of the training phase model provided in an embodiment of the present invention.

[0064] Figure 8 This is a training phase Adjust structure diagram provided in an embodiment of the present invention.

[0065] Figure 9 This is a structural diagram of the Fack photo generator during the training phase provided in an embodiment of the present invention.

[0066] Figure 10 This is a diagram of the Discriminator structure for the training phase provided in an embodiment of the present invention.

[0067] Figure 11 This is a schematic diagram of the structure of an identity verification processing device based on face recognition provided in an embodiment of the present invention.

[0068] Figure 12 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0070] Explanation of relevant terms:

[0071] ResNet-101: A backbone network used for feature extraction.

[0072] FPN: Feature Pyramid Network, used to integrate deep semantic information from neural networks into shallow features.

[0073] RPN: Region Proposal Network, used to generate candidate boxes in detection tasks.

[0074] RoiAlign: An algorithm for extracting feature regions from bounding boxes.

[0075] Figure 1 This is a flowchart illustrating an identity verification method based on face recognition provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the identity verification processing method based on face recognition provided in this embodiment of the invention includes:

[0076] Step S1: Obtain the face image to be verified, extract features from the face image to be verified, and fine-tune the face detection box based on the extracted features; the face image to be verified is an image containing the face of a user who has put on makeup.

[0077] Step S2: Extract makeup features based on the fine-tuned face detection box and the extracted features, and separate the makeup features to obtain a bare face image.

[0078] Step S3: Compare the image similarity between the unedited face image and the reference face image to obtain the identity verification result based on face recognition.

[0079] In step S1 above, the device acquires a face image to be verified, extracts features from the face image, and fine-tunes the face detection box based on the extracted features; the face image to be verified is an image containing the face of a user who has applied makeup. The device can be a computer device that performs this method, such as a server. It should be noted that the acquisition and analysis of data involved in this embodiment of the invention are authorized by the client.

[0080] like Figure 2 As shown, the overall process of the identity verification method based on face recognition provided in this embodiment of the invention is described below:

[0081] Bank employees use high-speed scanners to take facial photos of customers, preprocess the images including resizing, extract features from the backbone network, and use facial contour, texture, and semantic features for face detection. The detected faces are then "removed of makeup" to generate bare-faced images, which are then compared with bare-faced ID photos to obtain a similarity score.

[0082] like Figure 3 As shown, the prediction stage of the overall model is explained as follows: Before extracting features from the face image to be verified, the face image can be preprocessed first, and then features can be extracted, such as... Figure 3 As shown, C1-C5 correspond to the shallow facial features respectively, and C2 is the layer above C1. Each shallow facial feature corresponds to three values. Taking C1 as an example, the first two values ​​represent the image resolution of 28×28, and the last value represents the number of channels of 512.

[0083] The feature extraction based on the face image to be verified includes:

[0084] Extract the shallow and deep facial features of the face image to be verified; such as... Figure 3 As shown, the shallow facial features of each layer include C1-C5, while the deep facial features are semantic features, such as the semantic descriptions of specific parts of the face, including the eyes, nose, and mouth.

[0085] The deep facial features are added to the shallow facial features of each higher-level face layer to obtain the fused features of each higher-level face layer; wherein, the shallow facial features of each higher-level face layer are the shallow facial features of each layer retained after removing the shallow facial features of the first layer; for example... Figure 3 As shown, the first layer of face shallow features C1 is removed, and the remaining high-level face shallow features are C2-C5. The deep face features are added to C2-C5 to obtain the high-level fusion features P2-P5.

[0086] The highest-level fusion feature among the high-level fusion features is extracted again to obtain the high-level deep fusion features. The highest-level fusion feature among the high-level fusion features is P5. P5 is extracted again to obtain P6. The high-level deep fusion features are P2-P6.

[0087] It should be noted that in actual business transactions, when bank employees verify the identity of customers, there is only one customer currently conducting business within the shooting range. There is only one image in the picture, and its position is relatively centered. For face detection, after obtaining the initial face detection frame, the second adjustment only requires fine-tuning the coordinates of the four corners of the detection frame. AdjustII adds an additional adjustment to the face detection frame to improve the accuracy of face detection.

[0088] The step of fine-tuning the face detection box based on the extracted features includes:

[0089] Based on the aforementioned high-level deep fusion features and the preset face detection bounding box fine-tuning model, the coordinates of the four endpoints of the face detection bounding box are numerically adjusted to obtain the coordinate adjustment values ​​corresponding to the four endpoints respectively; for example... Figure 4 As shown, each high-level deep fusion feature is represented by Pn, P2-P6.

[0090] The step of adjusting the coordinates of the four endpoints of the face detection box based on the deep fusion features of each high layer and the preset face detection box fine-tuning model includes:

[0091] The face detection bounding box is used to extract corresponding feature segments from each high-level deep fusion feature; the initial face detection bounding box RoIs generated by RPN is used to extract corresponding feature segments from the corresponding output features P2-P6 of FPN, and the extraction method is not limited to RoiAlign.

[0092] The coordinate adjustment values ​​are based on the output of the preset face detection box fine-tuning model. The feature fragment is input into the preset face detection box fine-tuning model, and the output of the preset face detection box fine-tuning model is used as the coordinate adjustment values. Since the face detection box is a rectangle with four endpoints, each coordinate adjustment value is specifically the coordinate adjustment value corresponding to the four endpoints of the rectangle.

[0093] The preset face detection box fine-tuning model includes three convolutional layers and one fully connected layer connected in series. It should be noted that the convolutional layers and subsequent transposed convolutional layers in this embodiment of the invention are all followed by a ReLU function. This ReLU function corresponds one-to-one with each convolutional layer or transposed convolutional layer and is used to increase the nonlinear relationship between the layers of the neural network.

[0094] like Figure 4 As shown, the first of these three convolutional layers is a 3x3 convolutional layer with 512 channels, used to adjust the image resolution and number of channels.

[0095] The second of these three convolutional layers is a 14x14 convolutional layer with 1024 channels, which is also used to adjust the image resolution and number of channels.

[0096] The third of these three convolutional layers is a 1x1 convolutional layer with 1024 channels. By using three consecutive convolutional layers, it is possible to smoothly adjust the image resolution and the number of channels.

[0097] The function of the fully connected layer is to output the adjustment values ​​of each coordinate in a 1x1 matrix, namely two adjustment values ​​in the horizontal direction and two adjustment values ​​in the vertical direction.

[0098] The face detection bounding box is fine-tuned based on the coordinate adjustment values ​​to obtain the fine-tuned face detection bounding box. That is, the face detection bounding box is fine-tuned by adjusting the coordinate positions corresponding to the four endpoints.

[0099] In step S2 above, the device extracts makeup features based on the fine-tuned face detection box and the extracted features, and separates the makeup features to obtain a bare-faced face image. The step of extracting makeup features based on the fine-tuned face detection box and the extracted features, and separating the makeup features to obtain a bare-faced face image includes:

[0100] The fine-tuned face detection bounding box is used to extract target feature segments from the extracted features; the target feature segments include high-level deep fusion features and shallow face features from each layer; such as... Figure 5 As shown, the preset makeup separation model is explained below:

[0101] The preset makeup separation model includes an upsampling model unit (corresponding to...) Figure 5Upsampling in the model and makeup separation model unit (corresponding to) Figure 5 The Makeup unit in the model is mainly responsible for restoring the resolution, while the Makeup Separation unit is mainly responsible for extracting the features of the makeup layer and guiding feature extraction through an attention mechanism.

[0102] The upsampling model unit includes a first channel and a second channel connected in parallel;

[0103] The first channel includes two cascaded transposed convolutional layers, which are used to adjust the resolution of each high-level deep fusion feature image; the two transposed convolutional layers in the first channel adjust the resolution to 28×28 and the number of channels to 512.

[0104] The second channel includes a transposed convolutional layer, which is used to adjust the image resolution of the shallow facial features in each layer; a transposed convolutional layer in the second channel adjusts the resolution to 28×28 and the number of channels to 512.

[0105] At the ends of the first and second channels are two cascaded transposed convolutional layers used to adjust the feature size and number of channels of the spliced ​​features to obtain the intermediate features; splicing corresponds to Figure 5 The concatenation method in the image produces a feature with a resolution of 28×28 and 1024 channels. After two adjustments, the intermediate feature is obtained with a resolution of 112×112 and 256 channels.

[0106] The makeup separation model unit includes two cascaded depthwise separable convolutional layers for obtaining makeup layer features. The first depthwise separable convolutional layer performs a depthwise convolution operation and ReLU activation on the intermediate features. The depthwise convolution operation allows the features to avoid inter-channel information interaction and only target independent channel features.

[0107] The second depthwise separable convolutional layer performs a depthwise convolution operation and a sigmoid function activation on the result of the first depthwise separable operation to generate attention weights. The attention weights are multiplied by the intermediate features to obtain the makeup layer features. Then, the makeup layer features are subtracted from the intermediate features to obtain the subtraction result.

[0108] After obtaining the subtraction result, the upsampling model unit further includes three cascaded transposed convolutional layers to improve feature resolution and reduce channel dimensions, thereby obtaining a bare-face image. Figure 5 As shown, adjusting the number of channels to 3 yielded a bare-faced image, i.e., a face photo.

[0109] Preset makeup separation model Figure 5 The Fack photo generator in the image uses a finely tuned face detection box to extract shallow face features C1-C5 and deep fusion features P2-P6 from the extracted features.

[0110] Based on the preset makeup separation model, the image resolution of each high-level deep fusion feature and each layer of facial shallow features is adjusted, and the two types of features after resolution adjustment are stitched together; please refer to the above instructions.

[0111] The feature size and number of channels of the stitched features are adjusted to obtain intermediate features. Two depthwise separable convolution operations are performed on the intermediate features to obtain attention weights. The attention weights are multiplied by the intermediate features to obtain the makeup layer features; see the above description.

[0112] Subtract the makeup layer features from the intermediate features, and then perform feature resolution enhancement and channel dimension reduction operations on the subtraction result to obtain a bare face image; refer to the above description.

[0113] In step S3 above, the device compares the unedited facial image with a reference facial image to obtain an identity verification result based on facial recognition. The reference facial image can be a photo corresponding to an ID card. Figure 2 In IDphoto, image similarity is implemented based on the Discrominator model.

[0114] like Figure 6 As shown, the Discrominator model is explained below:

[0115] First, the baseline face image needs to undergo face detection processing. Any face detection method can be used to obtain a face image with the same size of 448×448×3.

[0116] The generated unedited face image and the baseline face image are stitched together. Referring to ResNet-101, the Conv-block and Identity-block modules are used, and the operation is repeated 6 times to obtain a feature size of 7×7 and 1024 channels. A 7×7×1204 convolution operation is performed and activated by the ReLU function, with an output feature size of 1×1 and 1024 channels. A 1×1×1024 convolution operation is performed and activated by the ReLU function, with the output feature size remaining unchanged. A fully connected Dense layer is then added, with Nums_classes representing the number of classes. In this case, it is a binary classification problem, activated by the Softmax function, to obtain the face similarity score.

[0117] The face recognition-based identity verification method further includes:

[0118] The preset face detection box fine-tuning model is pre-trained using a first loss function;

[0119] The first loss function is the smoothing loss function L1.

[0120] The face recognition-based identity verification method further includes:

[0121] The preset makeup separation model is pre-trained using a second loss function;

[0122] The second loss function is obtained by adding the perceptual loss function and the mean square error (MSE) function.

[0123] like Figure 7 As shown, the training set consists of photos of faces with makeup and corresponding ID photos without makeup. One photo of a face with makeup corresponds to one ID photo without makeup. Each photo corresponds to a binary image of a face, with the face part having a value of 1 and the rest having a value of 0. That is, the face part is white and the rest is black, which is used to calculate the coordinates of the face detection location. Figure 7 and Figure 3 Most of the content is the same, so I won't repeat it here. The only difference is in the training loss function. The loss function corresponding to RPN is Loss_RPN, the loss function corresponding to AdjustII is Loss_Adjust (the first loss function), the loss function corresponding to the preset makeup separation model is Loss_G (the second loss function), and the loss function corresponding to the Discrominator model is Loss_D.

[0124] The loss function expression for AdjustII is as follows:

[0125]

[0126] L reg (c i ,c i * )=∑smooth L1 (c i -c i * )

[0127]

[0128] Among them, c i It is the network prediction adjustment value, c i * These are the true adjusted values ​​in the target set, where x1 and x2 represent the horizontal coordinate adjustment values ​​of the frame, y1 and y2 represent the vertical coordinate adjustment values ​​of the frame, λ represents the weights of the loss function, and N...reg This represents the total number of samples, and the loss is calculated using four adjusted values ​​for smoothness. L1 The loss is calculated as follows: The corresponding target set is obtained by subtracting the corresponding RoIs coordinates from the actual face location coordinates.

[0129] Figure 8 The Loss_Adjust parameter can be found in the above explanation; the rest can be found in the following explanation. Figure 4 Related explanations.

[0130] Figure 9 The MSE in this context can be referenced above. Specifically, the ID photo peer features are obtained by processing the ID photo through the same convolutional network and are not used in network training. The perceptual loss function is calculated by combining the ID photo peer features and the features after makeup removal. This is for deep supervision to improve the accuracy of the bare-faced features. The remaining parts can be referenced... Figure 5 Related explanations.

[0131] Figure 10 The loss function part will not be elaborated here; the rest can be found in [reference needed]. Figure 6 Related explanations.

[0132] The face recognition-based identity verification method provided in this invention acquires a face image to be verified, extracts features from the face image, and fine-tunes the face detection box based on the extracted features. The face image to be verified is an image containing the face of a user with makeup. Makeup features are extracted based on the fine-tuned face detection box and the extracted features, and the makeup features are separated to obtain a bare-faced face image. The bare-faced face image is compared with a reference face image to obtain the identity verification result based on face recognition. This method can improve the accuracy of face recognition with makeup, thereby improving the efficiency of face-based identity verification.

[0133] Further, the feature extraction based on the face image to be verified includes:

[0134] Extract the shallow and deep facial features of the face image to be verified; this can be described with reference to the above embodiments and will not be repeated here.

[0135] The deep facial features are added to the shallow facial features of each high-level layer to obtain the high-level fusion features; wherein, the shallow facial features of each high-level layer are the shallow facial features of each layer retained after removing the shallow facial features of the first layer; the above embodiments can be referred to for explanation, and will not be repeated here.

[0136] The highest-level fusion feature among the high-level fusion features is extracted again to obtain the deep fusion features of each high-level feature. This can be referred to the above embodiment for explanation, and will not be repeated here.

[0137] The identity verification processing method based on face recognition provided in this invention extracts deep fusion features from various high-level layers, which can improve the precision of image recognition.

[0138] Furthermore, the step of fine-tuning the face detection box based on the extracted features includes:

[0139] Based on the aforementioned deep fusion features and the preset face detection box fine-tuning model, the coordinates of the four endpoints of the face detection box are numerically adjusted to obtain the coordinate adjustment values ​​corresponding to the four endpoints respectively; this can be referred to the above embodiment for explanation, and will not be repeated here.

[0140] The preset face detection box fine-tuning model includes three convolutional layers and one fully connected layer connected in series; please refer to the above embodiments for further explanation.

[0141] The face detection bounding box is fine-tuned based on the coordinate adjustment values ​​to obtain the fine-tuned face detection bounding box. This can be referred to the above embodiment for further explanation, and will not be repeated here.

[0142] The face recognition-based identity verification method provided in this invention can improve the accuracy of identity verification by fine-tuning the face detection box.

[0143] Furthermore, the step of numerically adjusting the coordinates of the four endpoints of the face detection box based on the high-level deep fusion features and the preset face detection box fine-tuning model includes:

[0144] The face detection bounding box is used to extract corresponding feature segments from each high-level deep fusion feature; this can be referred to the above embodiments for explanation, and will not be repeated here.

[0145] The coordinate adjustment values ​​output by the fine-tuning model based on the preset face detection bounding box are described above. Further details are omitted here.

[0146] The identity verification processing method based on face recognition provided in this embodiment of the invention can improve the processing efficiency of numerical adjustment by outputting the coordinate adjustment values ​​of each coordinate through a preset face detection box fine-tuning model.

[0147] Further, the step of extracting makeup features based on the fine-tuned face detection bounding box and the extracted features, and separating the makeup features to obtain a bare-faced face image, includes:

[0148] The finely tuned face detection bounding box is used to extract target feature fragments from the extracted features; the target feature fragments include each high-level deep fusion feature and each layer of face shallow feature; the above embodiments can be referred to for explanation, and will not be repeated here.

[0149] Based on the preset makeup separation model, the image resolution of each high-level deep fusion feature and each layer of facial shallow features is adjusted, and the two types of features after resolution adjustment are stitched together; the above embodiments can be referred to for explanation, and will not be repeated here.

[0150] The feature size and number of channels of the stitched features are adjusted to obtain intermediate features. Two depthwise separable convolution operations are performed on the intermediate features to obtain attention weights. The attention weights are multiplied by the intermediate features to obtain the makeup layer features. The above embodiments can be referred to for explanation, and will not be repeated here.

[0151] The makeup layer features are subtracted from the intermediate features, and the result of the subtraction is subjected to operations to increase feature resolution and reduce channel dimension to obtain a bare face image; the above embodiments can be referred to for explanation, and will not be repeated here.

[0152] The preset makeup separation model includes an upsampling model unit and a makeup separation model unit; the above embodiments can be referred to for explanation, and will not be repeated here.

[0153] The upsampling model unit includes a first channel and a second channel connected in parallel; this can be referred to the above embodiment for explanation, and will not be repeated here.

[0154] The first channel includes two cascaded transposed convolutional layers, which are used to adjust the resolution of the deep fusion feature images of each high layer; this can be referred to the above embodiments for explanation, and will not be repeated here.

[0155] The second channel includes a transposed convolutional layer, which is used to adjust the image resolution of the shallow facial features in each layer; this can be referred to the above embodiment for explanation, and will not be repeated here.

[0156] The first and second channels are each connected to a transposed convolutional layer, which is used to adjust the feature size and number of channels of the spliced ​​features to obtain the intermediate features; the above embodiments can be referred to for explanation, and will not be repeated here.

[0157] The makeup separation model unit includes two cascaded depthwise separable convolutional layers used to obtain makeup layer features; please refer to the above embodiments for further explanation.

[0158] After obtaining the subtraction result, the upsampling model unit further includes three cascaded transposed convolutional layers to improve feature resolution and reduce channel dimensions, thereby obtaining a bare-face image. This can be referred to the above embodiments for further explanation, and will not be repeated here.

[0159] The identity verification processing method based on face recognition provided in this invention can accurately obtain makeup layer features and improve the accuracy of bare face images by separating makeup layer features.

[0160] Furthermore, the face recognition-based identity verification processing method further includes:

[0161] The preset face detection box fine-tuning model is pre-trained using a first loss function;

[0162] The first loss function is the smoothing loss function L1. This can be referred to the above embodiments for further explanation, and will not be repeated here.

[0163] The identity verification processing method based on face recognition provided in this embodiment of the invention can improve the training efficiency of the fine-tuning model of the preset face detection box.

[0164] Furthermore, the face recognition-based identity verification processing method further includes:

[0165] The preset makeup separation model is pre-trained using a second loss function;

[0166] The second loss function is obtained by adding the perceptual loss function and the mean squared error function. This can be referred to the above embodiment for explanation, and will not be repeated here.

[0167] The identity verification processing method based on face recognition provided in this embodiment of the invention can improve the training efficiency of the preset makeup separation model.

[0168] It should be noted that the identity verification processing method based on face recognition provided in this embodiment of the invention can be used in the financial field, or in any technical field other than the financial field. This embodiment of the invention does not limit the application field of the identity verification processing method based on face recognition.

[0169] Figure 11 This is a schematic diagram of the structure of an identity verification processing device based on face recognition provided in an embodiment of the present invention, as shown below. Figure 11 As shown, the identity verification processing device based on face recognition provided in this embodiment of the invention includes an acquisition module 1101, a separation unit 1102, and a verification unit 1103, wherein:

[0170] The acquisition unit 1101 is used to acquire the face image to be verified, extract features from the face image to be verified, and fine-tune the face detection box based on the extracted features; the face image to be verified is an image containing the face of a user with makeup; the separation unit 1102 is used to extract makeup features based on the fine-tuned face detection box and the extracted features, and separate the makeup features to obtain a bare face image; the verification unit 1103 is used to compare the bare face image with a reference face image to obtain an identity verification result based on face recognition.

[0171] Specifically, the acquisition unit 1101 in the device is used to acquire the face image to be verified, extract features from the face image to be verified, and fine-tune the face detection box based on the extracted features; the face image to be verified is an image containing the face of a user with makeup; the separation unit 1102 is used to extract makeup features based on the fine-tuned face detection box and the extracted features, and separate the makeup features to obtain a bare face image; the verification unit 1103 is used to compare the bare face image with a reference face image to obtain an identity verification result based on face recognition.

[0172] The face recognition-based identity verification processing device provided in this invention acquires a face image to be verified, extracts features from the face image, and fine-tunes the face detection box based on the extracted features. The face image to be verified is an image containing the face of a user with makeup. Makeup features are extracted based on the fine-tuned face detection box and the extracted features, and the makeup features are separated to obtain a bare-faced face image. The bare-faced face image is compared with a reference face image to obtain the identity verification result based on face recognition. This can improve the accuracy of face recognition with makeup, thereby improving the efficiency of face-based identity verification.

[0173] Furthermore, the acquisition unit 1101 is specifically used for:

[0174] Extract the shallow and deep facial features of each layer of the face image to be verified;

[0175] The deep facial features are added to the shallow facial features of each high-level face to obtain the high-level fusion features; wherein, the shallow facial features of each high-level face are the shallow facial features of each layer retained after removing the shallow facial features of the first layer.

[0176] The highest-level fusion feature among the high-level fusion features is extracted again to obtain the deep fusion features of each high-level feature.

[0177] The identity verification processing device based on face recognition provided in this embodiment of the invention extracts deep fusion features from various high-level layers, which can improve the precision of image recognition.

[0178] Furthermore, the acquisition unit 1101 is also specifically used for:

[0179] Based on the aforementioned high-level deep fusion features and the preset face detection box fine-tuning model, the coordinates of the four endpoints of the face detection box are numerically adjusted to obtain the coordinate adjustment values ​​corresponding to the four endpoints respectively.

[0180] The preset face detection box fine-tuning model includes three convolutional layers and one fully connected layer connected in series.

[0181] The face detection box is fine-tuned based on the coordinate adjustment values ​​to obtain the fine-tuned face detection box.

[0182] The face recognition-based identity verification processing device provided in this embodiment of the invention can improve the accuracy of identity verification by fine-tuning the face detection box.

[0183] Furthermore, the acquisition unit 1101 is also specifically used for:

[0184] The face detection box is used to extract corresponding feature segments from each high-level deep fusion feature;

[0185] The coordinate adjustment values ​​of each output by the fine-tuning model based on the preset face detection box are determined.

[0186] The identity verification processing device based on face recognition provided in this embodiment of the invention can improve the processing efficiency of numerical adjustment by outputting the coordinate adjustment values ​​of each coordinate through a preset face detection box fine-tuning model.

[0187] Furthermore, the separation unit 1102 is specifically used for:

[0188] The finely tuned face detection bounding box is used to extract target feature fragments from the extracted features; the target feature fragments include each high-level deep fusion feature and each layer of shallow face features;

[0189] Based on the preset makeup separation model, the image resolution of each high-level deep fusion feature and each layer of face shallow features is adjusted, and the two types of features after resolution adjustment are stitched together.

[0190] The feature size and number of channels of the spliced ​​features are adjusted to obtain intermediate features. Two depthwise separable convolution operations are performed on the intermediate features to obtain attention weights. The attention weights are multiplied by the intermediate features to obtain the makeup layer features.

[0191] Subtract the makeup layer features from the intermediate features, and perform feature resolution enhancement and channel dimension reduction operations on the subtraction result to obtain a bare face image;

[0192] The preset makeup separation model includes an upsampling model unit and a makeup separation model unit.

[0193] The upsampling model unit includes a first channel and a second channel connected in parallel;

[0194] The first channel includes two cascaded transposed convolutional layers, used to adjust the resolution of the deep fusion feature images of each high layer;

[0195] The second channel includes a transposed convolutional layer, which is used to adjust the image resolution of the shallow facial features in each layer;

[0196] Two cascaded transposed convolutional layers are included at the ends of the first and second channels to adjust the feature size and number of channels of the spliced ​​features to obtain the intermediate features;

[0197] The makeup separation model unit includes two cascaded depthwise separable convolutional layers used to obtain makeup layer features;

[0198] After obtaining the subtraction result, the upsampling model unit also includes three cascaded transposed convolutional layers to improve feature resolution and reduce channel dimension, thereby obtaining a bare face image.

[0199] The identity verification processing device based on face recognition provided in this embodiment of the invention can accurately obtain makeup layer features and improve the accuracy of bare face images by separating makeup layer features.

[0200] Furthermore, the face recognition-based identity verification processing device is also used for:

[0201] The preset face detection box fine-tuning model is pre-trained using a first loss function;

[0202] The first loss function is the smoothing loss function L1.

[0203] The identity verification processing device based on face recognition provided in this embodiment of the invention can improve the training efficiency of the preset face detection box fine-tuning model.

[0204] Furthermore, the face recognition-based identity verification processing device is also used for:

[0205] The preset makeup separation model is pre-trained using a second loss function;

[0206] The second loss function is obtained by adding the perceptual loss function and the mean squared error function.

[0207] The identity verification processing device based on face recognition provided in this embodiment of the invention can improve the training efficiency of the preset makeup separation model.

[0208] The embodiments of the present invention provide an identity verification processing device based on face recognition, which can be used to execute the processing flow of the above method embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the above method embodiments.

[0209] Figure 12 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 12 As shown, the electronic device includes: a processor 1201, a memory 1202, and a bus 1203;

[0210] The processor 1201 and the memory 1202 communicate with each other via the bus 1203.

[0211] The processor 1201 is used to call program instructions in the memory 1202 to execute the methods provided in the above-described method embodiments, including, for example:

[0212] A face image to be verified is obtained; features are extracted from the face image; and the face detection box is fine-tuned based on the extracted features. The face image to be verified is an image containing the face of a user who has applied makeup.

[0213] Based on the finely adjusted face detection box and the extracted features, makeup features are extracted and separated to obtain a bare face image;

[0214] The image similarity of the unedited face image is compared with that of a reference face image to obtain the identity verification result based on face recognition.

[0215] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as:

[0216] A face image to be verified is obtained; features are extracted from the face image; and the face detection box is fine-tuned based on the extracted features. The face image to be verified is an image containing the face of a user who has applied makeup.

[0217] Based on the finely adjusted face detection box and the extracted features, makeup features are extracted and separated to obtain a bare face image;

[0218] The image similarity of the unedited face image is compared with that of a reference face image to obtain the identity verification result based on face recognition.

[0219] This embodiment provides a computer-readable storage medium storing a computer program that causes the computer to execute the methods provided in the above-described method embodiments, including, for example:

[0220] A face image to be verified is obtained; features are extracted from the face image; and the face detection box is fine-tuned based on the extracted features. The face image to be verified is an image containing the face of a user who has applied makeup.

[0221] Based on the finely adjusted face detection box and the extracted features, makeup features are extracted and separated to obtain a bare face image;

[0222] The image similarity of the unedited face image is compared with that of a reference face image to obtain the identity verification result based on face recognition.

[0223] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0225] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0226] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0227] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0228] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A face recognition-based identity verification processing method, characterized in that, include: Obtain the face image to be verified, extract features from the face image to be verified, and fine-tune the face detection box based on the extracted features; The face image to be verified is an image containing the face of a user who has applied makeup; Based on the finely adjusted face detection box and the extracted features, makeup features are extracted and separated to obtain a bare face image; The image similarity of the unfaced human face image is compared with that of a reference human face image to obtain an identity verification result based on face recognition. The feature extraction based on the face image to be verified includes: extracting shallow and deep face features from each layer of the face image to be verified; adding the deep face features to each high-level shallow face feature to obtain each high-level fusion feature; wherein each high-level shallow face feature is the shallow face feature retained after removing the first-level shallow face feature; and extracting features again from the highest-level fusion feature in each high-level fusion feature to obtain each high-level deep fusion feature. The process of fine-tuning the face detection box based on the extracted features includes: adjusting the coordinates of the four endpoints of the face detection box based on the high-level deep fusion features and a preset face detection box fine-tuning model to obtain coordinate adjustment values ​​corresponding to the four endpoints; wherein the preset face detection box fine-tuning model includes three convolutional layers and one fully connected layer connected in series, wherein each convolutional layer is used to adjust the image resolution and number of channels, and the fully connected layer is used to output the coordinate adjustment values; and the face detection box is fine-tuned based on the coordinate adjustment values ​​to obtain the fine-tuned face detection box, wherein the convolutional layers are convolutional layers with ReLU functions, and the ReLU function corresponds one-to-one with each convolutional layer; The step of extracting makeup features based on the fine-tuned face detection box and the extracted features, and separating the makeup features to obtain a bare-faced face image, includes: using the fine-tuned face detection box to extract target feature fragments from the extracted features; the target feature fragments include high-level deep fusion features and shallow face features; adjusting the image resolution of each high-level deep fusion feature and each shallow face feature based on a preset makeup separation model, and then concatenating the two types of features after resolution adjustment; adjusting the feature size and number of channels of the concatenated features to obtain intermediate features; performing two depthwise separable convolution operations on the intermediate features to obtain attention weights; multiplying the attention weights by the intermediate features to obtain makeup layer features; subtracting the makeup layer features from the intermediate features, and performing operations to increase feature resolution and reduce channel dimension on the subtraction result. A bare-face image is obtained; wherein, the preset makeup separation model includes an upsampling model unit and a makeup separation model unit; the upsampling model unit includes a first channel and a second channel connected in parallel; the first channel includes two cascaded transposed convolutional layers for adjusting the resolution of each high-level deep fusion feature image; the second channel includes a transposed convolutional layer for adjusting the image resolution of each layer of shallow facial features; at the ends of the first and second channels, two cascaded transposed convolutional layers are included for adjusting the feature size and number of channels of the stitched features to obtain the intermediate features; the makeup separation model unit includes two cascaded depth-separable convolutional layers for obtaining makeup layer features; after obtaining the subtraction result, the upsampling model unit also includes three cascaded transposed convolutional layers for improving feature resolution and reducing channel dimensions to obtain a bare-face image.

2. The identity verification processing method based on face recognition according to claim 1, characterized in that, The step of adjusting the coordinates of the four endpoints of the face detection box based on the deep fusion features of each high layer and the preset face detection box fine-tuning model includes: The face detection box is used to extract corresponding feature segments from each high-level deep fusion feature; The coordinate adjustment values ​​of each output by the fine-tuning model based on the preset face detection box are determined.

3. The identity verification processing method based on face recognition according to claim 1, characterized in that, The face recognition-based identity verification method further includes: The preset face detection box fine-tuning model is pre-trained using a first loss function; The first loss function is the smoothing loss function L1.

4. The identity verification processing method based on face recognition according to claim 1, characterized in that, The face recognition-based identity verification method further includes: The preset makeup separation model is pre-trained using a second loss function; The second loss function is obtained by adding the perceptual loss function and the mean squared error function.

5. An identity verification processing device based on facial recognition, characterized in that, include: The acquisition unit is used to acquire a face image to be verified, extract features from the face image to be verified, and fine-tune the face detection box based on the extracted features; the face image to be verified is an image containing the face of a user who has applied makeup. The separation unit is used to extract makeup features based on the fine-tuned face detection box and the extracted features, and to separate the makeup features to obtain a bare face image; The verification unit is used to compare the unfaced human face image with the reference human face image to obtain an identity verification result based on face recognition. The acquisition unit is specifically used for: extracting shallow and deep facial features of each layer of the face image to be verified; adding the deep facial features to each high-level shallow facial feature to obtain each high-level fusion feature; wherein, each high-level shallow facial feature is the shallow facial feature of each layer retained after removing the first-level shallow facial feature; and extracting features again from the highest-level fusion feature in each high-level fusion feature to obtain each high-level deep fusion feature. The acquisition unit is further specifically used for: adjusting the coordinates of the four endpoints of the face detection box based on the high-level deep fusion features and the preset face detection box fine-tuning model, to obtain the coordinate adjustment values ​​corresponding to the four endpoints respectively; wherein the preset face detection box fine-tuning model includes three convolutional layers and one fully connected layer connected in series, wherein each convolutional layer is used to adjust the image resolution and number of channels, and the fully connected layer is used to output the coordinate adjustment values; and fine-tuning the face detection box according to the coordinate adjustment values ​​to obtain the fine-tuned face detection box, wherein the convolutional layers are convolutional layers with ReLU functions, and the ReLU function corresponds one-to-one with each convolutional layer; The separation unit is further specifically used for: using the fine-tuned face detection box to extract target feature fragments from the extracted features; the target feature fragments include each high-level deep fusion feature and each layer of face shallow features; adjusting the image resolution of each high-level deep fusion feature and each layer of face shallow features based on a preset makeup separation model, and stitching the two types of features after resolution adjustment; adjusting the feature size and number of channels of the stitched features to obtain intermediate features, performing two depthwise separable convolution operations on the intermediate features to obtain attention weights, multiplying the attention weights with the intermediate features to obtain makeup layer features; subtracting the makeup layer features from the intermediate features, and performing operations to increase feature resolution and reduce channel dimension on the subtraction result to obtain a bare face image; wherein, the preset makeup separation model The model includes an upsampling model unit and a makeup separation model unit. The upsampling model unit includes a first channel and a second channel connected in parallel. The first channel includes two cascaded transposed convolutional layers for adjusting the resolution of each high-level deep fusion feature image. The second channel includes a transposed convolutional layer for adjusting the image resolution of each layer of shallow facial features. At the ends of the first and second channels, two cascaded transposed convolutional layers are included for adjusting the feature size and number of channels of the stitched features to obtain the intermediate features. The makeup separation model unit includes two cascaded depth-separable convolutional layers for obtaining makeup layer features. After obtaining the subtraction result, the upsampling model unit also includes three cascaded transposed convolutional layers for improving feature resolution and reducing channel dimensions to obtain a bare-faced image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images