Face recognition method and device

Abstract

This application provides a facial recognition method, apparatus, device, and storage medium, relating to the field of facial recognition technology. The method comprises: obtaining a frontal facial image of a user; obtaining a side profile image of the user; determining a first matching degree based on the frontal facial image and preset facial data; determining a second matching degree based on the side profile image and preset facial data; and determining a facial recognition result for the user based on the first matching degree and the second matching degree. This method is applicable to facial recognition processes and is used to address the problem of being unable to recognize the face of a user wearing a mask.

Description

Technical Field

[0001] The present application relates to the field of face recognition technology, and in particular to a face recognition method and device. Background Art

[0002] Facial recognition technology is a biometric technology based on facial features. It is widely used in access control systems in schools, companies, shopping malls, and train stations.

[0003] When using face recognition technology, it is necessary to first collect face images and identify the facial feature information of the person from the face images.

[0004] However, when people need to wear masks, the masks will block some facial features in the collected facial images, making it impossible to perform normal facial recognition. Summary of the Invention

[0005] The present application provides a face recognition method and device that can perform face recognition on users wearing masks.

[0006] In a first aspect, the present application provides a face recognition method, which includes: obtaining a frontal face image of a user; obtaining a side face image of the user; determining a first matching degree based on the frontal face image and preset face data; determining a second matching degree based on the side face image and preset face data; and determining a face recognition result of the user based on the first matching degree and the second matching degree.

[0007] In one possible implementation, obtaining a profile image of the user includes: obtaining the profile image of the user when detecting that the user's face is obscured in the frontal image; determining a second degree of match based on the profile image and preset facial data, including: determining facial feature points of the obscured portion of the frontal image based on the profile image; generating a simulated image of the user based on the facial feature points of the obscured portion of the frontal image; and determining the second degree of match based on the simulated image and the preset facial data.

[0008] In another possible implementation, the preset facial data includes first facial data and second facial data; the first facial data is facial data corresponding to facial feature points in an unobstructed portion of the frontal facial image; and the second facial data is facial data corresponding to facial feature points in an obstructed portion of the frontal facial image. Determining a first degree of match based on the frontal facial image and the preset facial data includes: determining the first degree of match based on the frontal facial image and the first facial data. Determining a second degree of match based on the simulated image and the preset facial data includes: determining the second degree of match based on the simulated image and the second facial data.

[0009] In another possible implementation, the first facial data includes a first array, the first array includes multiple groups of preset first facial feature points, and multiple first numerical values corresponding one-to-one to the multiple groups of preset first facial feature points, each group of preset first facial feature points includes two first facial feature points, and the first numerical value corresponding to each group of preset first facial feature points represents the Euclidean distance between the two first facial feature points in the group of preset first facial feature points; determining the first matching degree based on the frontal face image and the first facial data, including: determining multiple groups of facial feature points in the unobstructed part of the frontal face image, each group of facial feature points includes two facial feature points; calculating the Euclidean distance between the two facial feature points in each group of facial feature points, and obtaining the second numerical value corresponding to each group of facial feature points; determining the second array based on the multiple groups of facial feature points and the second numerical values corresponding one-to-one to the multiple groups of facial feature points; and calculating the first similarity as the first matching degree based on each group of the second array and the first array.

[0010] In another possible implementation, the first facial data includes multiple groups of preset first facial feature points, and multiple first vectors corresponding one-to-one to the multiple groups of preset first facial feature points, each group of preset first facial feature points includes two first facial feature points, and the first facial feature points are facial feature points corresponding to the facial feature points of the unobstructed part of the frontal face image in the first facial data; based on the frontal face image and the first facial data, a first matching degree is determined, including: determining multiple second vectors corresponding one-to-one to the multiple groups of facial feature points in the unobstructed part of the frontal face image, each group of facial feature points includes two facial feature points; based on the cosine distance between each second vector and the corresponding first vector, a second similarity is calculated as the first matching degree.

[0011] In another possible implementation, the first facial data includes multiple groups of preset first facial feature points, multiple first vectors corresponding one-to-one to the multiple groups of preset first facial feature points, and a first array, the first array includes multiple groups of preset first facial feature points, and multiple first numerical values corresponding one-to-one to the multiple groups of preset first facial feature points, each group of preset first facial feature points includes two first facial feature points, the first facial feature points are facial feature points corresponding to the facial feature points of the unobstructed part of the frontal face image in the first facial data; the first numerical value corresponding to each group of preset first facial feature points represents the Euclidean distance between the two first facial feature points in the group of preset first facial feature points; the first matching degree is determined based on the frontal face image and the first facial data. , including: determining multiple groups of facial feature points in the unobstructed part of the frontal face image, each group of facial feature points includes two facial feature points; calculating the Euclidean distance between the two facial feature points in each group of facial feature points to obtain a second numerical value corresponding to each group of facial feature points; determining a second array based on the multiple groups of facial feature points and the second numerical values corresponding to the multiple groups of facial feature points; calculating a first similarity based on the first array and the second array; determining multiple second vectors corresponding to the multiple groups of facial feature points; calculating a second similarity based on the cosine distance between each second vector and the corresponding first vector; taking a weighted sum of the first similarity and the second similarity to obtain a first matching degree, and the weights of the first similarity and the second similarity respectively being greater than 0 and less than 1.

[0012] In another possible implementation, the second facial data includes a third array, the third array including multiple groups of preset second facial feature points and multiple third numerical values corresponding one-to-one to the multiple groups of preset second facial feature points, each group of preset second facial feature points including two second facial feature points, the second facial feature points being facial feature points corresponding to the facial feature points of the occluded portion of the frontal face image in the second facial data; the third numerical value corresponding to each group of preset second facial feature points represents the Euclidean distance between the two second facial feature points in the group of preset second facial feature points. Determining a second degree of match based on the simulated image and the second facial data includes: determining multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; calculating the Euclidean distance between the two facial feature points in each group of facial feature points to obtain a fourth numerical value corresponding to each group of facial feature points; determining a fourth array based on the multiple groups of facial feature points and the fourth numerical values corresponding one-to-one to the multiple groups of facial feature points; and calculating a third similarity based on the third and fourth arrays as the second degree of match.

[0013] In another possible implementation, the second facial data includes multiple sets of preset second facial feature points and multiple third vectors corresponding one-to-one to the multiple sets of preset second facial feature points, each set of preset second facial feature points including two second facial feature points, where the second facial feature points are facial feature points corresponding to facial feature points of an occluded portion of the frontal face image in the second facial data. Determining a second degree of match based on the simulated image and the second facial data includes: determining multiple sets of facial feature points in the simulated image, each set of facial feature points including two facial feature points; determining multiple fourth vectors corresponding one-to-one to the multiple sets of facial feature points in the simulated image, each set of facial feature points including two facial feature points; and calculating a fourth degree of similarity as the second degree of match based on the cosine distance between each fourth vector and the corresponding third vector.

[0014] In another possible implementation, the second facial data includes multiple groups of preset second facial feature points, multiple third vectors corresponding one-to-one to the multiple groups of preset second facial feature points, and a third array, the third array includes multiple groups of preset second facial feature points, and multiple third numerical values corresponding one-to-one to the multiple groups of preset second facial feature points, each group of preset second facial feature points includes two second facial feature points, and the second facial feature points are facial feature points corresponding to the facial feature points of the occluded part in the frontal face image in the second facial data; the third numerical value corresponding to each group of preset second facial feature points represents the Euclidean distance between two second facial feature points in the group of preset second facial feature points. Determining a second degree of matching based on a simulated image and second facial data includes: determining multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; calculating the Euclidean distance between the two facial feature points in each group of facial feature points to obtain a fourth numerical value corresponding to each group of facial feature points; determining a fourth array based on the multiple groups of facial feature points and the one-to-one correspondence of the multiple groups of facial feature points; calculating a third similarity based on the third array and the fourth array; determining multiple fourth vectors corresponding one-to-one to the multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; calculating a fourth similarity based on the cosine distance between each fourth vector and the corresponding third vector; and taking a weighted sum of the third similarity and the fourth similarity to obtain a second degree of matching, wherein the weights of the third similarity and the fourth similarity are respectively greater than 0 and less than 1.

[0015] In another possible implementation, determining a facial recognition result of the user based on the first matching degree and the second matching degree includes: performing a weighted summation of the first matching degree and the second matching degree to obtain a target matching degree, wherein the weights of the first matching degree and the second matching degree are respectively greater than 0 and less than 1. Determining a facial recognition result of the user based on the target matching degree.

[0016] In another possible implementation, generating a simulated image of the user based on facial feature points of the occluded portion in the frontal face image includes: generating a simulated image of the occluded portion based on facial feature points of the occluded portion in the frontal face image.

[0017] In another possible implementation, a simulated image of the user is generated based on the facial feature points of the obscured part of the frontal face image, including: generating a simulated image of the user's frontal face based on the facial feature points of the unobstructed part of the frontal face image and the facial feature points of the obscured part of the frontal face image.

[0018] In another possible implementation, a simulated image of the user is generated based on the facial feature points of the occluded part in the frontal face image, including: generating a two-dimensional sketch based on the facial feature points of the occluded part in the frontal face image; generating a three-dimensional model based on the two-dimensional sketch and a preset convolutional neural network model; and reducing the dimensionality of the three-dimensional model to obtain a simulated image.

[0019] In another possible implementation, the method further includes: performing bilinear interpolation optimization on the three-dimensional model based on the frontal face image and the side face image.

[0020] In another possible implementation, the preset facial data includes N groups, where N is an integer greater than 0, and each group of preset facial data corresponds to a first matching degree and a second matching degree. Determining a facial recognition result for the user based on the first matching degree and the second matching degree includes: determining target matching degrees corresponding to each of the N groups of preset facial data based on the first matching degrees and the second matching degrees corresponding to each of the N groups of preset facial data; and determining the facial recognition result for the user based on a maximum value among the target matching degrees corresponding to each of the N groups of preset facial data.

[0021] In another possible implementation, the preset facial data includes N groups, where N is an integer greater than 0, and each group of preset facial data corresponds to a first matching degree. The method further includes: determining M groups of candidate preset facial data based on the first matching degrees corresponding to the N groups of preset facial data, where M is an integer greater than 0 and less than N; the second matching degrees include M groups corresponding one-to-one to the M groups of candidate preset facial data in the N groups of preset facial data. Determining the user's facial recognition result based on the first matching degree and the second matching degree includes: determining the target matching degrees corresponding to the M groups of candidate preset facial data based on the first matching degrees and the second matching degrees corresponding to the M groups of candidate preset facial data; and determining the user's facial recognition result based on the maximum value of the target matching degrees corresponding to the M groups of candidate preset facial data.

[0022] In another possible implementation, M groups of alternative preset facial data are determined based on the first matching degrees corresponding to N groups of preset facial data, including: determining M groups of preset facial data whose corresponding first matching degrees are greater than a preset matching degree threshold from the N groups of preset facial data as alternative preset facial data.

[0023] In another possible implementation, M groups of alternative preset facial data are determined based on the first matching degrees corresponding to the N groups of preset facial data, including: sorting the N groups of preset facial data in descending order according to the corresponding first matching degrees; and determining the first M groups of preset facial data as alternative preset facial data.

[0024] In another possible implementation, the target matching degree corresponding to each set of preset facial data is the weighted sum of the first matching degree and the second matching degree corresponding to the preset facial data; the weight of the first matching degree in the weighted sum is the first weight, the weight of the second matching degree in the weighted sum is the second weight, the sum of the first weight and the second weight is 1, and the first weight and the second weight are respectively greater than 0 and less than 1.

[0025] The face recognition method provided by the present application can obtain a frontal face image of a user and a side face image of the user. A first matching degree is determined based on the frontal face image of the user and the preset face data. A second matching degree is determined based on the side face image of the user and the preset face data. Facial feature information of the obscured portion of the frontal face image can be obtained from the side face image, and the second matching degree is used to replace the matching degree between the obscured portion of the frontal face image and the preset face data. In the case where the user's face is obscured in the frontal face image, the face recognition result of the user can also be determined based on the first matching degree and the second matching degree, thereby realizing face recognition of users wearing masks.

[0026] In a second aspect, the present application provides a face recognition device, which includes various modules used in the method described in the first aspect or any possible implementation of the first aspect.

[0027] In a third aspect, the present application provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the steps of the relevant method described in the first aspect to implement the method described in the first aspect.

[0028] In a fourth aspect, the present application provides an electronic device comprising: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method described in the first aspect above.

[0029] In a fifth aspect, the present application provides a computer-readable storage medium, which includes: computer software instructions; when the computer software instructions are executed in an electronic device, the electronic device implements the method described in the first aspect above.

[0030] The beneficial effects of the second to fifth aspects mentioned above can be referred to the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the composition of the face recognition system provided in an embodiment of the present application;

[0032] Figure 2 A flowchart of a face recognition method provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a frontal face image provided in an embodiment of the present application;

[0034] Figure 4 Another schematic diagram of a frontal face image provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the Euclidean distance provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of a side face image provided in an embodiment of the present application;

[0037] Figure 7 Another flowchart of the face recognition method provided in the embodiment of the present application;

[0038] Figure 8 A schematic diagram of another flow chart of the face recognition method provided in an embodiment of the present application;

[0039] Figure 9 A schematic diagram of three-dimensional modeling provided in an embodiment of the present application;

[0040] Figure 10 Schematic diagram of the optimization of the three-dimensional model provided in the embodiment of the present application;

[0041] Figure 11 A schematic diagram of the composition of a face recognition device provided in an embodiment of the present application;

[0042] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In the following, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Thus, a feature designated as "first," "second," or "third," etc., may explicitly or implicitly include one or more of the features.

[0044] Facial recognition technology requires first capturing a facial image and identifying a person's facial features from that image. However, when someone wears a mask, such as during an epidemic, the mask can obscure some facial features, making facial recognition impossible.

[0045] Against this background technology, an embodiment of the present application provides a face recognition method that can obtain a user's front face image and side face image; then, based on preset face data, the front face image is identified, and a first matching degree of the unobstructed part of the front face image is determined; based on the preset face data, the side face image is identified to obtain a second matching degree, and the second matching degree is used to replace the matching degree between the obscured part of the front face image and the preset face data. When the user's face is obscured in the front face image, the face recognition result of the user can also be determined based on the first matching degree and the second matching degree, thereby realizing face recognition of users wearing masks.

[0046] The face recognition method provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0047] Figure 1 This is a schematic diagram of the composition of the face recognition system provided in the embodiment of the present application. Figure 1 As shown, the system may include: an image acquisition device 10 ( Figure 1 The image acquisition device 10 is connected to the computing and processing device 20 via a wired network or a wireless network.

[0048] The image acquisition device 10 can capture frontal and side-face images of a user. For example, the image acquisition device 10 can include a display screen that can display a user interface (UI) for capturing a user's facial image. The image acquisition device 10 can capture frontal and side-face images input by the user in the UI.

[0049] The computing and processing device 20 may store preset facial data. The computing and processing device 20 may identify a frontal face image based on the preset facial data, and determine a first degree of matching between the unobstructed portion of the frontal face image and the preset facial data; identify a side face image based on the preset facial data, and determine a second degree of matching between the obstructed portion of the frontal face image and the preset facial data; and determine the facial recognition result of the user based on the first degree of matching and the second degree of matching. For example, the computing and processing device 20 may be a computer, a server, or other device with computing capabilities, wherein the server may be a single server, or may be a server cluster composed of multiple servers. In some implementations, the server cluster may also be a distributed cluster. The embodiments of the present application do not limit the specific form of the server.

[0050] For example, the above Figure 1 The face recognition system shown can be applied to face recognition gates in schools, stations, and residential areas, attendance machines (card clocks) with face recognition functions in companies, and face payment devices in shopping malls. Figure 1 The specific application scenarios of the face recognition system shown are not limited.

[0051] It should be noted that, as described above, preset facial data may be stored in the computing and processing device 20. The image acquisition device 10 and the computing and processing device 20 may be integrated into one body, referred to as a facial recognition device (such as the aforementioned facial recognition gate), and the preset facial data may be stored in the facial recognition device. The image acquisition device 10 and the computing and processing device 20 may also be provided separately, the image acquisition device 10 may be referred to as a facial recognition device, and the computing and processing device 20 may be a back-end server connected to the facial recognition device via a wired network or a wireless network, and the preset facial data may be stored in the back-end server. This embodiment of the present application is not limited thereto.

[0052] It should also be noted that the preset facial data in the computing and processing device 20 may include multiple sets. Correspondingly, the first degree of matching between the unobstructed portion of the frontal facial image obtained by recognizing the frontal facial image based on the multiple sets of preset facial data and the preset facial data, and the second degree of matching between the obstructed portion of the frontal facial image and the preset facial data obtained by recognizing the side facial image based on the multiple sets of preset facial data, may also each include multiple sets. For ease of description, the facial recognition method provided in the embodiments of the present application will first be introduced using any one of the multiple sets of preset facial data as an example.

[0053] Figure 2 This is a flow chart of the face recognition method provided in the embodiment of the present application. The execution subject of this method can be the above Figure 1 The face recognition system shown in Figure 1 is as follows. Figure 2As shown, the method may include S101 to S105.

[0054] S101: Obtain a frontal face image of a user.

[0055] Among them, the user can be any one of Figure 1 The facial recognition system shown here performs identity authentication on users. For example, the user is a student or faculty member using a school's facial recognition gate to enter the campus; or the user is a passenger or station staff member using a station's facial recognition gate to enter the station; or the user is a homeowner or property management company staff using a residential complex's facial recognition gate to enter the community; or the user is an employee using a company's facial recognition attendance machine to clock in; or the user is a consumer using a facial payment device at a shopping mall, etc. This embodiment of the application does not limit the specific identity of the user.

[0056] S102: Obtain a side face image of the user.

[0057] For example, a facial recognition system may first obtain a frontal face image of a user, then use a facial occlusion recognition algorithm to determine whether the user's face is obscured in the frontal face image. Upon detecting that the user's face is obscured in the frontal face image, the system may obtain a profile image of the user. In other words, obtaining a profile image of the user may include: obtaining a profile image of the user upon detecting that the user's face is obscured in the frontal face image.

[0058] Optionally, after obtaining the front face image and the side face image of the user, the method may further include: grayscale processing the front face image and the side face image of the user.

[0059] S103: Determine a first matching degree based on the frontal face image and preset face data.

[0060] The preset facial data may include one or more of a preset facial image, preset facial array data, or preset facial vector data. This embodiment of the present application is not limited thereto. The first matching degree refers to the matching degree between the unobstructed portion of the frontal facial image and the first facial data.

[0061] For example, Figure 3 This is a schematic diagram of a frontal face image provided in an embodiment of the present application. Figure 3 As shown, the frontal face image includes a mask ( Figure 3 The mask divides the frontal face image into an obscured part and an unobscured part. The unobscured part may include multiple unobscured facial feature points, such as eyebrows, eyes, the root of the nose, and ears. Figure 3The blocked portion may include multiple blocked facial feature points, such as the nose tip, mouth, jaw, and mandible. Figure 3 The intersection of the dotted lines is shown as an example).

[0062] For example, Figure 4 Another schematic diagram of a frontal face image provided in the embodiment of the present application. Figure 4 As shown, in Figure 3 Based on the frontal face image shown, a coordinate system can also be introduced to mark facial feature points.

[0063] Optionally, the preset facial data can be stored in two types: facial data corresponding to facial feature points in the unobstructed portion of the frontal facial image is stored as one type, recorded as first facial data; and facial data corresponding to facial feature points in the obstructed portion of the frontal facial image is stored as another type, recorded as second facial data. The frontal facial image obtained in S101 can be compared separately with the first facial data. That is, determining the first degree of match based on the frontal facial image and the preset facial data in S103 can include determining the first degree of match based on the frontal facial data and the first facial data.

[0064] For example, the above Figure 3 Taking the frontal face image shown as an example, the first facial data may include facial data corresponding to facial feature points such as eyebrows, eyes, the root of the nose, and ears; the second facial data may include facial data corresponding to facial feature points such as the tip of the nose, mouth, chin, and mandible.

[0065] In one possible implementation, the first facial data may include a preset facial image. Determining the first degree of match between the frontal facial image and the preset facial data may include: directly comparing the frontal facial image with the preset facial image to determine the first degree of match.

[0066] In another possible implementation, the first facial data includes a first array, which may include multiple groups of preset first facial feature points and multiple first numerical values corresponding to the multiple groups of preset first facial feature points. Each group of preset first facial feature points includes two first facial feature points. The first facial feature points are facial feature points corresponding to the facial feature points of the unobstructed part of the frontal face image in the first facial data (for example, the above Figure 3The facial feature points of the frontal face image and the preset facial data are the eyebrows, eyes, the root of the nose, and the ears, etc.), and the first numerical value corresponding to each group of preset first facial feature points represents the Euclidean distance between two first facial feature points in the group of preset first facial feature points. Determining the first matching degree based on the frontal face image and the preset facial data may include: determining multiple groups of facial feature points in the unobstructed part of the frontal face image, each group of facial feature points including two facial feature points; calculating the Euclidean distance between two facial feature points in each group of facial feature points to obtain a second numerical value corresponding to each group of facial feature points; determining a second array based on the multiple groups of facial feature points and the one-to-one corresponding second numerical values of the multiple groups of facial feature points; and calculating a first similarity based on the first array and the second array as the first matching degree.

[0067] Exemplarily, the first similarity calculated based on the first array and the second array can be calculated using the following formulas (1) and (2).

[0068]

[0069] In formula (1), x1 and x2 represent the horizontal coordinates of any two unobstructed facial feature points in the frontal face image. y1 and y2 represent the vertical coordinates of any two unobstructed facial feature points in the frontal face image. d represents the Euclidean distance between any two unobstructed facial feature points in the frontal face image.

[0070] For example, Figure 5 This is a schematic diagram of the Euclidean distance provided in the embodiment of the present application. Figure 5 As shown, the frontal face image may include unobstructed facial feature points, such as feature point 1, feature point 2, feature point 3, feature point 4, feature point 5, feature point 6, feature point 7, feature point 8, feature point 9, feature point 10, feature point 11, feature point 12, and feature point 13. Among them, the line segment between feature point 1 and feature point 3 can be represented as O1, the line segment between feature point 3 and feature point 7 can be represented as O2, the line segment between feature point 10 and feature point 12 can be represented as O3, and the line segment between feature point 11 and feature point 13 can be represented as O4. According to the above formula (1), the Euclidean distance between feature point 1 and feature point 3 (that is, the length of line segment O1) can be calculated as d1, the Euclidean distance between feature point 2 and feature point 7 (that is, the length of line segment O2) is d2, the Euclidean distance between feature point 10 and feature point 12 (that is, the length of line segment O3) is d3, and the Euclidean distance between feature point 11 and feature point 13 (that is, the length of line segment O4) is d4.

[0071] After calculating multiple Euclidean distances between facial feature points in the unobstructed portion of the frontal face image, the multiple Euclidean distances can be recorded as a two-dimensional array A[i][j]=[d1, d2, d3, d4][3-1, 2-7, 12-10, 11-13]. This two-dimensional array can be called a second array. The first dimension of the second array can include multiple Euclidean distances calculated according to the above formula (1). The second dimension can include multiple groups of facial feature points corresponding to the multiple Euclidean distances of the first dimension (or the two endpoints corresponding to the Euclidean distances), each group of facial feature points including two facial feature points. The Euclidean distances between facial feature points in the unobstructed portion of the frontal face image can also be called a second value. As described above, the first facial data can include a first array. For example, the first array can be a two-dimensional array B[i][j]=[k1, k2, k3, k4][3-1, 2-7, 12-10, 11-13]. The first dimension may include multiple pre-stored Euclidean distances, which may also be referred to as first values. The second dimension may include multiple sets of preset first facial feature points corresponding one-to-one to the multiple Euclidean distances in the first dimension, each set of preset first facial feature points including two first facial feature points.

[0072]

[0073] In formula (2), O S It represents the similarity between the second array and the first array, which can also be called the first similarity. DPR(A, B) represents the dot product ratio (DPR) between the two-dimensional array A and the two-dimensional array B, and the value range of DPR(A, B) is The larger the dot product ratio, the more similar the second array A is to the first array B. n represents the number of Euclidean distance terms between the two arrays A and B with the smallest number of Euclidean distance terms. A[i] represents the Euclidean distance value of the i-th item in the first dimension of the second array A. B[i] represents the Euclidean distance value of the i-th item in the first dimension of the first array B.

[0074] In another possible implementation, the first facial data may include multiple sets of preset first facial feature points, and multiple first vectors corresponding one-to-one to the multiple sets of preset first facial feature points, each set of preset first facial feature points including two first facial feature points. Determining the first degree of match based on the frontal facial image and the preset facial data may include: determining multiple second vectors corresponding one-to-one to the multiple sets of facial feature points in an unobstructed portion of the frontal facial image, each set of facial feature points including two facial feature points; and calculating a second degree of similarity as the first degree of match based on a cosine distance between each second vector and the corresponding first vector.

[0075] Exemplarily, the second similarity is calculated as the first matching degree based on the cosine distance between each second vector and the corresponding first vector, which can be calculated according to the following formula (3) and formula (4).

[0076]

[0077] In formula (3), M p N represents the coordinate value of the vector (also called the second vector) M composed of any two unobstructed facial feature points in the frontal face image in the pth dimension. p represents the coordinate value of the preset user's facial feature vector N (i.e., the first vector mentioned above) corresponding to the aforementioned vector M in the pth dimension. q represents the dimension (degree) of the coordinate system in which vectors M and N are located. cosθ represents the cosine distance between vectors M and N.

[0078] After calculating the multiple cosine distances between vector M and vector N, the multiple cosine distances can also be recorded as a one-dimensional array C[t]=[cosθ1, cosθ2, ..., cosθ s ]. The one-dimensional array can include the cosine distances between multiple vectors M and N.

[0079] After the calculated cosine distance is recorded as a one-dimensional array, the cosine distance similarity between the unobstructed facial feature points and the preset facial data can be calculated according to the following formula (4).

[0080]

[0081] In formula (4), C A Represents the cosine distance similarity between the second vector and the first vector, also known as the second similarity. C[t] represents the value of the t-th cosine distance in the one-dimensional array C. s represents the number of cosine distance items in the one-dimensional array C.

[0082] In another possible implementation, the first matching degree can be obtained by weighted summing the Euclidean distance similarity (first similarity) and the cosine distance similarity (second similarity) after calculating the above-mentioned Euclidean distance similarity (first similarity) and the cosine distance similarity (second similarity). The specific calculation process is shown in the following formula (5).

[0083] α=η*O S +ξ*C A Formula (5)

[0084] In formula (4), α represents the first matching degree. η represents the weight of the Euclidean distance similarity, and η is greater than 0 and less than 1. ξ represents the weight of the cosine distance similarity, and ξ is greater than 0 and less than 1. η+ξ=1.

[0085] Among them, the weight of the Euclidean distance similarity and the weight of the cosine distance similarity can be preset in the face recognition system by the management personnel, or can be obtained by inputting the training samples into the above formulas (1) to (5) for training. For example, the training samples can be the frontal face image of Zhang San wearing a mask, and the facial features of Zhang San in the face feature library. The training process can include: first extracting the facial feature points that are not blocked by the mask from the frontal face image of Zhang San wearing a mask. Then, the weight of the Euclidean distance similarity and the weight of the cosine distance similarity are arbitrarily selected multiple times to calculate the first matching degree according to the above formulas (1) to (4). When the first matching degree is greater than the preset first training matching degree threshold, the weight of the Euclidean distance similarity and the weight of the cosine distance similarity corresponding to the first matching degree can be respectively determined as the target weight of the Euclidean distance similarity and the target weight of the cosine distance similarity and written into the above formula (5). The embodiment of the present application does not limit the specific method of obtaining the weight of the Euclidean distance similarity and the weight of the cosine distance similarity.

[0086] S104: Determine a second matching degree based on the side face image and the preset face data.

[0087] The second degree of match refers to the degree of match between the obscured portion of the frontal face image and the second facial data. In some possible embodiments, determining the second degree of match based on the profile image and the preset facial data may include: determining facial feature points of the obscured portion of the frontal face image based on the profile image; generating a simulated image of the user based on the facial feature points of the obscured portion of the frontal face image; and determining the second degree of match based on the simulated image and the preset facial data.

[0088] For example, Figure 6 A schematic diagram of a side face image provided in an embodiment of the present application. Figure 6 (a) is the side face image of the user on the left side. Figure 6 (b) in the figure is the side face image of the user on the right side. Figure 6 As shown in (a) and (b) in the figure, the facial feature points of the occluded part in the frontal face image can be directly collected in the profile face image, such as the facial feature points of the nose tip, chin, and mandible.

[0089] Optionally, generating a simulated image of the user based on facial feature points of the obscured portion in the frontal face image may include: generating a simulated image of the obscured portion based on facial feature points of the obscured portion in the frontal face image.

[0090] For example, Figure 7 Another flow chart of the face recognition method provided in the embodiment of the present application. Figure 7As shown, after obtaining the user's side face image, the three-dimensional model of the user's obscured part of the face is determined based on the user's side face image, and the three-dimensional model of the user's obscured part of the face is reduced in dimension to obtain a simulated image of the user's obscured part of the face.

[0091] Optionally, generating a simulated image of the user based on the facial feature points of the obscured part of the frontal face image may include: generating a simulated image of the user's frontal face based on the facial feature points of the unobstructed part of the frontal face image and the facial feature points of the obscured part of the frontal face image.

[0092] For example, Figure 8 This is another flow chart of the face recognition method provided in the embodiment of the present application. Figure 8 As shown, after obtaining a frontal face image and a side profile image of the user, a 3D model of the unobstructed portion of the user's face can be generated based on the frontal face image. A 3D model of the obstructed portion of the user's face can be generated based on the side profile image. The 3D models of the unobstructed and obstructed portions of the user's face are then concatenated to generate a 3D model of the user's complete face. The dimensionality of the 3D models is then reduced to obtain a simulated image of the user's complete face.

[0093] Optionally, generating a simulated image of the user based on the facial feature points of the occluded portion in the frontal face image may include: generating a two-dimensional sketch based on the facial feature points of the occluded portion in the frontal face image; generating a three-dimensional model based on the two-dimensional sketch and a preset convolutional neural network (CNN) model; and reducing the dimensionality of the three-dimensional model to obtain a simulated image.

[0094] For example, AlexNet is used as an example to illustrate the preset CNN model. Figure 9 This is a schematic diagram of three-dimensional modeling provided in the embodiment of this application. Figure 9As shown in the figure, a user's 2D sketch, unobstructed by occluders, is fed into AlexNet. The first layer of AlexNet is a convolutional layer. Its input is an image of size 224×224×3. The number of convolution kernels in the first layer is 96, the kernel size is 11×11×3, and the stride is 4. The resulting image size is 55×55×96. The second layer is a convolutional layer. Its input is an image of size 55×55×96. The number of convolution kernels in the second layer is 256, the kernel size is 5×5×48, and the stride is 1. The resulting image size is 27×27×256. The third layer is a convolutional layer. Its input is an image of size 13×13×256. The number of convolution kernels in the third layer is 384, the kernel size is 3×3×256, and the stride is 1. The resulting image size is 13×13×384. The fourth layer is a convolutional layer. Its input is an image of size 13×13×384. It uses 384 convolution kernels of 3×3×256 kernels with a stride of 1. The resulting image size is 13×13×384. The fifth layer is a convolutional layer. Its input is an image of size 13×13×384. It uses 256 convolution kernels of 3×3×384 kernels with a stride of 1. The resulting image size is 13×13×256. The fifth layer also includes an overlapping pooling layer. After pooling, the convolutional image is pooled, resulting in an output image of size 6×6×256. The sixth layer is a fully connected layer. Its input is an image of size 6×6×256. It contains 4096 neurons, each of which outputs a single result. The sixth output is 4096 data points. The seventh layer is a fully connected layer with 4096 neurons. The 4096 data points output by the sixth layer are fully connected to the 4096 neurons in the seventh layer. After processing with ReLU and Dropout, 4096 data points are generated. The eighth layer is a fully connected layer with 1000 neurons. The 4096 data points output by the seventh layer are fully connected to the 1000 neurons in the eighth layer, outputting 1000 trained data points. A 3D model of the user is generated based on the data output by the eighth layer.

[0095] Optionally, as described above, the preset facial data may include first facial data and second facial data, and the frontal facial image may be compared solely with the first facial data. The simulated image generated based on the profile facial image may also be compared solely with the second facial data. That is, determining the second degree of match based on the simulated image and the preset facial data may include determining the second degree of match based on the simulated image and the second facial data.

[0096] In one possible implementation, the second facial data may include a third array, which may include multiple groups of preset second facial feature points and multiple third values corresponding to the multiple groups of preset second facial feature points. Each group of preset second facial feature points includes two second facial feature points. The second facial feature points are facial feature points corresponding to the facial feature points of the occluded part of the frontal face image in the second facial data (for example, the above Figure 3 The third numerical value corresponding to each set of preset second facial feature points represents the Euclidean distance between two second facial feature points in the set of preset second facial feature points. Determining the second degree of matching based on the simulated image and the second facial data may include: determining multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; calculating the Euclidean distance between two facial feature points in each group of facial feature points to obtain a fourth numerical value corresponding to each group of facial feature points; determining a fourth array based on the multiple groups of facial feature points and the one-to-one correspondence between the multiple groups of facial feature points; and calculating a third similarity based on the third array and the fourth array as the second degree of matching.

[0097] In another possible implementation, the second facial data may include multiple sets of preset second facial feature points, and multiple third vectors corresponding one-to-one to the multiple sets of preset second facial feature points, each set of preset second facial feature points including two second facial feature points. Determining the second degree of match based on the simulated image and the second facial data includes: determining multiple sets of facial feature points in the simulated image, each set including two facial feature points; determining multiple fourth vectors corresponding one-to-one to the multiple sets of facial feature points in the simulated image, each set including two facial feature points; and calculating a fourth degree of similarity as the second degree of match based on the cosine distance between each fourth vector and the corresponding third vector.

[0098] In another possible implementation, determining a second degree of matching based on a simulated image and second facial data may include: determining multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; calculating the Euclidean distance between the two facial feature points in each group of facial feature points to obtain a fourth numerical value corresponding to each group of facial feature points; determining a fourth array based on the multiple groups of facial feature points and the one-to-one corresponding fourth numerical values of the multiple groups of facial feature points; calculating a third similarity based on the first array and the third array; determining multiple fourth vectors corresponding one-to-one to the multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; calculating a fourth similarity based on the cosine distance between each fourth vector and the corresponding third vector; and weighted summing the third similarity and the fourth similarity to obtain a second degree of matching, the weights of the third similarity and the fourth similarity respectively being greater than 0 and less than 1.

[0099] It should be noted that the three possible implementation methods of determining the second matching degree based on the simulated image and the second facial data can refer to the three possible implementation methods of determining the first matching degree based on the frontal face image and the first facial data, and will not be described in detail here.

[0100] In the face recognition method provided in the embodiments of the present application, the three-dimensional model of the user generated based on a two-dimensional sketch of the user that is not obstructed by an obstruction and a preset CNN model can be either a three-dimensional model of the obscured portion of the user's face or a three-dimensional model of the user's complete face. When the generated three-dimensional model of the user is a three-dimensional model of the obscured portion of the user's face, no comprehensive calculation of data for the unobstructed portion of the user's face is required during the generation of the three-dimensional model, resulting in less computational effort and greater computational flexibility, thereby improving the efficiency of face recognition. When the generated three-dimensional model of the user is a three-dimensional model of the user's complete face, the generated three-dimensional model of the obscured portion of the user's face needs to be coordinated and spliced together with the three-dimensional model of the unobstructed portion of the user's face. The three-dimensional model of the unobstructed portion of the user's face is generated based on unobstructed facial feature points in the user's frontal face image and has a higher degree of credibility. The coordination and splicing of the three-dimensional model of the obscured portion of the user's face with the more credible three-dimensional model of the unobstructed portion of the user's face improves the overall credibility of the three-dimensional model of the unobstructed portion of the user's face, thereby improving the accuracy of face recognition.

[0101] Optionally, after generating the three-dimensional model, the method may further include: optimizing the three-dimensional model according to the side face image.

[0102] Optionally, the profile face image may include multiple images. Optimizing the three-dimensional model based on the profile face image may include: performing bilinear interpolation optimization on the three-dimensional model based on the front face image and the profile face image. The bilinear interpolation optimization may be performed according to the following formula (5).

[0103]

[0104] In formula (5), C represents a vector group of original data point values of the three-dimensional model, and C may include multiple original data point values. represents the unit weight vector group, It can include multiple unit weight vectors, which can include 0 or 1, 0 indicates that the value of the corresponding original data point in C does not need to be optimized, and 1 indicates that the value of the corresponding original data point in C needs to be optimized. represents the expression weight vector group, It can include multiple expression vectors, and the data point values in the expression vectors are between (-1,1). and The face multi-granularity recognition model of opencv2 can be used to recognize the front face image and multiple side face images. Each face image (front face image or side face image) corresponds to a and Y represents a vector group of optimized data point values after optimizing C.

[0105] For example, take the vector group C = [4.3, 3.1, -2.4, 1, -0.5] of the original data point value of the three-dimensional model as an example, assuming that the face image 1 corresponds to Face image 1 corresponds to Then the vector group of optimized data point values after C optimization is

[0106] For example, the vector group C = [4.3, 3.1, -2.4, 1, -0.5] of the original data point value of the three-dimensional model is taken as an example. Assume that the face image 2 corresponds to Face image 2 corresponds to Then the vector group of optimized data point values after C optimization is

[0107] For example, Figure 10 This is an optimized schematic diagram of the three-dimensional model provided in the embodiment of this application. Figure 10 As shown in FIG, the 3D model of the user generated based on the side face image and the preset CNN model is quite different from the actual face of the user before optimization. After optimization using the bilinear principle shown in formula (5), it is more similar to the actual face of the user.

[0108] In the face recognition method provided in the embodiment of the present application, after generating a three-dimensional model of the user based on the side face image and the preset CNN model, the three-dimensional model can also be bilinearly interpolated and optimized based on multiple face images, so as to achieve "face pinching" shaping of the three-dimensional model. The three-dimensional model after "face pinching" is more similar to the real face, which improves the credibility of the three-dimensional model and thus improves the accuracy of face recognition.

[0109] S105: Determine a face recognition result of the user according to the first matching degree and the second matching degree.

[0110] The user's face recognition result may include pass or fail.

[0111] In one possible implementation, determining the user's face recognition result based on the first matching degree and the second matching degree may include: when the first matching degree is greater than a preset first matching degree threshold, and the second matching degree is greater than a preset second matching degree threshold, determining that the user's face recognition result is passed; when the first matching degree is less than the preset first matching degree threshold, and / or the second matching degree is less than the preset second matching degree threshold, determining that the user's face recognition result is failed.

[0112] In another possible implementation, determining the user's facial recognition result based on the first matching degree and the second matching degree may include: weighting the first matching degree and the second matching degree to determine a target matching degree; when the target matching degree is greater than a preset third matching degree threshold, determining that the user's facial recognition result is passed. It should be noted that when the target matching degree is less than the preset third matching degree threshold, the method can also determine that the user's facial recognition result is failed. That is, after determining the target matching degree, the method can also determine whether the target matching degree is greater than the preset third matching degree threshold. When the target matching degree is greater than the preset third matching degree threshold, the user's facial recognition result is determined to be passed; when the target matching degree is less than the preset third matching degree threshold, the user's facial recognition result is determined to be failed.

[0113] It should also be noted that when the target matching degree is equal to the preset third matching degree threshold, the method can determine that the user's facial recognition result is passed, or determine that the user's facial recognition result is failed. This embodiment of the application does not limit whether the user's facial recognition result is determined to be passed or determined to be failed when the target matching degree is equal to the preset third matching degree threshold.

[0114] Exemplarily, the first matching degree and the second matching degree are weighted to determine the target matching degree, which can be performed according to the following formula (6).

[0115] P=λ*α+δ*β Formula (6)

[0116] In formula (6), P represents the target matching degree. λ represents the weight of the first matching degree, also known as the first weight, and is greater than 0 and less than 1. δ represents the weight of the second matching degree, also known as the second weight, and is greater than 0 and less than 1. The sum of λ and δ is equal to 1.

[0117] The weight of the first matching degree and the weight of the second matching degree can be preset in the face recognition system by the management personnel, or can be obtained by training using training samples in a similar method to the above-mentioned Euclidean distance weight and cosine distance weight. For example, the training samples can be a front face image of Zhang San wearing a mask, a side face image of Zhang San wearing a mask, and a front face image of Zhang San not wearing a mask. The training process can include: first, calculating the first matching degree and the second matching degree for the training samples according to the above-mentioned S103 and S104 respectively. Then, the weight of the first matching degree and the weight of the second matching degree are arbitrarily selected multiple times to calculate the target matching degree according to the above-mentioned formula (6). When the target matching degree is greater than the preset second training matching degree threshold, the weight of the first matching degree and the weight of the second matching degree corresponding to the target matching degree are respectively determined as the target weight of the first matching degree and the target weight of the second matching degree and written into the above-mentioned formula (6). For example, the weight of the first matching degree is 0.5 and the weight of the second matching degree is 0.5; or, the weight of the first matching degree is 0.6 and the weight of the second matching degree is 0.4, etc. The embodiment of the present application does not limit the specific values of the weight of the first matching degree and the weight of the second matching degree.

[0118] The face recognition method provided in the embodiment of the present application can obtain a front face image and a side face image of a user. A first matching degree is determined based on the front face image of the user and the preset face data. A second matching degree is determined based on the side face image of the user and the preset face data. Facial feature information of the obscured portion of the front face image can be obtained from the side face image, and the second matching degree is used to replace the matching degree between the obscured portion of the front face image and the preset face data. In the case where the user's face is obscured in the front face image, the face recognition result of the user can also be determined based on the first matching degree and the second matching degree, thereby realizing face recognition of users wearing masks.

[0119] In some possible embodiments, as described above, the preset facial data may include multiple groups, and the first matching degree and the second matching degree may also correspond to multiple groups respectively. Then, the facial recognition result of the user may be determined according to the target matching degree corresponding to each group of preset facial data after respectively calculating the target matching degree corresponding to each group of preset facial data. For example, the preset facial data may include N groups, where N is an integer greater than 0, and each group of preset facial data may correspond to a first matching degree and a second matching degree respectively. Then, determining the facial recognition result of the user according to the first matching degree and the second matching degree in the above S105 may include: determining the target matching degree corresponding to each of the N groups of preset facial data according to the first matching degree and the second matching degree respectively corresponding to the N groups of preset facial data; and determining the facial recognition result of the user according to the maximum value of the target matching degrees respectively corresponding to the N groups of preset facial data.

[0120] Optionally, based on the first matching degrees and the second matching degrees corresponding to the N groups of preset facial data, the specific calculation process for determining the target matching degrees corresponding to the N groups of preset facial data can refer to the calculation of the weighted sum of the first matching degree and the second matching degree in the above formula (6), which will not be repeated here.

[0121] In some other possible embodiments, M groups of candidate preset facial data may be first screened out from N groups of preset facial data, and then the target matching degrees corresponding to the M groups of candidate preset facial data are calculated, and the user's facial recognition result is determined based on the target matching degrees corresponding to the M groups of candidate preset facial data. That is, the preset facial data includes N groups, N is an integer greater than 0, and each group of preset facial data corresponds to a first matching degree; the second matching degree includes M groups corresponding one-to-one to the M groups of candidate preset facial data in the N groups of preset facial data, M is an integer greater than 0 and less than N. Determining the user's facial recognition result based on the first matching degree and the second matching degree in the above S105 may include: determining the target matching degrees corresponding to the M groups of candidate preset facial data based on the first matching degree and the second matching degree corresponding to the M groups of candidate preset facial data; and determining the user's facial recognition result based on the maximum value of the target matching degrees corresponding to the M groups of candidate preset facial data.

[0122] The M groups of candidate preset face data are determined according to the first matching degrees corresponding to the N groups of preset face data.

[0123] For example, before determining the target matching degrees corresponding to the M groups of alternative preset facial data based on the first matching degrees and the second matching degrees corresponding to the M groups of alternative preset facial data, the method may also include: determining the M groups of alternative preset facial data based on the first matching degrees corresponding to the N groups of said preset facial data.

[0124] In one possible implementation, determining M groups of alternative preset facial data based on the first matching degrees corresponding to N groups of preset facial data may include: determining M groups of preset facial data whose corresponding first matching degrees are greater than a preset matching degree threshold from the N groups of preset facial data as alternative preset facial data.

[0125] For example, taking N as 5, the 5 groups of preset facial data are facial data 1, facial data 2, facial data 3, facial data 4, and facial data 5. Assuming that the first matching degree corresponding to facial data 1 is 75%, the first matching degree corresponding to facial data 2 is 80%, the first matching degree corresponding to facial data 3 is 85%, the first matching degree corresponding to facial data 4 is 90%, and the first matching degree corresponding to facial data 5 is 95%, and the preset matching degree threshold is 83%, then from facial data 1, facial data 2, facial data 3, facial data 4, and facial data 5, facial data 3 (first matching degree is 85%), facial data 4 (first matching degree is 90%), and facial data 5 (first matching degree is 95%) whose corresponding first matching degrees are greater than the preset matching degree threshold (83%) can be determined as alternative preset facial data, that is, M is 3.

[0126] In another possible implementation, determining M groups of alternative preset facial data based on the first matching degrees corresponding to the N groups of preset facial data may include: sorting the N groups of preset facial data in descending order according to the corresponding first matching degrees; and determining the first M groups of preset facial data as alternative preset facial data.

[0127] For example, taking the above N as 5, the 5 groups of preset facial data are facial data 1, facial data 2, facial data 3, facial data 4, and facial data 5, the first matching degree corresponding to facial data 1 is 75%, the first matching degree corresponding to facial data 2 is 80%, the first matching degree corresponding to facial data 3 is 85%, the first matching degree corresponding to facial data 4 is 90%, and the first matching degree corresponding to facial data 5 is 95%. Assume that the administrator presets M to 2, then facial data 1, facial data 2, facial data 3, facial data 4, and facial data 5 are arranged in descending order according to the corresponding first matching degrees: facial data 5 (first matching degree is 95%), facial data 4 (first matching degree is 90%), facial data 3 (first matching degree is 85%), facial data 2 (first matching degree is 80%), and facial data 1 (first matching degree is 75%), and the first two facial data 5 and facial data 4 are determined as alternative preset facial data.

[0128] Optionally, based on the first matching degrees and the second matching degrees corresponding to the M groups of alternative preset facial data, the specific calculation process of determining the target matching degrees corresponding to the M groups of alternative preset facial data can also refer to the calculation of the weighted sum of the first matching degree and the second matching degree in the above formula (6), which will not be repeated here.

[0129] The face recognition method provided in the embodiment of the present application can first perform a screening based on the first matching degree for N groups of preset face data to screen out M groups of alternative preset face data, and then only needs to calculate the target matching degree based on the first matching degree and the second matching degree corresponding to the M groups of alternative preset face data, which reduces the amount of calculation and improves the efficiency of face recognition.

[0130] It can be seen that the above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. Those skilled in the art should easily appreciate that, in combination with the modules and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0131] The embodiment of the present application can divide the functional modules of the above-mentioned face recognition system according to the above-mentioned method example. For example, each functional module can be divided according to each function, or two or more functional units can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0132] In an exemplary embodiment, the present application also provides a face recognition device. Figure 11 This is a schematic diagram of the composition of the face recognition device provided in the embodiment of the present application. Figure 11 As shown, the facial recognition device may include: an acquisition module 1101 and a processing module 1102, which are connected to each other. Acquisition module 1101 is configured to acquire a frontal face image of a user; and acquire a side face image of the user. Processing module 1102 is configured to determine a first matching degree based on the frontal face image and preset facial data; and a second matching degree based on the side face image and the preset facial data; and determine a facial recognition result for the user based on the first matching degree and the second matching degree.

[0133] In some possible embodiments, acquisition module 1101 is specifically configured to acquire a profile image of the user when detecting that the user's face is obscured in the frontal face image. Processing module 1102 is specifically configured to determine, based on the profile image, facial feature points of the obscured portion of the frontal face image; generate a simulated image of the user based on the facial feature points of the obscured portion of the frontal face image; and determine a second degree of match based on the simulated image and preset facial data.

[0134] In some other possible embodiments, the preset facial data includes first facial data and second facial data; the first facial data is facial data corresponding to facial feature points in an unobstructed portion of the frontal facial image; and the second facial data is facial data corresponding to facial feature points in an obstructed portion of the frontal facial image. Processing module 1102 is specifically configured to determine a first degree of match based on the frontal facial image and the first facial data; and to determine a second degree of match based on the simulated image and the second facial data.

[0135] In some other possible embodiments, the first facial data includes a first array, the first array including multiple groups of preset first facial feature points and multiple first numerical values corresponding one-to-one to the multiple groups of preset first facial feature points, each group of preset first facial feature points including two first facial feature points, and the first numerical value corresponding to each group of preset first facial feature points representing the Euclidean distance between the two first facial feature points in the group of preset first facial feature points. Processing module 1102 is specifically configured to determine multiple groups of facial feature points in an unobstructed portion of a frontal face image, each group of facial feature points including two facial feature points; calculate the Euclidean distance between the two facial feature points in each group of facial feature points to obtain a second numerical value corresponding to each group of facial feature points; determine a second array based on the multiple groups of facial feature points and the one-to-one correspondence between the multiple groups of facial feature points; and calculate a first similarity as a first matching degree based on each group of the second array and the first array.

[0136] In some further possible embodiments, the first facial data includes multiple sets of preset first facial feature points and multiple first vectors corresponding one-to-one to the multiple sets of preset first facial feature points, each set of preset first facial feature points including two first facial feature points, where the first facial feature points are facial feature points corresponding to facial feature points in an unobstructed portion of the frontal facial image in the first facial data. Processing module 1102 is specifically configured to determine multiple second vectors corresponding one-to-one to the multiple sets of facial feature points in the unobstructed portion of the frontal facial image, each set of facial feature points including two facial feature points; and calculate a second similarity as the first matching degree based on the cosine distance between each second vector and the corresponding first vector.

[0137] In some other possible embodiments, the first facial data includes multiple groups of preset first facial feature points, multiple first vectors corresponding one-to-one to the multiple groups of preset first facial feature points, and a first array, the first array includes multiple groups of preset first facial feature points, and multiple first numerical values corresponding one-to-one to the multiple groups of preset first facial feature points, each group of preset first facial feature points includes two first facial feature points, the first facial feature points are facial feature points corresponding to the facial feature points of the unobstructed part of the frontal face image in the first facial data; the first numerical value corresponding to each group of preset first facial feature points represents the Euclidean distance between the two first facial feature points in the group of preset first facial feature points. Processing module 1102 is specifically used to determine multiple groups of facial feature points in the unobstructed part of the frontal face image, each group of facial feature points includes two facial feature points; calculate the Euclidean distance between the two facial feature points in each group of facial feature points to obtain a second numerical value corresponding to each group of facial feature points; determine a second array based on the multiple groups of facial feature points and the second numerical values corresponding to the multiple groups of facial feature points; calculate a first similarity based on the first array and the second array; determine multiple second vectors corresponding to the multiple groups of facial feature points; calculate a second similarity based on the cosine distance between each second vector and the corresponding first vector; and perform weighted summation of the first similarity and the second similarity to obtain a first matching degree, where the weights of the first similarity and the second similarity are respectively greater than 0 and less than 1.

[0138] In some other possible embodiments, the second facial data includes a third array, the third array including multiple sets of preset second facial feature points and multiple third numerical values corresponding one-to-one to the multiple sets of preset second facial feature points, each set of preset second facial feature points including two second facial feature points, the second facial feature points being facial feature points corresponding to the facial feature points of the occluded portion of the frontal face image in the second facial data; the third numerical value corresponding to each set of preset second facial feature points represents the Euclidean distance between the two second facial feature points in the set of preset second facial feature points. Processing module 1102 is specifically configured to determine multiple sets of facial feature points in the simulated image, each set of facial feature points including two facial feature points; calculate the Euclidean distance between the two facial feature points in each set of facial feature points to obtain a fourth numerical value corresponding to each set of facial feature points; determine a fourth array based on the multiple sets of facial feature points and the one-to-one fourth numerical values corresponding to the multiple sets of facial feature points; and calculate a third similarity based on the third array and the fourth array as the second matching degree.

[0139] In some other possible embodiments, the second facial data includes multiple sets of preset second facial feature points and multiple third vectors corresponding one-to-one to the multiple sets of preset second facial feature points, each set of preset second facial feature points including two second facial feature points, where the second facial feature points correspond to facial feature points of the occluded portion of the frontal face image in the second facial data. Processing module 1102 is specifically configured to determine multiple sets of facial feature points in the simulated image, each set of facial feature points including two facial feature points; determine multiple fourth vectors corresponding one-to-one to the multiple sets of facial feature points in the simulated image, each set of facial feature points including two facial feature points; and calculate a fourth similarity as the second matching degree based on the cosine distance between each fourth vector and the corresponding third vector.

[0140] In some other possible embodiments, the second facial data includes multiple groups of preset second facial feature points, multiple third vectors corresponding one-to-one to the multiple groups of preset second facial feature points, and a third array, the third array includes multiple groups of preset second facial feature points, and multiple third numerical values corresponding one-to-one to the multiple groups of preset second facial feature points, each group of preset second facial feature points includes two second facial feature points, and the second facial feature points are facial feature points corresponding to the facial feature points of the occluded part in the frontal image in the second facial data; the third numerical value corresponding to each group of preset second facial feature points represents the Euclidean distance between two second facial feature points in the group of preset second facial feature points. Processing module 1102 is specifically used to determine multiple groups of facial feature points in a simulated image, each group of facial feature points including two facial feature points; calculate the Euclidean distance between the two facial feature points in each group of facial feature points to obtain a fourth numerical value corresponding to each group of facial feature points; determine a fourth array based on the multiple groups of facial feature points and the one-to-one corresponding fourth numerical values of the multiple groups of facial feature points; calculate a third similarity based on the third array and the fourth array; determine multiple fourth vectors corresponding one-to-one to the multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; calculate a fourth similarity based on the cosine distance between each fourth vector and the corresponding third vector; perform a weighted sum of the third similarity and the fourth similarity to obtain a second matching degree, where the weights of the third similarity and the fourth similarity are respectively greater than 0 and less than 1.

[0141] In some other possible embodiments, the processing module 1102 is specifically configured to generate a simulated image of the obscured portion based on facial feature points of the obscured portion in the frontal face image.

[0142] In some other possible embodiments, the processing module 1102 is specifically used to generate a simulated image of the user's front face based on the facial feature points of the unobstructed part of the front face image and the facial feature points of the obstructed part of the front face image.

[0143] In some other possible embodiments, the processing module 1102 is specifically used to generate a two-dimensional sketch based on the facial feature points of the occluded part of the frontal face image; generate a three-dimensional model based on the two-dimensional sketch and a preset convolutional neural network model; and reduce the dimension of the three-dimensional model to obtain a simulated image.

[0144] In some other possible embodiments, the processing module 1102 is further configured to perform bilinear interpolation optimization on the three-dimensional model according to the frontal face image and the side face image.

[0145] In some further possible embodiments, the preset facial data includes N groups, where N is an integer greater than 0, and each group of preset facial data corresponds to a first matching degree and a second matching degree. Processing module 1102 is specifically configured to determine target matching degrees corresponding to each of the N groups of preset facial data based on the first matching degrees and the second matching degrees corresponding to each of the N groups of preset facial data; and determine a facial recognition result for the user based on a maximum value among the target matching degrees corresponding to each of the N groups of preset facial data.

[0146] In some other possible embodiments, the preset facial data includes N groups, where N is an integer greater than 0, and each group of preset facial data corresponds to a first matching degree. The processing module 1102 is further configured to determine M groups of alternative preset facial data based on the first matching degrees corresponding to the N groups of preset facial data, where M is an integer greater than 0 and less than N; the second matching degrees include M groups that correspond one-to-one to the M groups of alternative preset facial data in the N groups of preset facial data. The processing module 1102 is specifically configured to determine the target matching degrees corresponding to the M groups of alternative preset facial data based on the first matching degrees and the second matching degrees corresponding to the M groups of alternative preset facial data; and determine the user's facial recognition result based on the maximum value of the target matching degrees corresponding to the M groups of alternative preset facial data.

[0147] In some other possible embodiments, the processing module 1102 is specifically configured to determine, from N groups of preset facial data, M groups of preset facial data corresponding to first matching degrees greater than a preset matching degree threshold as candidate preset facial data.

[0148] In some other possible embodiments, the processing module 1102 is specifically configured to sort the N groups of preset facial data in descending order according to the corresponding first matching degrees; and determine the first M groups of preset facial data as candidate preset facial data.

[0149] In some other possible embodiments, the target matching degree corresponding to each set of preset facial data is the weighted sum of the first matching degree and the second matching degree corresponding to the preset facial data; the weight of the first matching degree in the weighted sum is the first weight, the weight of the second matching degree in the weighted sum is the second weight, the sum of the first weight and the second weight is 1, and the first weight and the second weight are respectively greater than 0 and less than 1.

[0150] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the above-mentioned related method steps to implement the face recognition method in the aforementioned method embodiment.

[0151] In an exemplary embodiment, the present application also provides an electronic device. Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 12 As shown, the electronic device may include: a processor 1201 and a memory 1202; the memory 1202 stores instructions executable by the processor 1201; when the processor 1201 is configured to execute the instructions, the electronic device implements the method described in the aforementioned method embodiment.

[0152] In an exemplary embodiment, the present application also provides a computer-readable storage medium having computer program instructions stored thereon; when the computer program instructions are executed by a network device, the network device implements the method described in the aforementioned embodiment. The computer-readable storage medium can be a non-transitory computer-readable storage medium, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0153] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A face recognition method, characterized in that: The method comprises: Get the user's front face image; When it is detected that the face of the user is blocked in the frontal face image, obtaining a side face image of the user; Determining a first matching degree of an unobstructed portion of the frontal face image based on the frontal face image and preset facial data; Determining a second matching degree based on the side face image and the preset face data; the second matching degree is used to represent the matching degree between the blocked portion of the front face image and the preset face data; Determine a face recognition result of the user according to the first matching degree and the second matching degree.

2. The method according to claim 1, characterized in that The determining a second matching degree based on the side face image and the preset face data includes: Determining facial feature points of an obscured portion of the frontal face image based on the side face image; generating a simulated image of the user based on facial feature points of the obscured portion of the frontal face image; A second matching degree is determined based on the simulated image and the preset facial data.

3. The method according to claim 2, characterized in that The preset facial data includes first facial data and second facial data; the first facial data is facial data corresponding to facial feature points of an unobstructed portion of the frontal face image; the second facial data is facial data corresponding to facial feature points of an obstructed portion of the frontal face image; Determining a first matching degree based on the frontal face image and the preset face data includes: determining a first matching degree based on the frontal face image and the first facial data; Determining a second matching degree based on the simulated image and the preset facial data includes: A second matching degree is determined based on the simulated image and the second facial data.

4. The method according to claim 3, characterized in that The first facial data includes a first array, the first array includes multiple groups of preset first facial feature points, and multiple first numerical values corresponding one-to-one to the multiple groups of preset first facial feature points, each group of the preset first facial feature points includes two first facial feature points, and the first facial feature points are facial feature points corresponding to facial feature points of an unobstructed portion of the frontal face image in the first facial data; the first numerical value corresponding to each group of the preset first facial feature points represents the Euclidean distance between two first facial feature points in the group of the preset first facial feature points; Determining a first matching degree based on the frontal face image and the first facial data includes: Determine multiple groups of facial feature points in the unobstructed portion of the frontal face image, each group of facial feature points including two facial feature points; Calculating the Euclidean distance between two facial feature points in each group of facial feature points to obtain a second value corresponding to each group of facial feature points; Determining a second array based on the multiple groups of facial feature points and the second numerical values corresponding one-to-one to the multiple groups of facial feature points; A first similarity is calculated based on each group of the second array and the first array to be used as the first matching degree.

5. The method according to claim 3, characterized in that The first facial data includes a plurality of groups of preset first facial feature points and a plurality of first vectors corresponding one-to-one to the plurality of groups of preset first facial feature points, each group of the preset first facial feature points includes two first facial feature points, and the first facial feature points are facial feature points corresponding to facial feature points of an unobstructed portion of the frontal face image in the first facial data; Determining a first matching degree based on the frontal face image and the first facial data includes: Determine a plurality of second vectors corresponding one-to-one to a plurality of groups of facial feature points in an unobstructed portion of the frontal face image, wherein each group of facial feature points includes two facial feature points; A second similarity is calculated based on the cosine distance between each second vector and the corresponding first vector to obtain the second similarity as the first matching degree.

6. The method according to claim 3, characterized in that The first facial data includes multiple groups of preset first facial feature points, multiple first vectors corresponding one-to-one to the multiple groups of preset first facial feature points, and a first array, the first array includes the multiple groups of preset first facial feature points and multiple first numerical values corresponding one-to-one to the multiple groups of preset first facial feature points, each group of the preset first facial feature points includes two first facial feature points, the first facial feature points are facial feature points corresponding to facial feature points of an unobstructed portion of the frontal face image in the first facial data; the first numerical value corresponding to each group of the preset first facial feature points represents the Euclidean distance between two first facial feature points in the group of the preset first facial feature points; Determining a first matching degree based on the frontal face image and the first facial data includes: Determine multiple groups of facial feature points in the unobstructed portion of the frontal face image, each group of facial feature points including two facial feature points; Calculating the Euclidean distance between two facial feature points in each group of facial feature points to obtain a second value corresponding to each group of facial feature points; Determining a second array based on the multiple groups of facial feature points and the second numerical values corresponding one-to-one to the multiple groups of facial feature points; Calculating a first similarity based on the first array and the second array; Determine a plurality of second vectors corresponding one-to-one to the plurality of groups of facial feature points; Calculating a second similarity based on a cosine distance between each second vector and the corresponding first vector; The first similarity and the second similarity are weighted and summed to obtain the first matching degree, where the weights of the first similarity and the second similarity are respectively greater than 0 and less than 1.

7. The method according to claim 3, characterized in that The second facial data includes a third array, the third array including a plurality of groups of preset second facial feature points, and a plurality of third numerical values corresponding one-to-one to the plurality of groups of preset second facial feature points, each group of the preset second facial feature points including two second facial feature points, the second facial feature points being facial feature points corresponding to facial feature points of an obscured portion of the frontal face image in the second facial data; the third numerical value corresponding to each group of the preset second facial feature points represents the Euclidean distance between two second facial feature points in the group of preset second facial feature points; Determining a second matching degree based on the simulated image and the second facial data includes: Determining multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; Calculating the Euclidean distance between two facial feature points in each group of facial feature points to obtain a fourth value corresponding to each group of facial feature points; determining a fourth array based on the multiple groups of facial feature points and the fourth numerical values corresponding one-to-one to the multiple groups of facial feature points; A third similarity is calculated based on the third array and the fourth array as the second matching degree.

8. The method according to claim 3, characterized in that The second facial data includes a plurality of groups of preset second facial feature points, and a plurality of third vectors corresponding one-to-one to the plurality of groups of preset second facial feature points, each group of the preset second facial feature points includes two second facial feature points, and the second facial feature points are facial feature points corresponding to the facial feature points of the occluded portion of the frontal face image in the second facial data; Determining a second matching degree based on the simulated image and the second facial data includes: Determining multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; Determine a plurality of fourth vectors corresponding one-to-one to a plurality of groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; A fourth similarity is calculated based on the cosine distance between each of the fourth vectors and the corresponding third vector to obtain the fourth similarity as the second matching degree.

9. The method according to claim 3, characterized in that The second facial data includes multiple groups of preset second facial feature points, multiple third vectors corresponding one-to-one to the multiple groups of preset second facial feature points, and a third array, the third array including the multiple groups of preset second facial feature points and multiple third numerical values corresponding one-to-one to the multiple groups of preset second facial feature points, each group of the preset second facial feature points includes two second facial feature points, and the second facial feature points are facial feature points corresponding to the facial feature points of the occluded portion of the frontal face image in the second facial data; the third numerical value corresponding to each group of the preset second facial feature points represents the Euclidean distance between two second facial feature points in the group of the preset second facial feature points; Determining a second matching degree based on the simulated image and the second facial data includes: Determining multiple groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; Calculating the Euclidean distance between two facial feature points in each group of facial feature points to obtain a fourth value corresponding to each group of facial feature points; determining a fourth array based on the multiple groups of facial feature points and the fourth numerical values corresponding one-to-one to the multiple groups of facial feature points; Calculating a third similarity based on the third array and the fourth array; Determine a plurality of fourth vectors corresponding one-to-one to a plurality of groups of facial feature points in the simulated image, each group of facial feature points including two facial feature points; Calculating a fourth similarity based on a cosine distance between each of the fourth vectors and the corresponding third vector; The second matching degree is obtained by performing a weighted summation on the third similarity and the fourth similarity, wherein the weights of the third similarity and the fourth similarity are respectively greater than 0 and less than 1.

10. The method according to claim 2, characterized in that Generating a simulated image of the user based on facial feature points of the obscured portion of the frontal face image includes: generating a two-dimensional sketch based on facial feature points of the obscured portion of the frontal face image; Generate a three-dimensional model based on the two-dimensional sketch and a preset convolutional neural network model; The three-dimensional model is reduced in dimension to obtain the simulated image.

11. The method according to claim 10, characterized in that The method further comprises: According to the frontal face image and the side face image, bilinear interpolation optimization is performed on the three-dimensional model.

12. The method according to claim 1, characterized in that The preset face data includes N groups, where N is an integer greater than 0, and each group of the preset face data corresponds to one first matching degree and one second matching degree; Determining a face recognition result of the user according to the first matching degree and the second matching degree includes: Determining target matching degrees corresponding to the N sets of preset face data respectively according to the first matching degrees and the second matching degrees corresponding to the N sets of preset face data respectively; The face recognition result of the user is determined according to the maximum value of the target matching degrees corresponding to the N groups of preset face data.

13. The method according to claim 1, wherein The preset facial data includes N groups, where N is an integer greater than 0, and each group of the preset facial data corresponds to a first matching degree; The method further comprises: Determining M groups of candidate preset facial data based on the first matching degrees corresponding to the N groups of preset facial data, where M is an integer greater than 0 and less than N; The second matching degree includes M sets of candidate preset face data corresponding one to one with the M sets of preset face data in the N sets of preset face data; Determining a face recognition result of the user according to the first matching degree and the second matching degree includes: Determining target matching degrees corresponding to the M groups of candidate preset face data respectively according to the first matching degrees and the second matching degrees corresponding to the M groups of candidate preset face data respectively; The face recognition result of the user is determined according to the maximum value of the target matching degrees corresponding to the M groups of candidate preset face data.

14. The method according to claim 13, characterized in that The determining, based on the first matching degrees respectively corresponding to the N groups of preset facial data, the M groups of candidate preset facial data includes: M groups of preset facial data corresponding to the first matching degree greater than a preset matching degree threshold are determined from the N groups of preset facial data as the candidate preset facial data.

15. The method according to claim 13, characterized in that The determining, based on the first matching degrees respectively corresponding to the N groups of preset facial data, the M groups of candidate preset facial data includes: sorting the N groups of preset face data in descending order according to the corresponding first matching degrees; The first M groups of preset face data are determined as the candidate preset face data.

16. The method according to any one of claims 12 to 15, characterized in that: The target matching degree corresponding to each set of the preset facial data is a weighted sum of the first matching degree and the second matching degree corresponding to the preset facial data; The weight of the first matching degree in the weighted sum is the first weight, the weight of the second matching degree in the weighted sum is the second weight, the sum of the first weight and the second weight is 1, and the first weight and the second weight are respectively greater than 0 and less than 1.

17. A face recognition device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is configured to acquire a frontal face image of the user; and when it is detected that the face of the user is blocked in the frontal face image, acquire a side face image of the user; The processing module is used to determine a first matching degree of the unobstructed part of the front face image based on the front face image and the preset face data; determine a second matching degree based on the side face image and the preset face data; the second matching degree is used to characterize the matching degree between the obstructed part of the front face image and the preset face data; and determine the face recognition result of the user based on the first matching degree and the second matching degree.

Images