Face recognition method, apparatus, device, medium, and product

Abstract

The application discloses a face recognition method, device, equipment, medium and product, and relates to the field of face recognition. The method comprises the following steps: collecting a user face image to be recognized; performing face reconstruction on the user face image to be recognized by using a generative adversarial network (GAN) according to pre-acquired face feature information, to obtain a user reconstructed face image; and performing face recognition by using the user reconstructed face image, to obtain a recognition result. Thus, after the user face image to be recognized is collected, the user face image to be recognized in a face recognition request is reconstructed by using the GAN according to the pre-acquired face feature information, to obtain the user reconstructed face image, so that the influence of face occlusion on the recognition result is avoided. Then, face recognition is performed by using the user reconstructed face image, to obtain the recognition result, and the accuracy of face recognition is improved.

Description

Technical Field

[0001] This application relates to the field of facial recognition, and more particularly to a facial recognition method, apparatus, device, medium, and product. Background Technology

[0002] Facial recognition technology is based on human facial features. It analyzes specific facial information, such as organ position information, from input facial images or video streams, and further extracts the identity features contained in each face based on this information. It then compares these features with known faces to identify the identity of each face.

[0003] With the development of biometric identification technology, biometric features, including face, fingerprint, palm print, iris, retina, voice, body shape, and personal habits (such as the force and frequency of keystrokes, signature), have been more widely used. Correspondingly, identification technologies include face recognition, fingerprint recognition, palm print recognition, iris recognition, retinal recognition, voice recognition (voice recognition can be used for identity verification or for recognizing voice content, but only the former belongs to biometric identification technology), body shape recognition, keystroke recognition, signature recognition, etc.

[0004] However, current face reconstruction technology in the field of face recognition usually reconstructs face images by training convolutional neural networks. This method is too costly and the accuracy of contour extraction is not high enough.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a face recognition method, apparatus, device, medium, and product, which aims to improve the accuracy of face recognition.

[0007] To achieve the above objectives, this application proposes a face recognition method, which is applied to a face recognition system, and the method includes:

[0008] Collect facial images of the user to be identified;

[0009] Based on the pre-acquired facial feature information, the user's facial image is reconstructed using a Generative Adversarial Network (GAN) to obtain the user's reconstructed facial image.

[0010] The user's reconstructed facial image is used for facial recognition to obtain the recognition result.

[0011] In one embodiment, before the step of reconstructing the user's face image using a generative adversarial network (GAN) based on pre-acquired facial feature information to obtain a reconstructed user face image, the method further includes:

[0012] Obtain the first set of face images;

[0013] The first set of face images is binarized to obtain the second set of face images, and the contours of the second set of face images are extracted using an edge detection algorithm to obtain the third set of face images.

[0014] Feature extraction is performed on the third set of face images to obtain face feature information.

[0015] In one embodiment, the step of reconstructing the user's face image using a Generative Adversarial Network (GAN) based on pre-acquired facial feature information to obtain a reconstructed user face image includes:

[0016] When the user's face image is occluded, feature extraction is performed on the user's face image to obtain face occlusion feature information and face unoccluded feature information;

[0017] Based on the face occlusion feature information, similarity calculation is performed on the user's face image using the first face image set to obtain similar face images;

[0018] Extract the facial contour information of the similar facial images, and reconstruct the user's facial image using the GAN based on the facial contour information, facial feature information, and facial unocclusion information to obtain a grayscale facial image;

[0019] The grayscale face image is filled with RGB values ​​to obtain the user's reconstructed face image.

[0020] In one embodiment, the step of calculating the similarity of the user's face image using the first face image set based on the face occlusion feature information to obtain similar face images includes:

[0021] The face images in the first face image set are replaced with grayscale using the face occlusion feature information to obtain the fourth face image set;

[0022] The image similarity is obtained by calculating the image grayscale values ​​of the fourth set of face images using the image grayscale values ​​of the user's face image;

[0023] Based on the image similarity, similar face images are extracted from the fourth face image set. In one embodiment, the step of performing RGB filling on the grayscale face image to obtain the user-reconstructed face image includes:

[0024] RGB values ​​are extracted from the unobstructed facial features to obtain a first RGB set;

[0025] The RGB values ​​of the similar face images are extracted to obtain a second RGB set;

[0026] The grayscale face image is filled with RGB using the first RGB set and the second RGB set to obtain the user's reconstructed face image.

[0027] In one embodiment, the step of performing face recognition using the reconstructed face image from the user to obtain the recognition result includes:

[0028] The first set of face images is matched with the facial contour and facial area of ​​the reconstructed face image by the user to obtain a reference face image;

[0029] Based on the facial features of the user's reconstructed face image, cosine similarity matching is performed on the reference face image to obtain the matching result;

[0030] If the matching result is successful, the output recognition result is "Recognition passed";

[0031] If the matching result is a failure, the output recognition result is "recognition failed".

[0032] Furthermore, to achieve the above objectives, this application also proposes a face recognition device, which is applied to a face recognition system, and the device includes:

[0033] The acquisition module is used to acquire images of the user's face to be identified;

[0034] The reconstruction module is used to reconstruct the user's face image based on the pre-acquired facial feature information using a generative adversarial network (GAN) to obtain the reconstructed face image of the user.

[0035] The verification module is used to perform face recognition based on the reconstructed face image of the user and obtain the recognition result.

[0036] In addition, to achieve the above objectives, this application also proposes a face recognition device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the face recognition method as described above.

[0037] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the face recognition method described above.

[0038] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the face recognition method described above.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] This application proposes a face recognition method, apparatus, device, storage medium, and computer program product. The method involves acquiring a user's face image to be recognized; reconstructing the user's face image using a Generative Adversarial Network (GAN) based on pre-acquired facial feature information; and then performing face recognition using the reconstructed face image to obtain a recognition result. Therefore, after acquiring the user's face image to be recognized, the method uses a GAN to reconstruct the user's face image in the face recognition request based on pre-acquired facial feature information, avoiding the influence of face occlusion on the recognition result. Subsequently, face recognition is performed using the reconstructed face image to obtain the recognition result, thus improving the accuracy of face recognition. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating an embodiment of the applicant's face recognition method.

[0044] Figure 2 This is a schematic diagram illustrating the face recognition method of the applicant, which involves reconstructing a face image.

[0045] Figure 3 This is a flowchart illustrating Embodiment 2 of the applicant's face recognition method;

[0046] Figure 4This is a schematic diagram illustrating the extraction of facial feature information involved in the applicant's face recognition method;

[0047] Figure 5 This is a schematic diagram of the reference points involved in the applicant's face recognition method;

[0048] Figure 6 This is a schematic diagram of the module structure of the face recognition device according to an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the face recognition method in the embodiments of this application.

[0050] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0052] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of this application embodiment is as follows: First, a first set of face images is acquired; the first set of face images is binarized to obtain a second set of face images, and contour extraction is performed on the second set of face images using an edge detection algorithm to obtain a third set of face images; features are extracted from the third set of face images to obtain face feature information. When the user's face image is occluded, features are extracted from the user's face image to obtain occluded face feature information and unoccluded face feature information; based on the occluded face feature information, similarity calculation is performed on the user's face image using the first set of face images to obtain similar face images; face contour information is extracted from the similar face images, and based on the face contour information, face feature information, and unoccluded face information, the user's face image is reconstructed using the GAN to obtain a grayscale face image; the grayscale face image is filled with RGB to obtain the reconstructed user face image. The face images in the first face image set are replaced with grayscale using the face occlusion feature information to obtain a fourth face image set. The grayscale values ​​of the user's face image are then compared with the grayscale values ​​of the images in the fourth face image set to obtain an image similarity. Similar face images are extracted from the fourth face image set based on the image similarity. RGB values ​​are extracted from the unoccluded face features to obtain a first RGB set. RGB values ​​are extracted from the similar face images to obtain a second RGB set. The grayscale face images are then filled with RGB values ​​using the first and second RGB sets to obtain a user-reconstructed face image. The face contour and face area of ​​the user-reconstructed face image are matched against the first face image set to obtain a reference face image. Cosine similarity matching is performed on the reference face image based on the face features of the user-reconstructed face image to obtain a matching result. If the matching result is successful, the recognition result is output as "Recognition Passed"; if the matching result is unsuccessful, the recognition result is output as "Recognition Failed". This improves the accuracy of face recognition, enabling face verification and enhancing the overall accuracy of face recognition. Based on this invention, addressing the problem that current face reconstruction techniques typically rely on training convolutional neural networks (CNNs), which is time-consuming, costly, and lacks sufficient precision in contour extraction, resulting in low accuracy, this invention proposes a new face recognition method. The effectiveness of this method was verified during face verification, demonstrating a significant improvement in the accuracy of face recognition using this method.

[0054] In this embodiment, for ease of description, the following description uses a face recognition device as the execution subject.

[0055] In the field of facial recognition, current facial reconstruction techniques typically rely on training convolutional neural networks (CNNs) to reconstruct facial images. Training CNNs is time-consuming and extremely costly, and subsequent maintenance also requires significant investment, resulting in excessively high costs for facial reconstruction. Furthermore, the presence of irrelevant factors during facial recognition necessitates high accuracy in contour extraction, which current technologies still suffer from insufficient precision. This leads to low accuracy in facial recognition and negatively impacts the user experience.

[0056] This application provides a solution for face recognition systems. By combining techniques such as Generative Adversarial Networks (GAN), HOG feature extraction, and RGB filling, a face recognition algorithm with low computational cost, high accuracy, and suitability for occluded environments is proposed. At the same time, this application reduces the reliance on complex deep neural networks, lowers computational costs, and provides users with better services.

[0057] As can be seen from the above embodiments, this application acquires a user's face image to be identified; based on pre-acquired facial feature information, it reconstructs the user's face image using a Generative Adversarial Network (GAN) to obtain a reconstructed user face image; and then performs face recognition using the reconstructed user face image to obtain a recognition result. Therefore, after acquiring the user's face image to be identified, it reconstructs the user's face image in the face recognition request using a GAN based on pre-acquired facial feature information, obtaining a reconstructed user face image. This avoids the influence of face occlusion on the recognition result. Subsequently, it uses the reconstructed user face image to perform face recognition, obtaining a recognition result and improving the accuracy of face recognition.

[0058] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or facial recognition device capable of performing the above functions. The following description uses a facial recognition device as an example to illustrate this embodiment and the subsequent embodiments.

[0059] Based on this, the embodiments of this application provide a face recognition method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the applicant's face recognition method.

[0060] In this embodiment, the face recognition method is applied to a face recognition system, and the method includes steps S01 to S06:

[0061] Step S01: Acquire the user's face image to be identified;

[0062] Before the implementation of this embodiment, it should be clear that with the development of Internet technology, more and more applications, systems and devices are using facial recognition technology to identify people. Facial recognition technology is based on human facial features. It analyzes facial information (such as organ position information) on the input facial image or video stream, and further extracts the identity features contained in each face based on this information, and compares them with a normal face to identify each person's identity.

[0063] With the development of biometric recognition technology, biometric features, including face, fingerprint, palm print, iris, retina, voice, body shape, and personal habits (such as the force and frequency of keystrokes, signature), have been more widely used. Face recognition is the most widely used in the market. However, in actual use, users may wear masks, glasses, or headphones, which can significantly hinder the use of face recognition technology. Currently, face reconstruction technology in the field of face recognition can solve this problem. It reconstructs face images by training convolutional neural networks. However, this method is too costly and the accuracy of contour extraction is not high enough.

[0064] Therefore, in this embodiment, a face recognition method is proposed, which is mainly applied to a face recognition system. This system can be used for enterprise access control, online shopping payment, and face verification in government scenarios. This embodiment takes online shopping payment as an example. After the user selects the goods and prepares to pay, a payment request will be issued. At this time, the face recognition system simultaneously collects the face image of the user to be recognized, so as to verify the user's face in the future.

[0065] Step S05: Based on the pre-acquired facial feature information, perform face reconstruction on the user's face image using a Generative Adversarial Network (GAN) to obtain the user's reconstructed face image.

[0066] It should be clear that there may be instances of user occlusion in user face images. Therefore, in such cases, a Generative Adversarial Network (GAN) can be used to reconstruct the user's face based on pre-acquired facial feature information, resulting in a reconstructed user face image. Here, the facial feature information is obtained by pre-collecting user data. In practical applications, user faces are collected, feature information is analyzed, and the data is stored in the first set of face images in a face image database for subsequent face recognition. A Generative Adversarial Network (GAN) is a deep learning model used to generate realistic data samples. A GAN consists of two networks: a generator and a discriminator. They learn through adversarial training. The generator learns to generate data that is as realistic as possible, while the discriminator learns to distinguish whether the data is real or not. The two networks compete against each other and improve together. Ultimately, the generator can generate fake data that is almost indistinguishable from real data.

[0067] Step S06: Perform face recognition using the reconstructed face image of the user to obtain the recognition result.

[0068] Once the reconstructed user face image is obtained, face recognition can be performed on it. At this stage, the user face image has already been reconstructed. Although in most cases, users will actively remove masks, headphones, or glasses when performing face recognition, the recognition may not be accurate enough due to the influence of weather, lighting, and human factors (such as changes in weight or skin color). Therefore, using face reconstruction methods at this time can greatly avoid recognition errors and achieve more accurate recognition results.

[0069] Specifically, step S05 above, which involves reconstructing the user's face image using a Generative Adversarial Network (GAN) based on pre-acquired facial feature information to obtain the reconstructed user face image, includes the following steps:

[0070] Step S051: When the user's face image is occluded, feature extraction is performed on the user's face image to obtain face occlusion feature information and face unoccluded feature information.

[0071] Step S052: Based on the face occlusion feature information, perform similarity calculation on the user's face image through the first face image set to obtain similar face images;

[0072] Step S053: Extract the facial contour information of the similar face image, and reconstruct the user's face image using the GAN based on the facial contour information, facial feature information, and unoccluded face information to obtain a grayscale face image;

[0073] Step S054: RGB fill is applied to the grayscale face image to obtain the user-reconstructed face image.

[0074] In actual use, users may not wear masks, glasses or other devices, so when reconstructing images, it is also necessary to determine whether the user's face image is obscured.

[0075] When a user's face image is occluded, feature extraction is performed to obtain the user's occluded feature information and unoccluded feature information. In this embodiment, the occluded feature information is the outline and area of ​​the occluder in the user's face image, while the unoccluded feature information is the outline and area of ​​the unoccluded parts of the user's eyes, eyebrows, nose, mouth, and ears. For example, if the user's face image is a face image of the user wearing a mask, then the occluded feature information is the outline and area of ​​the mask worn by the user, and the unoccluded feature information is the outline and area of ​​the eyes and eyebrows. Similarly, if the user's face image is a face image of the user wearing glasses, then the occluded feature information is the outline and area of ​​the frame of the glasses worn by the user, and the unoccluded feature information is the outline and area of ​​the eyebrows, nose, mouth, and ears.

[0076] To obtain a more accurate reconstructed image, this embodiment also requires aligning each face image in the first face image set with the user's face image. The method for aligning each face image in the first face image set with the user's face image is as follows:

[0077] ① Obtain the center position between the two eyebrows and the tip of the nose in each face image and the user's face image in the first face image set, and connect the center position between the two eyebrows and the tip of the nose with a straight line.

[0078] ② Obtain the two intersection points of the line connecting the center position between the two eyebrows and the tip position of the nose with the facial contour in the face image;

[0079] ③ Obtain the line segment between the two intersection points to obtain the first set of line segments and the second set of line segments. The line segment between the two intersection points is a segment of the straight line connecting the center position between the two eyebrows and the tip position of the nose. The first line segment is the line segment between the two intersection points in the user's face image.

[0080] The second line segment is the line segment between two intersection points in the user's face image;

[0081] ④ Obtain the midpoint of the line segment, align the midpoint of all user face images with the midpoint of the user face image, determine whether the first line segment and the second line segment in the first line segment set coincide, if they do not coincide, rotate the user face image corresponding to the non-coincident first line segment around the midpoint until all the first line segments and the second line segments in the first line segment set coincide.

[0082] ⑤ Complete the alignment operation between each face image in the first face image set and the user's face image. After alignment, the midpoint of each face image in the first face image set and the user's face image are located at the center of their respective face images.

[0083] After the alignment operation is completed, the similarity of the user's face images is calculated using the first set of face images, and the image with the highest similarity in the first set of face images is selected as the similar face image.

[0084] Subsequently, the contour extraction method within the Histogram of Oriented Gradients (HOG) feature extraction approach is used to extract the contours of faces in similar face images. Based on the first face feature information and unoccluded face information from the first face image set, the user's eyes, eyebrows, nose, mouth, and ears are filled into the contours of faces in similar face images to obtain the reconstructed user face image. For example:

[0085] ① The user's face image is obtained when the user is wearing a mask. The outline of the user's face is extracted from similar face images. Since the user is wearing a mask, the only unobstructed information that can be extracted from the user's face image is the outline of the eyes, eyebrows, and ears. Therefore, the outlines of the eyes, eyebrows, and ears in the user's face and their corresponding position information are used to fill in the corresponding positions of the outline of the user's face in the obtained similar face images to obtain a face image filled with eyes, eyebrows, and ears.

[0086] ② Obtain the distance from the root of the user's nose to the tip of the nose and the height of the tip of the nose in the user's face image to reconstruct the user's nose. Specifically, the method for reconstructing the user's nose involves obtaining the distance from the root of the nose to the tip of the nose, the height of the tip of the nose, and the shape of the nose in the face image corresponding to the image database of similar face images. Based on the obtained distance from the root of the user's nose to the tip of the nose in the user's face image, the user's nose in the similar face image is scaled proportionally to obtain the reconstructed user's nose in the user's face image. The reconstructed user's nose in the user's face image is then used to fill the corresponding position in the outline of the user's face in the similar face image. The height of the tip of the nose can be obtained by obtaining the distance from the highest point of the mask bulge to the face screen.

[0087] ③ Using the above contour and area extraction methods, extract the areas corresponding to the eyes, eyebrows, ears, and nose of the reconstructed face that has been filled with eyes, eyebrows, ears, and nose, and obtain the proportion of the eyes, eyebrows, ears, and nose in the reconstructed face.

[0088] ④ Based on the proportion of eyes, eyebrows, ears, and nose in the reconstructed face, obtain the face images in the first face image set that have the same proportion of eyes, eyebrows, ears, and nose in the face. Obtain the outline, area, and position information of the mouth in the face image. Based on the position information of the mouth in the face image, fill in the corresponding position of the outline of the user's face in the similar face images according to the outline and area of ​​the mouth to obtain the reconstructed user face image (i.e., the user reconstructed face image).

[0089] Since using grayscale images directly for face recognition lacks the important feature information of RGB values, which affects the accuracy of face recognition, it is necessary to perform RGB filling on the reconstructed face image before face recognition to obtain a reconstructed color face image, thereby improving the accuracy of face recognition.

[0090] More specifically, step S052 above, which involves calculating the similarity of the user's face image with the first face image set based on the face occlusion feature information to obtain similar face images, includes:

[0091] Step S0521: The face images in the first face image set are replaced with grayscale using the face occlusion feature information to obtain the fourth face image set;

[0092] Step S0522: Calculate the similarity between the grayscale values ​​of the user's face image and the grayscale values ​​of the fourth face image set to obtain the image similarity.

[0093] Step S0523: Extract similar face images from the fourth face image set based on the image similarity.

[0094] In practice, it is necessary to first obtain the location information and grayscale value information corresponding to the face occlusion feature information, and then perform occlusion operation on the face images in the first face image set according to the location information and grayscale value information of the face occlusion feature information to obtain the fourth face image set.

[0095] One method for obtaining the location information and grayscale value information corresponding to the face occlusion feature information is to obtain the coordinate position information and corresponding grayscale value of all pixels in the region corresponding to the outline of the occluder in the face occlusion feature information in the user's face image, so as to obtain the location information and grayscale value information corresponding to the face occlusion feature information.

[0096] Subsequently, the gray values ​​of pixels at the corresponding positions in the face images of the first face image set are replaced according to the corresponding gray values ​​of the face occlusion feature information to obtain the fourth face image set.

[0097] Then, the face image most similar to the user's face image in the fourth face image set is obtained to obtain similar face images. The variance of the image gray value can be used to represent the degree of dispersion of the gray value of each pixel in the image. The covariance of the gray values ​​of two images can be used to represent whether the variation pattern of the gray value of each pixel in the two images is consistent. The similarity between the two images is determined by judging whether the degree of dispersion and variation pattern of each pixel in the two images are consistent.

[0098] Therefore, the loss value between each face image in the fourth face image set and the user face image can be calculated based on the variance and covariance of the gray values ​​of the face images in the fourth face image set and the variance and covariance of the gray values ​​of the user face image.

[0099] Obtaining the most similar face image to the user's face image from the fourth face image set can be achieved by calculating the loss value between each face image in the fourth face image set and the user's face image, thus obtaining a set of loss values. The most similar face image in the fourth face image set is then obtained based on this set of loss values. The formula used to calculate the loss value between each face image in the fourth face image set and the user's face image to obtain the set of loss values ​​is as follows:

[0100]

[0101] In the formula, M represents the image width of the face image and the user's face image in the fourth face image set, N represents the image height of the face image and the user's face image in the fourth face image set, μ1 represents the average grayscale value of the face image in the fourth face image set, and μ2 represents the grayscale value of the user's face image. This represents the variance of the grayscale values ​​of the face images in the fourth set of face images. σ represents the variance of the grayscale values ​​in a user's face image. 12 C1 represents the covariance between the grayscale values ​​of the face images in the fourth set of face images and the user's face image, while C1, C2, and C3 represent three different constants used to avoid the denominator of the formula being 0.

[0102] Therefore, it can be clearly seen that in this embodiment, generative adversarial networks (GANs) are mainly used for face reconstruction in partially occluded images. The specific formula is as follows:

[0103] L=λ1L GAN +λ2L recon +λ3L prceptual

[0104] Where L GAN To combat the losses, L recon For reconstruction loss, L prceptual To perceive the loss, GAN generates realistic face reconstruction images through a discriminator, reducing the impact of occlusion;

[0105] When a user wears a mask or glasses, the user's face image is obscured, making it impossible to directly perform face recognition using the first set of face images. In this case, it is necessary to reconstruct the face image based on the face feature information.

[0106] like Figure 2 As shown, face reconstruction based on facial feature information can be achieved by extracting occluders from occluded face images, using the extracted occluders to occlude face images in a first set of face images to obtain a fourth set of face images, then calculating the face image in the fourth set that is most similar to the occluded face image to obtain similar face images, extracting the contours of similar face images, and using the facial feature information of the target user to fill in the contours of similar face images to complete the reconstruction of the target user's face.

[0107] Further, step S054 above, which involves performing RGB filling on the grayscale face image to obtain the user-reconstructed face image, includes:

[0108] Step S0541: Extract RGB values ​​from the unobstructed facial features to obtain a first RGB set;

[0109] Step S0542: Extract RGB values ​​from the similar face images to obtain a second RGB set;

[0110] Step S0543: The grayscale face image is filled with RGB using the first RGB set and the second RGB set to obtain the user reconstructed face image.

[0111] In order to obtain more accurate face recognition results, this embodiment obtains the RGB values ​​of the user's skin in the unoccluded part of the user's face image, as well as the RGB values ​​of the eyes and eyebrows, to obtain a first RGB set;

[0112] Obtain the RGB values ​​of the nose and mouth of the face images in the first set of face images corresponding to similar face images, so as to obtain the second set of RGB values;

[0113] The reconstructed face image is filled with RGB values ​​based on the first RGB set and the second RGB value set to obtain the reconstructed color face image.

[0114] One method for obtaining RGB values ​​is to obtain the RGB values ​​and position information of all pixels in the corresponding region of the face image in the first face image set, and fill all pixels in the corresponding position in the reconstructed face image according to the obtained RGB values ​​to complete the RGB filling of the face.

[0115] This embodiment employs the above-described scheme, specifically by acquiring a user's facial image to be identified; based on pre-acquired facial feature information, performing facial reconstruction using a Generative Adversarial Network (GAN) to obtain a reconstructed facial image; and then performing facial recognition using this reconstructed image to obtain the recognition result. Thus, after acquiring the user's facial image to be identified, the GAN is used to reconstruct the user's facial image in the facial recognition request based on pre-acquired facial feature information, avoiding the impact of facial occlusion on the recognition result. Subsequently, the reconstructed facial image is used for facial recognition to obtain the recognition result, improving the accuracy of facial recognition.

[0116] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Before step S05, the face recognition method further includes steps S02 to S04:

[0117] Step S02: Obtain the first set of face images;

[0118] Step S03: Binarize the first set of face images to obtain the second set of face images, and extract the contours of the second set of face images using an edge detection algorithm to obtain the third set of face images;

[0119] Step S04: Extract features from the third set of face images to obtain face feature information.

[0120] In this embodiment, facial feature information is obtained as follows: Figure 4 As shown, it can be done through:

[0121] ① Obtain the first set of face images.

[0122] All users' facial images can be acquired using conventional image acquisition methods (such as cameras, infrared cameras, etc.) as the first set of facial images. The facial images are the unobstructed facial images corresponding to the users. The unobstructed facial images of the users are collected by this recognition system when entering user information. The collected unobstructed facial images can be used as reference facial images when performing facial recognition.

[0123] ② The first set of face images described above is binarized to remove background noise and obtain clear face contours, thus obtaining a second set of face images. Specifically, this can be done by obtaining the pixel matrix corresponding to the face image. This pixel matrix can be obtained by acquiring the grayscale values ​​of all pixels in the face image and the position information of each pixel within the face image. The grayscale value of each pixel is then placed at the position corresponding to its position information. The grayscale value of each pixel in the pixel matrix is ​​then checked against a preset grayscale threshold. Pixels with grayscale values ​​greater than the threshold are set to 0, and pixels with grayscale values ​​less than or equal to the threshold are set to 255. This results in a binarized image consisting only of pixels with grayscale values ​​of 0 or 255, thus obtaining the second set of face images. A grayscale value of 0 corresponds to black, and a grayscale value of 255 corresponds to white.

[0124]

[0125] Where I(x,y) is the gray value of the image, and T is the threshold, which is automatically determined by an adaptive algorithm;

[0126] ③ Obtain the region where the user's face is located in the second set of face images. Extract the contour of the image within the region of the second set of face images to obtain the third set of face images. In the contour extraction of the face images, the Canny edge detection algorithm is used to extract the face contour, which helps to generate a clearer input image and provides high-quality contour information for subsequent face feature extraction and reconstruction. The specific formula is as follows:

[0127]

[0128] Where G(x,y) is the gradient intensity of the pixel, and Gx and Gy are the gradients in the horizontal and vertical directions, respectively.

[0129] For example, when a user undergoes facial recognition in a crowded place, the resulting facial image often contains other faces besides the user, causing facial recognition to fail. In this case, the area with the highest concentration and largest area of ​​pixels with a grayscale value of 0 in the second set of facial images can be obtained to determine the area where the user's face is located in the facial image, thus eliminating the interference of other faces besides the user. The image in the area where the user's face is located in the second set of facial images can then be identified as the third set of facial images.

[0130] ④ Extract features from the face images in the third set of face images to obtain face feature information.

[0131] This embodiment uses the HOG (Histograms of Oriented Gradients) feature extraction method to extract multimodal features from the image, specifically including:

[0132] (1) Gradient magnitude calculation:

[0133]

[0134] Where M(x,y) represents the gradient magnitude at coordinates (x,y). This formula calculates the difference between the gradients of the image in the horizontal direction (x-axis) and the vertical direction (y-axis) to obtain the magnitude of the local gradient.

[0135] (2) Gradient direction calculation

[0136]

[0137] Where θ(x,y) represents the gradient direction, and the gradient direction of each pixel is obtained by calculating the gradient ratio of the vertical direction to the horizontal direction;

[0138] The outline and area of ​​the user's face, as well as the outline and area of ​​the eyes, eyebrows, nose, mouth, and ears, are extracted from the face images in the third face image set. When the user is wearing glasses or a mask, the outline and area of ​​the user's eyes, nose, and mouth cannot be obtained due to the obstruction of the glasses or mask. However, the outline and area of ​​the glasses or mask can be obtained to obtain the facial feature information. The facial feature information includes the outline and area of ​​the user's face, the outline and area of ​​the eyes, eyebrows, nose, mouth, and ears, and the outline and area of ​​the glasses or mask in the face images of the third face image set.

[0139] More specifically, a pixel is randomly selected from the face images in the third face image set as a reference point. The gray values ​​of the eight adjacent pixels in the eight directions of the reference point are obtained. Starting from 0°, the gray values ​​of the eight adjacent pixels in the eight directions of the reference point are judged in a clockwise direction to see if the corresponding gray values ​​are equal to 0.

[0140] The specific direction of 0° is as follows: Figure 5 As shown, if a pixel with a gray value of 0 is adjacent in one direction, then this pixel with a gray value of 0 is taken as a new reference point. The gray values ​​of the eight adjacent pixels of the new reference point are then checked to see if they are equal to 0. This process continues until all pixels in the face images of the third face image set are traversed. If the gray values ​​of the eight adjacent pixels of the reference point are not equal to 0, then the reference point is discarded. A new reference point is randomly selected from the face images of the second face image set, and the above method is repeated until all pixels in the face images of the third face image set are traversed. The regions corresponding to the eyes, eyebrows, nose, mouth, and ears of the user's face in the face image are obtained to obtain the first region, the second region, the third region, the fourth region, and the fifth region.

[0141] The first region represents the position of the eyes on the user's face, the second region represents the position of the eyebrows, the third region represents the position of the nose, the fourth region represents the position of the mouth, and the fifth region represents the position of the ears. The reference points in each of these regions are connected end-to-end with a smooth curve to obtain the contours corresponding to the eyes, eyebrows, nose, mouth, and ears on the user's face. The number of reference points and the area of ​​each reference point are then calculated. Multiplying the number of reference points by the area of ​​each reference point gives the area corresponding to the eyes, eyebrows, nose, mouth, and ears on the user's face.

[0142] Connect the remaining reference points to obtain the outline of the user's face, and use the same method as obtaining the area corresponding to the eyes, eyebrows, nose, mouth and ears of the user's face to obtain the area of ​​the user's face.

[0143] Specifically, in the above embodiments, step S06, which involves performing face recognition using the reconstructed face image from the user to obtain the recognition result, includes:

[0144] Step S061: Match the first set of face images with the face contour and face area of ​​the reconstructed face image by the user to obtain a reference face image;

[0145] Step S062: Based on the facial features of the user's reconstructed face image, perform cosine similarity matching on the reference face image to obtain the matching result;

[0146] Step S063: If the matching result is successful, the recognition result is output as "Recognition passed".

[0147] Step S064: If the matching result is a matching failure, the recognition result is output as recognition failed.

[0148] Step 6: Use the reconstructed color face image for matching to complete face recognition.

[0149] In this embodiment, cosine similarity is used to match the reconstructed face with the face feature vector in the database. The cosine similarity in this embodiment is:

[0150]

[0151] Where A and B are the feature vectors of two faces, Α·Β is the dot product, and ∥A∥ and ∥B∥ are the L2 norms of the vectors;

[0152] The system can obtain the outline and area of ​​the user's face in the reconstructed face image, as well as facial feature information such as the outline and area of ​​the eyes, eyebrows, nose, mouth, and ears. First, the outline and area of ​​the user's face in the reconstructed face image are matched with reference face images in the database to obtain a reference face image that matches the outline and area of ​​the user's face. The reference face image in the database that matches the outline of the user's face is determined as the first reference face image. Then, the outline and area of ​​the user's eyes, eyebrows, nose, mouth, and ears in the reconstructed face image are matched with the first reference face image in turn to determine whether there is a matching result in the first reference face image. If there is a matching result, the face verification passes; if there is no matching result, the face verification fails.

[0153] This embodiment, through the above-described scheme, specifically involves acquiring a first set of face images; binarizing the first set of face images to obtain a second set of face images; extracting contours from the second set of face images using an edge detection algorithm to obtain a third set of face images; and extracting features from the third set of face images to obtain facial feature information. Thus, after acquiring the user's face image to be identified, based on the pre-acquired facial feature information, a Generative Adversarial Network (GAN) is used to reconstruct the user's face image in the face recognition request, obtaining a reconstructed user face image. This avoids the influence of face occlusion on the recognition result. Subsequently, the reconstructed user face image is used for face recognition to obtain the recognition result, improving the accuracy of face recognition.

[0154] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the applicant's face recognition method. Any simple modifications based on this technical concept are within the scope of protection of this application.

[0155] This application also provides a face recognition device, please refer to... Figure 6 The device is used in a face recognition system, and the device includes:

[0156] Acquisition module 10 is used to acquire the user's face image to be identified;

[0157] Reconstruction module 20 is used to reconstruct the user's face image by using a generative adversarial network (GAN) based on pre-acquired facial feature information to obtain a reconstructed user face image.

[0158] The recognition module 30 is used to perform face recognition based on the reconstructed face image of the user and obtain the recognition result.

[0159] The face recognition device provided in this application, employing the face recognition method described in the above embodiments, can improve the accuracy of face recognition. Compared with the prior art, the beneficial effects of the face recognition device provided in this application are the same as those of the face recognition method described in the above embodiments, and other technical features in the face recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0160] This application provides a face recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the face recognition method in Embodiment 1 above.

[0161] The following is for reference. Figure 7The diagram illustrates a structural schematic suitable for implementing the facial recognition device in the embodiments of this application. The facial recognition device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The face recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0162] like Figure 7 As shown, the face recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the face recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the face recognition device to communicate wirelessly or wiredly with other devices to exchange data. Although face recognition devices with various systems are shown in the figures, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0163] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0164] The face recognition device provided in this application, employing the face recognition method described in the above embodiments, can improve the accuracy of face recognition. Compared with the prior art, the beneficial effects of the face recognition device provided in this application are the same as those of the face recognition method described in the above embodiments, and other technical features of this face recognition device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0165] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0166] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0167] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the face recognition method in the above embodiments.

[0168] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0169] The aforementioned computer-readable storage medium may be included in the face recognition device; or it may exist independently and not assembled into the face recognition device.

[0170] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a face recognition device, cause the face recognition device to: acquire a user's face image to be recognized; reconstruct the user's face image using a generative adversarial network (GAN) based on pre-acquired face feature information to obtain a reconstructed user face image; and perform face recognition using the reconstructed user face image to obtain a recognition result.

[0171] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0173] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0174] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described face recognition method, which can improve the accuracy of face recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the face recognition method provided in the above embodiments, and will not be repeated here.

[0175] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the face recognition method described above.

[0176] The computer program product provided in this application can improve the accuracy of face recognition. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the face recognition method provided in the above embodiments, and will not be repeated here.

[0177] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A face recognition method, characterized in that, The method is applied to a face recognition system, and the method includes: Collect facial images of the user to be identified; Based on the pre-acquired facial feature information, the user's facial image is reconstructed using a Generative Adversarial Network (GAN) to obtain the user's reconstructed facial image. The user's reconstructed facial image is used for facial recognition to obtain the recognition result.

2. The method as described in claim 1, characterized in that, Before the step of reconstructing the user's face image using a Generative Adversarial Network (GAN) based on pre-acquired facial feature information to obtain the reconstructed user face image, the method further includes: Obtain the first set of face images; The first set of face images is binarized to obtain the second set of face images, and the contours of the second set of face images are extracted using an edge detection algorithm to obtain the third set of face images. Feature extraction is performed on the third set of face images to obtain face feature information.

3. The method as described in claim 2, characterized in that, The step of reconstructing the user's face image using a Generative Adversarial Network (GAN) based on pre-acquired facial feature information to obtain the reconstructed user face image includes: When the user's face image is occluded, feature extraction is performed on the user's face image to obtain face occlusion feature information and face unoccluded feature information; Based on the face occlusion feature information, similarity calculation is performed on the user's face image using the first face image set to obtain similar face images; Extract the facial contour information of the similar facial images, and reconstruct the user's facial image using the GAN based on the facial contour information, facial feature information, and facial unocclusion information to obtain a grayscale facial image; The grayscale face image is filled with RGB values ​​to obtain the user's reconstructed face image.

4. The method as described in claim 3, characterized in that, The step of calculating the similarity of the user's face image with the first face image set based on the face occlusion feature information to obtain similar face images includes: The face images in the first face image set are replaced with grayscale using the face occlusion feature information to obtain the fourth face image set; The image similarity is obtained by calculating the image grayscale values ​​of the fourth set of face images using the image grayscale values ​​of the user's face image; Based on the image similarity, similar face images are extracted from the fourth face image set.

5. The method as described in claim 3, characterized in that, The step of performing RGB filling on the grayscale face image to obtain the user's reconstructed face image includes: RGB values ​​are extracted from the unobstructed facial features to obtain a first RGB set; The RGB values ​​of the similar face images are extracted to obtain a second RGB set; The grayscale face image is filled with RGB using the first RGB set and the second RGB set to obtain the user's reconstructed face image.

6. The method as described in claim 1, characterized in that, The step of performing face recognition based on the reconstructed face image from the user to obtain the recognition result includes: The first set of face images is matched with the facial contour and facial area of ​​the reconstructed face image by the user to obtain a reference face image; Based on the facial features of the user's reconstructed face image, cosine similarity matching is performed on the reference face image to obtain the matching result; If the matching result is successful, the output recognition result is "Recognition passed"; If the matching result is a failure, the output recognition result is "recognition failed".

7. A face recognition device, characterized in that, The device is used in a face recognition system, and the device includes: The acquisition module is used to acquire images of the user's face to be identified; The reconstruction module is used to reconstruct the user's face image based on the pre-acquired facial feature information using a generative adversarial network (GAN) to obtain the reconstructed face image of the user. The recognition module is used to perform face recognition based on the reconstructed face image of the user and obtain the recognition result.

8. A facial recognition device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the face recognition method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the face recognition method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the face recognition method as described in any one of claims 1 to 6.

Images