Face recognition method, face recognition device, terminal equipment and storage medium

Abstract

The present application is applicable to the field of image processing technology, and provides a face recognition method, a face recognition device, a terminal device and a storage medium, including: obtaining an infrared image and three-dimensional point cloud data of a face to be recognized; determining the target infrared features of the face to be recognized based on the infrared image; determining the target three-dimensional point cloud features of the face to be recognized based on the three-dimensional point cloud data; fusing the target infrared features and the target three-dimensional point cloud features to obtain a fusion feature; performing face recognition based on the fusion feature to obtain a face recognition result. The present application can reduce the impact of the external environment on face recognition and improve the accuracy of face recognition.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a face recognition method, a face recognition device, a terminal device and a storage medium. Background Art

[0002] With the development of social informatization, individual identity recognition technology based on biometrics has been more and more widely used. Among them, face recognition technology has great advantages in terms of simplicity, non-contact and privacy, and occupies an important position in individual identity recognition technology based on biometrics. Traditional face recognition technology is easily affected by changes in the external environment (such as lighting, posture and expression, etc.), which greatly reduces the recognition accuracy of face recognition technology. Therefore, how to improve the recognition accuracy of face recognition technology under the influence of the external environment has become an important issue that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a face recognition method, a face recognition device, a terminal device and a storage medium, which can reduce the impact of the external environment on face recognition and improve the accuracy of face recognition.

[0004] In a first aspect, an embodiment of the present application provides a face recognition method, the face recognition method comprising:

[0005] Obtain infrared images and three-dimensional point cloud data of the face to be identified;

[0006] Determining target infrared features of the face to be identified based on the infrared image;

[0007] Determining target three-dimensional point cloud features of the face to be identified based on the three-dimensional point cloud data;

[0008] Fusing the target infrared feature with the target three-dimensional point cloud feature to obtain a fused feature;

[0009] Face recognition is performed according to the fusion features to obtain a face recognition result.

[0010] In a second aspect, an embodiment of the present application provides a face recognition device, the face recognition device comprising:

[0011] An acquisition module is used to acquire infrared images and three-dimensional point cloud data of the face to be recognized;

[0012] A first determination module, used to determine the target infrared features of the face to be identified based on the infrared image;

[0013] A second determination module is used to determine the target three-dimensional point cloud features of the face to be identified based on the three-dimensional point cloud data;

[0014] A fusion module, used for fusing the target infrared feature and the target three-dimensional point cloud feature to obtain a fusion feature;

[0015] The recognition module is used to perform face recognition according to the fusion features to obtain a face recognition result.

[0016] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the face recognition method described in the first aspect above are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the face recognition method described in the first aspect above are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of the face recognition method described in any one of the above-mentioned first aspects.

[0019] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application determines the target infrared features of the face to be identified by acquiring an infrared image of the face to be identified, wherein the target infrared features of the face can be extracted from the complex background and interference based on the acquired infrared image of the face; and then determines the target three-dimensional point cloud features of the face to be identified based on the acquired three-dimensional point cloud data. Since the three-dimensional point cloud features fully include the shape and texture features of the face, the target infrared features and the target three-dimensional point cloud features are fused to obtain fused features, and face recognition is performed based on the fused features, thereby reducing the impact of the external environment on face recognition and improving the accuracy of face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flowchart of a face recognition method provided in Example 1 of the present application;

[0022] Figure 2 This is a flow chart of a face recognition method provided in Embodiment 2 of the present application;

[0023] Figure 3 It is a structural schematic diagram of a face recognition device provided in Embodiment 3 of the present application;

[0024] Figure 4 It is a structural diagram of a terminal device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0026] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0027] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0030] The face recognition method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0031] As an example but not limitation, when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices developed by applying wearable technology to intelligently design daily wearables. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also realize powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can realize complete or partial functions without relying on smartphones, such as smart watches or smart glasses.

[0032] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0033] In order to illustrate the technical solution of the present application, specific embodiments are provided below.

[0034] Reference Figure 1 , shows a flow chart of a face recognition method provided in Embodiment 1 of the present application, the face recognition method is applied to a terminal device, and as shown in the figure, the face recognition method may include the following steps:

[0035] Step 101, obtaining an infrared image and three-dimensional point cloud data of a face to be recognized.

[0036] In an embodiment of the present application, the infrared image and three-dimensional point cloud data of the face to be identified can be acquired by the same 3D camera, and the acquisition time of the two is less than a preset time. The preset time can ensure that the acquired infrared image and three-dimensional point cloud data are for the same face.

[0037] Specifically, a 3D camera may refer to a camera composed of an infrared camera, an ordinary camera (for example, a camera with basic functions such as video recording and static image capture), an infrared laser transmitter, and an image processing chip. Among them, the infrared camera can be used to obtain the infrared image of the above-mentioned face to be identified, which contains the two-dimensional coordinate information of the face to be identified. Then, the depth information of the face to be identified, that is, the depth coordinate information, is obtained through the infrared laser transmitter, and the two-dimensional coordinate information and depth coordinate information of the face to be identified are sent to the image processing chip to obtain the three-dimensional data information of the face to be identified, that is, the three-dimensional point cloud data.

[0038] In a specific implementation, the infrared image and three-dimensional point cloud data of the face to be identified can be obtained from the face information collected by the terminal device with a 3D camera, and can also be obtained from the face information sent by other devices. Among them, the face information includes but is not limited to the collected or received images and video streams.

[0039] Exemplarily, when an infrared image of a face to be identified is obtained from a terminal device with a 3D camera, the image processing chip can perform face detection on the collected image using a face detection algorithm. When a face is detected in the image, the image is output and the image is determined to be an infrared image of the face to be identified. The depth information, i.e., depth coordinate information, of the point cloud data in the infrared image is obtained through an infrared laser transmitter. The two-dimensional coordinate information and depth coordinate information extracted from the infrared image are sent to the image processing chip to obtain point cloud data information (i.e., three-dimensional point cloud data) of the face to be identified. In addition, the infrared image and three-dimensional point cloud data of the face to be identified can also be obtained from a video stream collected by a terminal device with a 3D camera. The face detection algorithm is used to perform face detection on the collected video stream. When a video frame in the video stream is detected to contain a face, the video frame is output and the video frame is determined to be an infrared image of the face to be identified. After detecting that the video frame contains a face, the same operation as the above-mentioned acquisition of the depth coordinate information of the point cloud data is performed on the video frame to obtain the three-dimensional point cloud data of the face to be identified.

[0040] Exemplarily, when obtaining an infrared image and three-dimensional point cloud data of a face to be identified from facial information sent by other devices, first, the image sent by the other device is received, and face detection is performed on the image using the image processing chip in the terminal device of the present application (a face detection algorithm can be used to perform face detection on the image). When a face is detected in the image, the image is output, and it is determined that the image is an infrared image of the face to be identified. Then, the depth information, i.e., the depth coordinate information, of the point cloud data in the infrared image is obtained from the facial information sent by the other device, and the two-dimensional coordinate information and depth coordinate information extracted from the infrared image are sent to the image processing chip in the terminal device of the present application to obtain the point cloud data information, i.e., the three-dimensional point cloud data, of the face to be identified. In addition, the infrared image and three-dimensional point cloud data of the face to be identified can also be obtained from the video streams sent by other devices. First, the video stream sent by other devices is received, and then the image processing chip in the terminal device of this application is used to perform face detection on the video stream. When it is detected that a video frame in the video stream contains a face, the video frame is output, and the video frame is determined to be an infrared image of the face to be identified. After detecting that the video frame contains a face, the same operation as the above-mentioned acquisition of the depth coordinate information of the point cloud data is performed on the video frame to obtain the three-dimensional point cloud data of the face to be identified.

[0041] It should be noted that the format of the video stream obtained above can be an audio video interleaved format (Audio Video Interleaved, AVI), a video player format or a multimedia container format (Matroska, MKV), etc. This application does not limit the format of the video stream obtained.

[0042] It should also be noted that the above-mentioned obtaining the depth information of the point cloud data in the infrared image from the facial information sent from other devices means using the infrared laser transmitter possessed by other devices to obtain the depth information of the point cloud data in the infrared image, and sending the depth information to the terminal device of the present application.

[0043] In the embodiment of the present application, the face detection algorithm may be a feature-based face detection algorithm, and the feature-based face detection algorithm may specifically be: extracting features from the face information acquired by the mobile terminal, matching the features from the extracted face information with the face template image, and then using a classifier to determine whether there is a face, and obtaining a determination result. Among them, the features from the extracted face information are artificially set features (such as deep learning features), and the matching content is the extracted features.

[0044] It should be noted that the face detection algorithm used above can be a feature-based face detection algorithm or an image-based face detection algorithm. The face detection algorithm in this application includes but is not limited to the above detection algorithms.

[0045] Step 102: Determine the target infrared features of the face to be identified based on the infrared image.

[0046] The infrared image is an infrared image containing a face to be identified, and the target infrared features of the face to be identified may refer to facial feature points obtained using a facial feature point positioning algorithm. It should be noted that the present application does not limit the type and number of facial feature points, for example, the five facial feature points may be the left eye, right eye, nose tip, right corner of the mouth, and left corner of the mouth.

[0047] Optionally, determining the target infrared features of the face to be recognized according to the infrared image includes:

[0048] Determine a face image in the infrared image, where the face image refers to an image of a face region in the infrared image;

[0049] Obtain facial feature points in a face image;

[0050] Determine the target infrared features based on the facial feature points.

[0051] In the embodiment of the present application, determining the face image in the infrared image means first determining the face region in the infrared image, and then determining the image in the face region. The face region can be framed in the infrared image in the form of a face frame. Obtaining the face feature points in the face image can be the face feature points in the face image obtained by using a face feature point positioning algorithm, and then determining the target infrared features of the above face image through the face feature points.

[0052] In a specific implementation, determining the face image in the infrared image can be specifically: determining the range of the face area in the infrared image by an image enhancement algorithm, thereby selecting the face image within the face area, and determining the face image in the infrared image. The above-mentioned image enhancement algorithm can be: first converting the acquired infrared image containing the face to be identified into a grayscale image, that is, converting the grayscale value of each pixel in the infrared image containing the face to be identified to the range of [0, 255]. Secondly, extracting the pixels whose grayscale values ​​are at the maximum critical value in the infrared image (for example, the 5% of pixels with the largest grayscale values ​​in the infrared image can be extracted), and performing linear amplification processing so that the average grayscale value of these pixels reaches 255, thereby obtaining the grayscale image of the face area in the infrared image. The grayscale image of the face area can be firstly corroded, and then the corroded grayscale image can be expanded, so that the noise in the grayscale image can be removed, and the face image after the noise is removed can be selected to determine the face image in the infrared image.

[0053] In a specific implementation, the feature point positioning algorithm can be a facial feature point positioning algorithm based on an active shape model (ASM). Before using ASM, it is necessary to train ASM first. Training ASM can be specifically as follows: first constructing an ASM of a face, and selecting N face sample images as a training sample set, and then manually marking M feature points in each face sample image, wherein the M feature points include the features of eyebrows, eyes, nose, mouth, and the outer contour boundary of the face. The coordinates of the feature points marked in each face sample image in the training sample set are strung together into a feature vector, and then a feature vector group of feature points marked in the face sample images in the entire training sample set is obtained, and the face sample images in the training sample set are normalized and aligned. Finally, the aligned features are subjected to principal component analysis to obtain the training network of ASM, and the face image in the infrared image is input into the training network of ASM to obtain the face feature points in the face image. In order to reduce the amount of image calculation, this application selects 5 feature points as manually marked feature points in the sample image, including the left eye, right eye, nose tip, right corner of the mouth and left corner of the mouth of the face.

[0054] It should be noted that the feature point positioning algorithm in the present application can also adopt a facial feature point positioning algorithm based on an active appearance model or based on deep learning, and the present application does not limit the facial feature point positioning method.

[0055] Optionally, determining the target infrared feature according to the facial feature points includes: determining the facial feature points as the target infrared feature.

[0056] In the embodiment of the present application, facial feature points are determined as target infrared features, wherein the facial feature points are facial feature points obtained according to a feature point positioning algorithm, and the target infrared features are infrared features of the face to be identified obtained according to an infrared image. Since the face has texture features and non-rigid shape features, the position information contained in the facial feature points determined by the feature point positioning algorithm for different faces is different. For example, the face image determined according to the infrared image is a two-dimensional image, and the position information contained in the facial feature points is a two-dimensional coordinate data point. Moreover, the facial feature points obtained according to the feature point positioning algorithm are feature points with facial feature information, and the facial feature points can be directly used as infrared features of the face to be identified, that is, target infrared features.

[0057] Step 103: Determine the target three-dimensional point cloud features of the face to be recognized based on the three-dimensional point cloud data.

[0058] In the embodiment of the present application, the three-dimensional point cloud data may be three-dimensional coordinates containing facial feature information. Since the three-dimensional point cloud data is disordered in space, it is impossible to directly extract facial feature points. By preprocessing the three-dimensional point cloud data, standardized three-dimensional point cloud data can be obtained. According to the standardized three-dimensional point cloud data, the target three-dimensional point cloud features of the face to be identified are determined. Among them, the three-dimensional point cloud data refers to 3D point cloud data. Standardized three-dimensional point cloud data refers to mapping disordered point cloud data to a reference face model to obtain three-dimensional point cloud data corresponding to the reference face model.

[0059] In a specific implementation, the preprocessing of the 3D point cloud data includes denoising, reconstruction, and registration. Denoising may refer to removing some outlier point clouds that are obviously not facial features; reconstruction may refer to inputting the denoised 3D point cloud data into a trained 3D face reconstruction network to obtain reconstructed 3D point cloud data; registration may refer to registering the reconstructed 3D point cloud data using an iterative closest point algorithm (ICP) to obtain registered 3D point cloud data. The registered 3D point cloud data is compared with a template face to obtain the compared facial features, and the compared facial features are determined to be the target 3D point cloud features of the face to be identified.

[0060] Step 104, the target infrared feature and the target three-dimensional point cloud feature are fused to obtain a fused feature.

[0061] In an embodiment of the present application, the target infrared feature and the three-dimensional point cloud feature are feature vectors. Fusion of the target infrared feature and the target three-dimensional point cloud feature may be fusion of the feature vectors of the infrared image and the three-dimensional point cloud data to obtain fused features.

[0062] Optionally, fusing the target infrared feature and the target three-dimensional point cloud feature to obtain the fused feature includes: inputting the target infrared feature and the target three-dimensional point cloud feature into a trained fusion network to obtain the fused feature.

[0063] In the embodiment of the present application, the target infrared features and the target three-dimensional point cloud features are both represented in the form of feature vectors. Due to the disorder and sparsity of three-dimensional point cloud data, traditional convolutional neural networks cannot effectively process three-dimensional point cloud data, and the PointNet network structure uses multiple multi-layer perceptrons (Multilayer Perceptron, MLP), which can realize the extraction of N-dimensional features for each point cloud data. Therefore, the present application uses the PointNet network to process the three-dimensional point cloud data, determines that the three-dimensional point cloud data output by the PointNet network is the target three-dimensional point cloud feature, and then inputs the target infrared features and the target three-dimensional point cloud features into the fusion network of the convolutional neural network (Convolutional Neural Networks, CNN) and the PointNet network, and outputs the fusion features of the target infrared features and the target three-dimensional point cloud features. Among them, the above-mentioned fusion network uses a loss function for supervised training.

[0064] Optionally, fusing the target infrared feature and the target three-dimensional point cloud feature to obtain the fused feature further includes: linearly weighting the target infrared feature and the target three-dimensional point cloud feature to obtain the fused feature.

[0065] In the embodiment of the present application, when the external environment has a small impact on the face recognition technology, the target infrared feature can be set to have a higher weight when calculating the fusion feature, and the target three-dimensional point cloud feature can be set to have a lower weight when calculating the fusion feature, so as to reduce the calculation amount of the three-dimensional image. At this time, the target infrared feature determines the final face recognition effect; when the external environment has a greater impact on the face recognition technology, for example, when the angle of the face to be recognized is a non-frontal angle, the target three-dimensional point cloud feature can be set to have a higher weight when calculating the fusion feature. At this time, the acquired target infrared feature contains incomplete face information, while the acquired three-dimensional point cloud feature has more complete face information and is not affected by the angle of the face to be recognized. Therefore, the target three-dimensional point cloud feature is set to have a higher weight when calculating the fusion feature, which can reduce the impact of the external environment on face recognition and improve the accuracy of face recognition. Therefore, when the features are fused, different weights are given to the target infrared feature and the target three-dimensional point cloud feature, so as to ensure that the recognition features have better adaptability.

[0066] In a specific implementation, the target infrared feature is assigned a weight of α1, the target three-dimensional point cloud feature is assigned a weight of α2, and α1+α2=1.

[0067] Step 105, performing face recognition based on the fusion features to obtain a face recognition result.

[0068] In the embodiment of the present application, face recognition based on fusion features may refer to inputting the fusion feature vector of the face to be recognized into the face database, comparing it with the face features in the face database, obtaining the similarity between the face to be recognized and each face in the face database, and returning the face recognition result if the similarity is greater than a preset threshold; if the similarity is less than the preset threshold, the face recognition fails. The face features in the face database are fusion features of infrared features and three-dimensional point cloud features.

[0069] Specifically, the face recognition result may refer to a successful face match or a successful face recognition.

[0070] Exemplarily, when unlocking a mobile phone with a face, the fusion features of the face to be identified are obtained and compared with the fusion features of the preset face in the mobile phone to obtain the similarity between the face to be identified and the preset face. If the similarity is greater than a preset threshold, it indicates that the face recognition is successful, an instruction indicating successful face recognition is returned, and the mobile phone is unlocked.

[0071] This embodiment obtains an infrared image of the face to be identified, determines the target infrared features of the face to be identified, and then determines the target three-dimensional point cloud features of the face to be identified based on the obtained three-dimensional point cloud data. Since the three-dimensional point cloud features fully include the shape and texture features of the face and are not affected by the external environment, the target infrared features and the target three-dimensional point cloud features are fused to obtain fused features, and face recognition is performed based on the fused features, which can reduce the impact of the external environment on face recognition and improve the accuracy of face recognition.

[0072] Reference Figure 2 , shows a flow chart of a face recognition method provided in Embodiment 2 of the present application, the face recognition method is applied to a terminal device, and as shown in the figure, the face recognition method may include the following steps:

[0073] Step 201, obtaining an infrared image and three-dimensional point cloud data of a face to be recognized.

[0074] Step 201 of this embodiment is similar to step 101 of the above-mentioned embodiment, and they can refer to each other, and will not be described in detail in this embodiment.

[0075] Step 202: Determine the target infrared features of the face to be identified based on the infrared image.

[0076] In an embodiment of the present application, in order to improve the accuracy of the target infrared feature, a facial image containing facial feature points can be input into a trained convolutional neural network. When the convolutional neural network converges or the positions of the facial feature points no longer change, the facial feature points at this time are obtained as the target infrared features.

[0077] Optionally, determining the target infrared features based on facial feature points also includes: inputting the facial image into a trained convolutional neural network to obtain the target infrared features.

[0078] In an embodiment of the present application, the convolutional neural network may be a residual network (Residual Network, ResNet) after residual learning is added to a traditional convolutional neural network. Inputting a face image into a convolutional neural network may refer to inputting a face image into a ResNet network structure, processing the face feature points in the face image, thereby outputting the precisely located face feature points to obtain the target infrared features. The ResNet network structure may adopt a ResNet 101 network structure, where 101 refers to the depth of the ResNet network, i.e., the number of layers in the ResNet network that "need to update parameters through training".

[0079] It should be noted that the trained convolutional neural network can use the npair loss loss function for supervised training, or other loss functions for supervised training. This application does not limit the loss function for supervised training.

[0080] Optionally, before inputting the face image into the trained convolutional neural network, it also includes:

[0081] Performing face correction on the face image according to the face feature points to obtain a corrected face image;

[0082] Input the face image into the trained convolutional neural network, and obtain the target infrared features including:

[0083] The corrected face image is input into the trained convolutional neural network to obtain the target infrared features.

[0084] In an embodiment of the present application, since the facial images in the acquired infrared images usually include non-frontal facial images, face correction is performed on the facial images before they are input into the trained convolutional neural network to ensure the accuracy of locating facial feature points.

[0085] Specifically, performing face correction on a face image according to face feature points can be based on the positions of the face feature points, intercepting a face image of the minimum size including the face feature points, and performing slight face twisting and correction on the obtained face image of the minimum size including the face feature points to obtain a corrected face image. The size of the corrected face image can be a 224*224 face image, the corrected face image includes the face feature points obtained according to the feature point positioning, and the size of the corrected face image is smaller than the size of the face image obtained according to the infrared image. The size of the corrected face image is the face image of the minimum size including the face feature points, which can reduce the amount of image calculation.

[0086] Step 203: determine the face point cloud area in the three-dimensional point cloud data according to the face feature points.

[0087] In the embodiment of the present application, the facial feature points are obtained by using a feature point positioning algorithm based on an infrared image. The facial point cloud region in the three-dimensional point cloud data can be determined by taking the nose tip in the facial feature points as the center and r as the radius. After obtaining the facial point cloud region, the three-dimensional point cloud data in the facial point cloud region is determined and preprocessed. Among them, r can refer to the longest distance from the nose tip to the edge of the facial region in the infrared image, and can also refer to the longest distance from the nose tip to the face frame in the infrared image.

[0088] Step 204, denoising the face point cloud data in the face point cloud area to obtain denoised face point cloud data.

[0089] In an embodiment of the present application, denoising the face point cloud data in the face point cloud area may refer to performing bilateral filtering on the face point cloud data in the face point cloud area, thereby smoothing the noise in the face point cloud data while retaining the image edge features.

[0090] Optionally, performing bilateral filtering on the face point cloud data in the face point cloud region includes: obtaining a filtering center of the face point cloud data, wherein the filtering center is any point in the face point cloud data in the face point cloud region;

[0091] Obtaining a neighborhood range of a filtering center, obtaining face point cloud data within the neighborhood range according to the neighborhood range of the filtering center, and calculating a value after filtering the face point cloud data within the neighborhood range of the filtering center;

[0092] Obtain pixel values ​​of the face point cloud data at the filtering center and pixel values ​​of the face point cloud data within the neighborhood of the filtering center, and calculate the value of the face point cloud data pixels after filtering within the neighborhood of the filtering center according to the pixel values ​​obtained above;

[0093] According to the filtered values ​​of the face point cloud data within the filter center neighborhood and the filtered values ​​of the face point cloud data pixels within the filter center neighborhood, the filtered face point cloud data are obtained, and the filtered face point cloud data are determined to be the denoised face point cloud data;

[0094] In the embodiment of the present application, the value of the face point cloud data pixel after filtering within the filter center neighborhood range may be calculated by calculating the distance between the pixel value of the face point cloud data at the filter center and the pixel value of the face point cloud data within the filter center neighborhood range and the brightness difference between the pixels. It can be assumed that the pixel value of the face point cloud data at the filter center is I p , the pixel value of the face point cloud data within the neighborhood is I q , and point q is within the neighborhood.

[0095] In the specific implementation, the face point cloud data in the face point cloud area is subjected to bilateral filtering, and the filtered face point cloud data can be expressed as follows:

[0096]

[0097] Among them, f(·) is a Gaussian filter centered on point cloud p (p is any point in the face point cloud data in the face point cloud area); g(·) is a Gaussian filter centered on point cloud p. p Ω is the size of the filter kernel, i.e. the size of the filter center neighborhood; K p is the sum of the weights of the f(·)*g(·) filters, which is used to normalize the filtered point cloud data; J p is the filtered point cloud. Bilateral filtering is used to denoise the face point cloud data, that is, the point cloud data is denoised from both the spatial domain and the pixel domain, which can not only retain the edge features of the face point cloud area, but also remove the noise in the non-edge features, and realize the smoothing of the non-edge features.

[0098] Step 205: input the denoised face point cloud data into the trained 3D face reconstruction network to obtain reconstructed face point cloud data.

[0099] In an embodiment of the present application, a three-dimensional face reconstruction network is obtained through training. A large number of cameras are used to collect three-dimensional point clouds from different facial angles. The point cloud images from different perspectives collected by the same camera are fused through the DynamicFusion algorithm to obtain a real three-dimensional face model under different cameras, thereby using these large number of three-dimensional face models for supervised training to obtain a reconstruction network.

[0100] In the specific implementation, the denoised facial point cloud data is input into a trained three-dimensional face reconstruction network. When the reconstruction network converges, a three-dimensional face model output by the three-dimensional face reconstruction network is obtained, and the point cloud data extracted from the three-dimensional face model is determined to be the reconstructed facial point cloud data.

[0101] Step 206, registering the reconstructed face point cloud data to obtain registered face point cloud data.

[0102] In the embodiment of the present application, registering the reconstructed face point cloud data may refer to performing ICP registration on the reconstructed face point cloud data. In the present application, the five feature points of the face are mainly used for ICP registration, the rotation parameters and translation parameters of the face are calculated, and then the reconstructed point cloud data are registered according to the rotation parameters and translation parameters to obtain the registered face point cloud data.

[0103] Optionally, the five feature points of the face are used for ICP registration, and the rotation parameters and translation parameters of the face are calculated to include:

[0104] Acquire facial point cloud data to be registered and reference facial point cloud data, wherein the facial point cloud data to be registered is reconstructed facial point cloud data, and the facial point cloud data to be registered is located in a first coordinate system, and the reference facial point cloud data is located in a second coordinate system;

[0105] Obtain reference face point cloud data corresponding to the face point cloud data to be registered in the first coordinate system in the second coordinate system;

[0106] According to the facial point cloud data to be registered and the corresponding reference facial point cloud data, the rotation parameter and the translation parameter between the facial point cloud data to be registered and the reference facial point cloud data are calculated.

[0107] In an embodiment of the present application, when calculating the rotation parameters and translation parameters between the facial point cloud data to be registered and the reference facial point cloud data, the distance between the facial point cloud data to be registered and the reference facial point cloud data can be set as the objective function, and the variable value at which the objective function is minimized is determined to be the rotation parameters and translation parameters between the facial point cloud data to be registered and the reference facial point cloud data.

[0108] In a specific implementation, the calculation method of the rotation parameter and the translation parameter can be expressed as follows:

[0109]

[0110] Among them, input p s is the face point cloud data to be registered, p t It is the reference face point cloud data. is the i-th face point cloud data to be registered, is the reference face point cloud data corresponding to the i-th face point cloud data to be registered, R is the rotation parameter, t is the translation parameter, and these rotation parameters and translation parameters are used to normalize the reconstructed face point cloud data; argmin represents the minimum value of the objective function.

[0111] It should be noted that when performing ICP alignment, the present application does not limit the number of feature points to 5, nor does it limit the feature points to the left eye, right eye, nose tip, right corner of the mouth, and left corner of the mouth of the face. That is, the present application does not limit the type and number of feature points used in ICP alignment.

[0112] In the embodiment of the present application, the reconstructed face point cloud data may be normalized by using the above rotation parameters and translation parameters:

[0113] According to the rotation parameters and the translation parameters, the face point cloud data to be registered is registered to obtain the face point cloud data in the third coordinate system;

[0114] If the average distance between the face point cloud data in the third coordinate system and the reference face point cloud data in the second coordinate system is greater than the preset distance, the face point cloud data in the third coordinate system is used as new face point cloud data to be registered, and iterative calculation is continued;

[0115] If the average distance between the facial point cloud data in the third coordinate system and the reference facial point cloud data in the second coordinate system is less than or equal to the preset distance, the facial point cloud data in the third coordinate system is determined to be the registered facial point cloud data, thereby achieving normalization of the reconstructed facial point cloud data.

[0116] It should be understood that the third coordinate system can be a world coordinate system, and the average distance between the facial point cloud data in the third coordinate system and the reference point cloud data in the second coordinate system refers to the average value of the distance between the facial point cloud data in the third coordinate system and the corresponding reference point cloud data in the second coordinate system.

[0117] It should also be understood that the iterative calculation refers to returning to execute the registration of the face point cloud data to be registered according to the rotation parameters and the translation parameters, and obtaining the face point cloud data in the third coordinate system.

[0118] Step 207, determining the target three-dimensional point cloud features according to the registered face point cloud data.

[0119] In an embodiment of the present application, in order to improve the accuracy of the target three-dimensional point cloud features, the registered facial point cloud data is input into the trained PointNet network. When the trained PointNet network converges, the facial point cloud data at the time of network convergence is determined as the target three-dimensional point cloud features.

[0120] Optionally, before determining that the face point cloud data at the time of network convergence is the target three-dimensional point cloud feature, the method further includes:

[0121] The face point cloud data is normalized when the network converges, and the normalized face point cloud data is determined as the target three-dimensional point cloud feature.

[0122] Specifically, the normalization processing of the face point cloud data when the network converges can be expressed as follows:

[0123] p' x =(2*p x -minp x -maxp x ) / (maxp x -minp x )

[0124] p' y =(2*p y -minp y -maxp y ) / (maxp y -minp y )

[0125] p' z =(2*p z -minp z -maxp z ) / (maxp z -minp z )

[0126] Among them, (p x , p y , p z ) is the three-dimensional coordinate of the face point cloud data extracted when the network converges, minp x , minp y , minp z They are the minimum values ​​of the x, y, and z axis coordinates in the three-dimensional coordinates of the face point cloud data, maxp x , maxp y , maxp z are the maximum values ​​of the x, y, and z axis coordinates in the three-dimensional coordinates of the face point cloud data, p' x , p' y , p' z It is the normalized face point cloud data.

[0127] It should be understood that due to the disorder and sparsity of facial point cloud data, traditional convolutional neural networks cannot effectively process the shape data of point clouds. The PointNet network structure uses multiple MLPs and can realize feature extraction of each point cloud data through shared weight convolution.

[0128] Optionally, determining the target three-dimensional point cloud features according to the registered face point cloud data includes:

[0129] The registered face point cloud data is determined as the target three-dimensional point cloud feature.

[0130] In the embodiment of the present application, since the registered face point cloud data is standard face three-dimensional point cloud data, in order to reduce the computational complexity of feature extraction, the registered face point cloud data can be directly determined as the target three-dimensional point cloud feature.

[0131] Step 208: Fusing the target infrared feature and the target three-dimensional point cloud feature to obtain a fused feature.

[0132] Step 208 of this embodiment is similar to step 104 of the above-mentioned embodiment, and they can refer to each other, and will not be described in detail in this embodiment.

[0133] Step 209, performing face recognition based on the fusion features to obtain a face recognition result.

[0134] Step 209 of this embodiment is similar to step 105 of the above-mentioned embodiment, and they can refer to each other, and will not be described in detail in this embodiment.

[0135] In order to improve the accuracy of the acquired target infrared features and target three-dimensional point cloud features, this embodiment inputs an image containing facial feature points into a trained convolutional neural network, and determines the facial feature points output by the convolutional neural network as the target infrared features; at the same time, the facial point cloud data obtained after registration is input into the trained PointNet network, and the facial point cloud data output by the PointNet network is determined as the target three-dimensional point cloud features. By inputting the target infrared features and the target three-dimensional point cloud features into the trained fusion network, high-precision fusion features can be extracted for face recognition. The above technical solution can improve the accuracy of face recognition technology.

[0136] Reference Figure 3 , shows a schematic diagram of the structure of a face recognition device provided in Embodiment 3 of the present application. For the sake of convenience, only the part related to the embodiment of the present application is shown. The face recognition device may specifically include the following modules:

[0137] An acquisition module 301 is used to acquire infrared images and three-dimensional point cloud data of a face to be recognized;

[0138] A first determination module 302 is used to determine the target infrared features of the face to be identified based on the infrared image;

[0139] The second determination module 303 is used to determine the target three-dimensional point cloud features of the face to be recognized based on the three-dimensional point cloud data;

[0140] A fusion module 304 is used to fuse the target infrared feature and the target three-dimensional point cloud feature to obtain a fusion feature;

[0141] The recognition module 305 is used to perform face recognition according to the fusion features to obtain a face recognition result.

[0142] In the embodiment of the present application, the first determining module 302 may specifically include the following submodules:

[0143] An image determination submodule is used to determine a face image in an infrared image, where the face image refers to an image of a face region in the infrared image;

[0144] A feature acquisition submodule is used to acquire facial feature points in a facial image;

[0145] The feature determination submodule is used to determine the target infrared features based on the facial feature points.

[0146] Optionally, the feature determination submodule may specifically include the following units:

[0147] An infrared determination unit, used to determine facial feature points as target infrared features;

[0148] The image input unit is used to input the face image into the trained convolutional neural network to obtain the target infrared features.

[0149] Optionally, the feature determination submodule further includes the following units:

[0150] A face correction unit, used to perform face correction on the face image according to the face feature points to obtain a corrected face image;

[0151] The above-mentioned image input unit is specifically used to input the corrected face image into the trained convolutional neural network to obtain the target infrared features.

[0152] In the embodiment of the present application, the second determining module 303 may specifically include the following submodules:

[0153] The region determination submodule is used to determine the face point cloud region in the three-dimensional point cloud data according to the face feature points;

[0154] A denoising submodule is used to perform denoising on the face point cloud data in the face point cloud area to obtain denoised face point cloud data;

[0155] The reconstruction submodule is used to input the denoised face point cloud data into the trained 3D face reconstruction network to obtain the reconstructed face point cloud data;

[0156] A registration submodule is used to register the reconstructed face point cloud data to obtain the registered face point cloud data;

[0157] The point cloud determination submodule is used to determine the target three-dimensional point cloud features based on the registered face point cloud data.

[0158] Optionally, the point cloud determination submodule may specifically include the following units:

[0159] A feature determination unit, used to determine the registered face point cloud data as a target three-dimensional point cloud feature;

[0160] The data input unit is used to input the registered face point cloud data into the trained PointNet network to obtain the target three-dimensional point cloud features.

[0161] In the embodiment of the present application, the fusion module 304 may specifically include the following submodules:

[0162] The feature input submodule is used to input the target infrared features and the target three-dimensional point cloud features into the trained fusion network to obtain fusion features;

[0163] The linear weighting submodule is used to linearly weight the target infrared features and the target three-dimensional point cloud features to obtain fusion features.

[0164] The face recognition device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0165] Figure 4 Schematic diagram of the structure of the terminal device provided in the fourth embodiment of the present application. Figure 4 As shown, the terminal device 4 of this embodiment includes: at least one processor 410 ( Figure 4 Only one is shown in the figure) a processor, a memory 420, and a computer program 421 stored in the memory 420 and executable on the at least one processor 410, wherein the processor 410 implements the steps in any of the above-mentioned face recognition method embodiments when executing the computer program 421.

[0166] The terminal device 400 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 410 and a memory 420. Those skilled in the art will appreciate that Figure 4It is only an example of the terminal device 400 and does not constitute a limitation on the terminal device 400. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0167] The processor 410 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0168] In some embodiments, the memory 420 may be an internal storage unit of the terminal device 400, such as a hard disk or memory of the terminal device 400. In other embodiments, the memory 420 may also be an external storage device of the terminal device 400, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 400. Further, the memory 420 may also include both an internal storage unit of the terminal device 400 and an external storage device. The memory 420 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 420 may also be used to temporarily store data that has been output or is to be output.

[0169] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0170] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0171] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0172] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0176] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0177] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A face recognition method, characterized in that: The face recognition method comprises: Obtain infrared images and three-dimensional point cloud data of the face to be identified; Determining target infrared features of the face to be identified based on the infrared image; Determining target three-dimensional point cloud features of the face to be identified based on the three-dimensional point cloud data; Linearly weighting the target infrared feature and the target three-dimensional point cloud feature to obtain a fusion feature, wherein different weights are assigned to the target infrared feature and the target three-dimensional point cloud feature according to the influence of the external environment on face recognition; Performing face recognition according to the fusion features to obtain a face recognition result; Wherein, determining the target three-dimensional point cloud features of the face to be identified according to the three-dimensional point cloud data includes: Determine a face point cloud region in the three-dimensional point cloud data according to the face feature points; Performing bilateral filtering on the face point cloud data in the face point cloud area to obtain denoised face point cloud data, wherein the bilateral filtering includes: obtaining filtered face point cloud data according to the filtered values ​​of the face point cloud data within the neighborhood of the filtering center and the filtered values ​​of the face point cloud data pixels within the neighborhood of the filtering center to obtain denoised face point cloud data, wherein the filtering center is any point in the face point cloud data in the face point cloud area; The target three-dimensional point cloud features are determined based on the denoised face point cloud data.

2. The face recognition method according to claim 1, characterized in that: Determining the target infrared features of the face to be identified according to the infrared image includes: Determine a face image in the infrared image, where the face image refers to an image of a face area in the infrared image; Acquire facial feature points in the facial image; The target infrared features are determined according to the facial feature points.

3. The face recognition method according to claim 2, characterized in that: Determining the target infrared feature according to the facial feature points includes: Determining the facial feature points as the target infrared features; Alternatively, the face image is input into a trained convolutional neural network to obtain the target infrared feature.

4. The face recognition method according to claim 3, characterized in that: Before inputting the face image into the trained convolutional neural network, the method further includes: Performing face correction on the face image according to the face feature points to obtain a corrected face image; The step of inputting the face image into a trained convolutional neural network to obtain the target infrared feature comprises: The corrected face image is input into the trained convolutional neural network to obtain the target infrared feature.

5. The face recognition method according to claim 2, characterized in that: The determining the target three-dimensional point cloud features based on the denoised face point cloud data includes: Inputting the denoised face point cloud data into a trained three-dimensional face reconstruction network to obtain reconstructed face point cloud data; Registering the reconstructed face point cloud data to obtain registered face point cloud data; The target three-dimensional point cloud features are determined according to the registered face point cloud data.

6. The face recognition method according to claim 5, characterized in that: Determining the target three-dimensional point cloud features according to the registered face point cloud data includes: Determining the registered face point cloud data as the target three-dimensional point cloud feature; Alternatively, the registered face point cloud data is input into a trained PointNet network to obtain the target three-dimensional point cloud features.

7. A face recognition device, characterized in that: The face recognition device comprises: An acquisition module is used to acquire infrared images and three-dimensional point cloud data of the face to be recognized; A first determination module, used to determine the target infrared features of the face to be identified based on the infrared image; A second determination module is used to determine the target three-dimensional point cloud features of the face to be identified based on the three-dimensional point cloud data; A fusion module, used for linearly weighting the target infrared feature and the target three-dimensional point cloud feature to obtain a fusion feature, wherein different weights are given to the target infrared feature and the target three-dimensional point cloud feature according to the influence of the external environment on face recognition; A recognition module, used for performing face recognition according to the fusion features to obtain a face recognition result; Wherein, the second determining module is specifically used to: Determine a face point cloud region in the three-dimensional point cloud data according to the face feature points; Performing denoising processing on the face point cloud data in the face point cloud area to obtain denoised face point cloud data, wherein the denoising processing includes: obtaining filtered face point cloud data according to the value of face point cloud data filtered within the neighborhood of the filtering center and the value of face point cloud data pixel filtered within the neighborhood of the filtering center to obtain denoised face point cloud data, wherein the filtering center is any point in the face point cloud data in the face point cloud area; The target three-dimensional point cloud features are determined based on the denoised face point cloud data.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the face recognition method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the face recognition method according to any one of claims 1 to 6 are implemented.

Images