A direction-adaptive multi-pose point cloud face recognition method and system

Abstract

This invention proposes a direction-adaptive multi-pose point cloud face recognition method and system, belonging to the field of face recognition technology. The specific scheme includes: extracting multi-pose features from the point cloud face to be recognized based on an improved LightCNN network; estimating the rotation angle of the face pose under different rotation axes by using the positions of facial key points in the point cloud face to be recognized; using the rotation angle of the face pose under different rotation axes as the rotation coefficients to control the depth residual identity mapping module, mapping the multi-pose features to approximate frontal face features; and performing face recognition using the approximate frontal face features to obtain the recognition result. This invention starts by establishing a mapping relationship between side faces and frontal faces at the feature level, and performs recognition based on approximate frontal face features, thereby improving the recognition capability and robustness of multi-pose faces.

Description

Technical Field

[0001] This invention belongs to the field of face recognition technology, and in particular relates to a method and system for face recognition based on orientation-adaptive multi-pose point cloud. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In unrestricted scenarios, the accuracy and robustness of 3D point cloud face recognition are still affected by various factors, such as occlusion, pose, and facial expression changes. Especially in multi-pose point cloud face recognition tasks, the recognition accuracy drops sharply due to the imbalance between frontal and multi-pose faces in the training set and the potential loss of a large number of feature points during pose changes. Considering the impact of pose changes on point cloud face recognition results, learning geometrically invariant deep features to cope with multi-pose changes remains a challenging task.

[0004] Methods using RGB images as input data typically employ generative adversarial networks (GANs) to generate frontal faces from side-view images for face recognition. The TP-GAN algorithm, a representative of GAN-based methods for generating high-quality frontal face images, utilizes a competitive mechanism between generative and discriminative models. However, because GAN-based frontal face generation methods require learning a large number of parameters, they consume significant computational resources. Methods based on 3D face data generally use fitted 3DMMs as prior knowledge to generate frontal faces from side-view images, such as the Pose Invariant Model (PIM). This model includes two modules: a face frontalization subnetwork and a classification subnetwork. Frontalization of multi-pose faces utilizes a 3D variable model as prior knowledge to generate frontal faces through a GAN. Some algorithms treat multi-pose face recognition as a multi-task learning task (such as MTL), requiring adaptive learning for different tasks, which also suffers from severe computational resource consumption.

[0005] Therefore, existing multi-pose face recognition algorithms have two important problems in the above methods: First, the relationship between side faces and front faces learned by generative adversarial networks makes the two faces similar at the pixel level, rather than directly generating face features for face classification; Second, multi-pose face recognition methods based on generative adversarial networks and multi-task learning have large computational resource overhead and cannot be applied to devices with limited computing resources. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a direction-adaptive multi-pose point cloud face recognition method and system. Starting from establishing the mapping relationship between side faces and front faces at the feature level, it performs recognition based on approximate front face features, thereby improving the recognition capability and robustness of multi-pose faces.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of this invention provides a method for orientation-adaptive multi-pose point cloud face recognition.

[0009] A direction-adaptive multi-pose point cloud face recognition method includes:

[0010] Based on the improved LightCNN network, multi-pose features are extracted from the point cloud face to be identified;

[0011] By using the positions of facial key points in the point cloud face to be identified, the rotation angle of the face pose under different rotation axes can be estimated.

[0012] Using the rotation angle of the face pose under different rotation axes as the rotation coefficient to control the depth residual identity mapping module, multi-pose features are mapped to approximately frontal face features.

[0013] Face recognition is performed using features that approximate a frontal human face, and the recognition results are obtained.

[0014] Furthermore, the point cloud face to be identified undergoes channel preprocessing before being input into the improved LightCNN network to synthesize a three-channel image;

[0015] The first channel uses the gridfit algorithm to fit the point cloud into a surface of z(x,y); the second and third channels contain the azimuth and pitch angles of the surface normal vector obtained from the point cloud in the spherical coordinate system.

[0016] Furthermore, the improved LightCNN network incorporates the C3 module into the LightCNN model and selects a convolution kernel size suitable for 3D synthesized images based on the C3 module.

[0017] Furthermore, the estimation of the rotation angle of the face pose under different rotation axes is specifically as follows:

[0018] A 3D camera is used to acquire point cloud images of the face to be identified. The spatial coordinates of the key points on the face are obtained by using the key point detection function. Based on the position of the key points on the face in space, the rotation angles of different rotation axes required to basically restore the face to a frontal view are calculated.

[0019] Furthermore, the deep residual identity mapping module consists of an activation layer and a fully connected layer, and the input of the deep residual identity mapping module is the extracted multi-pose features;

[0020] The activation layer is used to determine whether rotation is needed to restore the face to a frontal view based on the rotation coefficient.

[0021] The fully connected layer is used to map multi-pose features to approximate frontal face features.

[0022] Furthermore, the depth residual identity mapping module adds a pose loss function to the center loss function to optimize the distance between the approximate frontal face features obtained by the depth residual identity mapping module and the frontal face features.

[0023] The center loss function reduces the intra-class distance, while the pose loss function increases the inter-class distance.

[0024] Furthermore, it also includes a method for generating point cloud faces using 3DMM technology and calculating bending energy to fuse faces with the greatest rigidity difference, thereby further optimizing the generated point cloud faces and constructing a training dataset.

[0025] A second aspect of the present invention provides a direction-adaptive multi-pose point cloud face recognition system.

[0026] An orientation-adaptive multi-pose point cloud face recognition system includes a feature extraction module, an angle estimation module, a feature mapping module, and a face recognition module.

[0027] The feature extraction module is configured to extract multi-pose features from the point cloud face to be identified based on the improved LightCNN network.

[0028] The angle estimation module is configured to estimate the rotation angle of the face pose under different rotation axes by using the positions of facial key points in the point cloud face to be identified.

[0029] The feature mapping module is configured to use the rotation angle of the face pose under different rotation axes as the rotation coefficient to control the depth residual identity mapping module, and map multi-pose features to approximate frontal face features.

[0030] The face recognition module is configured to perform face recognition using near-frontal facial features to obtain the recognition result.

[0031] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a direction-adaptive multi-pose point cloud face recognition method as described in the first aspect of the present invention.

[0032] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a direction-adaptive multi-pose point cloud face recognition method as described in the first aspect of the present invention.

[0033] The above one or more technical solutions have the following beneficial effects:

[0034] This invention proposes a Direction-adaptive Deep Residual Equivariant Mapping Module (DA-REM) for multi-pose 3D point cloud faces. It can map multi-pose faces to frontal faces in the feature dimension, which solves the problem of difficult multi-pose face recognition to a certain extent.

[0035] This invention proposes a pose loss function based on the center loss function, combined with the cross-entropy loss function. By reducing the distance between side faces and frontal faces and increasing the distance between different categories of mapped faces, this invention improves the model's ability to recognize multi-pose faces and its robustness.

[0036] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0037] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0038] Figure 1 This is a flowchart of the method in the first embodiment.

[0039] Figure 2 The diagram shows the improved structural model based on LightCNN for the first embodiment.

[0040] Figure 3 This is a structural diagram of the direction-adaptive depth residual identity mapping module for the first embodiment.

[0041] Figure 4 This is the face generation image for the first embodiment.

[0042] Figure 5 This is a graph showing the relationship between hyperparameters and accuracy in the first embodiment experiment. Detailed Implementation

[0043] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] Learning geometrically invariant deep features is crucial for solving the multi-pose point cloud face recognition problem on low-quality point cloud face datasets with multiple poses. Addressing the issue that existing 3D point cloud face recognition algorithms cannot handle the impact of pose changes on face recognition accuracy and robustness, this invention designs a lightweight and efficient multi-pose point cloud face recognition scheme. It proposes a direction-adaptive depth residual identity mapping module and a point cloud face recognition algorithm based on a pose loss function. First, deep features are extracted using a backbone network based on LightCNN. The estimated face angle is used as a rotation coefficient to control the input of the features extracted by the backbone network into the depth residual identity mapping module, mapping multi-pose face features to frontal face features. Second, the pose loss function is designed to reduce the distance between side-view faces and frontal faces, and to increase the distance between mapped faces of different categories, thereby improving the model's recognition ability and robustness for multi-pose faces. Furthermore, this invention optimizes 3DMM to generate a multi-pose 3D face dataset containing 10,000 identities, providing data for training multi-pose face recognition models. Experimental results on the open-source datasets Bosphorus and Lock3DFace show that this algorithm has better recognition performance in low-quality multi-pose 3D point cloud face recognition tasks compared with current mainstream algorithms.

[0046] Example 1

[0047] In one or more embodiments, a direction-adaptive multi-pose point cloud face recognition method is disclosed, such as... Figure 1 As shown, it includes the following steps:

[0048] Step S1: Based on the improved LightCNN network, extract multi-pose features from the point cloud face to be identified.

[0049] The point cloud face to be identified is obtained by using a high-precision camera such as Kinect. Before being input into the improved LightCNN network, channel preprocessing is performed to synthesize a three-channel image.

[0050] The first channel uses the gridfit algorithm to fit the point cloud into a surface of z(x,y); the second and third channels contain the azimuth and pitch angles of the surface normal vector obtained from the point cloud in the spherical coordinate system.

[0051] The improved LightCNN network is based on the LightCNN network and is designed for feature extraction of 3D faces. It replaces some residual blocks with KC3 modules, which are more suitable for feature extraction of 3D synthesized face images.

[0052] Specifically, LightCNN networks are designed for 2D face recognition problems and can be used to process images with significant texture variations within a small area. However, 3D face surfaces are typically smoother, and filters with larger convolutional kernels are more suitable for extracting features from 3D face data. Therefore, such as... Figure 1 The feature extraction section shown replaces the two MFM residual blocks used in LightCNN with KC3 blocks using larger convolutional kernels. Essentially, it introduces the C3 module into the LightCNN model and selects a kernel size suitable for 3D synthetic images based on the C3 module. To distinguish it from the original C3 module, the processed C3 module is denoted as KC3. Figure 1 As shown, the MFM (Max-Feature-Map) module achieves this through competition, which not only separates noise signals from information signals but also plays a role in feature selection between two feature maps. The KC3 block is used to extract 3D facial feature information. The original MFM residual block has a convolution kernel size of 7×7, padding of 3, and stride of 1. By replacing the convolution kernel size of the original LightCNN residual block, richer feature information on key points can be extracted from the 3D face synthesis image.

[0053] Figure 2 This is a detailed structural diagram of the MFM and KC3 modules in the improved LightCNN network, such as... Figure 2As shown, the improved LightCNN network takes a 3×128×128 three-channel image as input; then, it extracts features through four blocks, with an MFM and pooling layer between each block to modify the feature size. Block 1 contains one KC3 module, with 48 output feature channels. Block 2 contains two KC3 modules, with 96 output channels. Block 3 contains three MFM residual modules, i.e., MFMBlock, with 192 output channels. Block 4 contains four MFM residual modules, i.e., MFMBlock, with 128 output channels. The feature sizes of each block are 48×64×64, 96×32×32, 192×16×16, and 128×16×16, respectively, and the last fully connected layer (FC) extracts features.

[0054] Step S2: Estimate the rotation angle of the face pose under different rotation axes by using the positions of facial key points in the point cloud face to be identified.

[0055] Since 3D camera acquisition devices typically include keypoint detection capabilities, enabling the acquisition of spatial coordinates of facial keypoints, this invention constructs an angle estimation module. This module calculates the facial pose estimate based on the spatial positions of facial feature points, essentially restoring the face to a frontal view using rotation angles along different axes, denoted by θ. x ,θ y ,θ z These represent the rotation angles along the x-axis, y-axis, and z-axis, respectively.

[0056] Step S3: Use the rotation angle of the face pose under different rotation axes as the rotation coefficient to control the depth residual identity mapping module, and map the multi-pose features to the approximate frontal face features.

[0057] Using the concept of feature identity mapping, the feature extraction process of a convolutional neural network can be viewed as mapping an image x∈X to a vector. The function g; if the input image has been transformed, g can be used If the feature map is used to represent it, then It is an equivariant of g; that is, there exists a mapping M. g An identity mapping can be achieved, expressed by the formula:

[0058]

[0059] Existing technologies only consider the face recognition problem based on yaw rotation, assuming that face images obtained around a single rotation axis can partially eliminate the influence of posture changes during preprocessing, such as the ability to essentially restore a frontal face by rotating the roll angle. However, in practical applications, face changes occur simultaneously on multiple coordinate axes, making it impossible to completely eliminate the influence of posture changes during preprocessing. To adaptively learn the posture change mapping relationship of faces at different angles, this invention proposes a depth residual identity mapping module.

[0060] Figure 3 This demonstrates the basic principle of the orientation-adaptive depth residual identity mapping module. The depth residual identity mapping module consists of three sets of activation layers (i.e., M... x M y M y The system consists of activation layers and fully connected layers. The fully connected layers, through linear transformations and weight learning, perform higher-level combinations and abstractions of input features, generating more discriminative feature representations. By combining activation layers and fully connected layers, more complex and abstract feature representations can be gradually learned, thereby improving the expressive power and pattern recognition capabilities of the input data. The input to this module is the last layer of features output from the backbone network (in this embodiment, the improved LightCNN network), i.e., the extracted multi-pose features. An angle estimation module performs nonlinear mapping of the face pose angles under different rotation axes. The smaller the face rotation angle, the smaller the threshold, controlling the feature input to the depth residual identity mapping module. Compared with existing methods, the depth residual identity mapping module in this embodiment can handle more complex pose transformations. Compared with image-level frontal face generation methods, the method proposed in this embodiment is simpler and more efficient.

[0061] Multi-pose point cloud face recognition relies on neural network models to extract effective feature information from face images. The design of the deep residual identity mapping module aims to represent changes in face pose with robust features. If the improved LightCNN network is represented by φ, then the operation of extracting features from φ in image x can be represented by φ(x). Inspired by formula (1), a deep residual identity mapping is proposed to map multi-pose faces to frontal faces in the feature dimension, expressed by the formula:

[0062]

[0063] Where, x f and x p Representing a frontal and profile face, R(θ) x ,θ y ,θ z ) represents the multi-pose face mapping relationship generated by the rotation coefficient through 3D face pose estimation.

[0064] As can be seen from formula (2), the rotation coefficient can highly control the depth residual identity mapping module. When the value of each angle is 0, the depth residual identity mapping module fails and the backbone network directly extracts the frontal face features. When the face starts to rotate, the direction-adaptive depth residual identity mapping module is activated.

[0065] The deep residual identity mapping module adds a pose loss function to the center loss function to optimize the distance between the approximate frontal face features and the frontal face features obtained by the deep residual identity mapping module; the center loss function reduces the intra-class distance, and the pose loss function expands the inter-class distance.

[0066] Specifically, for samples with the same identity (the same person), the center loss function can uniformly transform facial features of different poses into approximate frontal facial features, reducing the intra-class distance. The formula is expressed as follows:

[0067]

[0068] Where, x i For the input sample, c yi For class y i The category center is m, where m is the total number of samples.

[0069] For faces with different identities but the same pose, the features obtained after the deep residual identity mapping module are still similar, and profile faces tend to be matched with other profile faces. Therefore, the inter-class distance still needs to be considered in the optimization process. To solve this problem, a pose loss function is proposed to expand the inter-class distance, and the formula is expressed as follows:

[0070]

[0071] Among them, L c The first term is the center loss function, used to achieve intra-class similarity; the second term is used to achieve inter-class separation; N is the set of classes; c i and c j The centers are the category centers of category i and category j, respectively.

[0072] The weighted sum of the pose loss function and the cross-entropy loss function is used to balance the influence of the two parameters; that is, the overall loss function is expressed as:

[0073] L = L s +λL pos (5)

[0074] The pose loss function during backpropagation for input sample x i The partial derivatives can be calculated using formula (6):

[0075]

[0076] After the feature centers of the j-th class are randomly initialized, the difference between the class centers is calculated according to formula (7).

[0077]

[0078] Where δ is a condition function.

[0079] Set the learning rate to α, and update the class centers as shown in formula (8):

[0080]

[0081] By constraining the pose loss function, similar features within a class can be compressed, and classification performance can be improved by increasing the inter-class distance. In this process, features of the same class are mapped to nearby feature centers, while features of different classes are mapped to more distant feature centers, ultimately achieving efficient face recognition.

[0082] Step S4: Use near-frontal facial features to perform facial recognition and obtain the recognition result.

[0083] In this embodiment, face recognition based on fused features can refer to inputting the approximate frontal facial features of the face to be recognized into the face database, comparing it with the frontal facial features in the face database, obtaining the similarity between the face to be recognized and each face in the face database, and if the similarity is greater than a preset threshold, the face recognition result is returned; if the similarity is less than the preset threshold, the face recognition fails.

[0084] Specifically, the recognition result can refer to a successful face match or a successful face recognition.

[0085] The network model in the above steps needs to be trained using a training dataset. The training dataset is constructed by generating point cloud faces using 3DMM technology and calculating bending energy to fuse faces with the greatest rigidity difference. The generated point cloud faces are then further optimized.

[0086] The size of the training dataset still significantly impacts the accuracy of current 3D face recognition. A lack of datasets prevents the provision of sufficient data for model training and also affects the design of 3D face recognition models. To address this issue, this embodiment employs 3DMM technology to generate point cloud faces. For the generated faces, a method that calculates bending energy to fuse faces with the greatest rigidity difference is used for further optimization.

[0087] Traditional 3DMM is a technique that generates faces with different identities and expressions using principal component analysis (PCA) linear subspace representation; the face point cloud coordinates are represented as x, y, z, i.e., the face geometric coordinates, S = (r i ,g i ,b i ,…,r p ,g p ,b p ) T ∈R 3n The texture of the face is represented by the color information of the point cloud, i = 1, ..., N represents the number of faces, and p = 1, ..., p represents the number of points on each face; a new face can be created by a linear combination of the face shape and texture, as shown in the following formula:

[0088]

[0089] To ensure that the generated face parameters conform to typical human face data and prevent excessive facial variations, the GPMM face generation method, whose parameters follow a Gaussian distribution, is used. Each identity can be divided into two parts: a face shape model and a face expression model, represented as follows:

[0090] M s =(μ s ,σ s U s ),M e =(μ e ,σ e U e (10)

[0091] Where, μ {s,t} ∈R 3m It is the average value, σ {s,t} ∈R n-1 It is the standard deviation, U {s,t} =[u1,...u n ]∈R 3m×n-1 It is an orthogonal basis of the principal components of shape and expression. New faces can be generated from a linear combination of these principal components:

[0092]

[0093] By randomly drawing samples from the above distribution, a new face instance can be represented as:

[0094]

[0095] Since the face generated by the linear method is too smooth, the newly generated face is processed by randomly sampling some point cloud data, calculating the distance between the point and other sampled points, and then calculating the alpha shape parameter of the point based on these distances. The alpha shape is then used to generate a new point cloud, which is then added to the original point cloud face.

[0096] The new face can be represented by the formula F′=alpha(F,h), where h=1, indicating that a new point cloud is generated using an alpha shape with a radius of 1; 10,000 identities were generated from 20,000 independent α and 50 β; the newly generated face uses F i =[x p ,y p ,z p ] T Representation; a method for calculating the rigidity difference of a face, face F i and F j The difference between them can be defined as:

[0097]

[0098] Where, γ ij It is F i Transformed into F j Required bending energy, γ ij The calculation formula is as follows:

[0099] γ(i,j)=x T Bx+y T By+z T Bz (14)

[0100] Where x, y, z are vectors containing coordinates; B is defined as... The bending matrix, S = [1, x j ,y j ,z j ].

[0101] Formula (13) is used to calculate the differences between any two generated faces. The 10k identities with the greatest differences are selected to form a dataset. The face generation process is as follows: Figure 4 As shown, the new face can be represented as:

[0102]

[0103] By calculating the average value of the face with the maximum rigidity difference, a nonlinear variation is introduced into the generated face. The maximum rigidity difference calculation method is then used to compensate for the point cloud face deviation introduced by adding alpha shape details. The generated point cloud face is then converted into a three-channel image, such as... Figure 4As shown, the first column is the depth map, the second and third columns are the normal vectors, and the fourth column is the synthesized RGB image. The first channel uses the GridFT algorithm to fit the point cloud into a surface of z(x,y). The data for the second and third channels are the azimuth and pitch angles of the surface normal vectors obtained from the point cloud in the spherical coordinate system. The azimuth and pitch angles are fitted into surfaces of the form θ(x,y) and φ(x,y) to obtain the image representations of the second and third channels. The pixel values ​​of the three channels are normalized between 0 and 255 to obtain the rendered RGB image.

[0104] experiment

[0105] The experiment in this embodiment was built on the Mxnet framework. The hardware environment was a Tesla V100 graphics card, and the software environment was Ubuntu 18.04, Mxnet-cu101. The weight decay was set to 0.0005, the learning rate was 0.001, the batch size was 32 in the first stage of model training, and the batch size was 300 in the second stage of model training. The input data was adjusted to 128×128 for training.

[0106] Since 3D face datasets typically include spatial information of facial landmarks (e.g., the Lock3DFace and KinectFaces datasets, acquired via Kinect, label the spatial coordinates of five facial landmarks), the pose changes of the face can be obtained by calculating the spatial coordinate relationships. Therefore, during the training phase, the spatial coordinates or angle information of the facial landmarks stored in the dataset are used as the output of the angle estimation module. For unlabeled faces, the face rotation angle can be obtained using the EPnp algorithm. For the calculated face rotation angles around different rotation axes, the results are then processed using... A nonlinear mapping is performed to normalize it to [0,1], thereby controlling the input of the features extracted by the backbone network into the deep residual identity mapping module.

[0107] During the training phase, the model uses the three-channel images generated in step S1 as training data and synthesizes faces with richer pose variations by adjusting the virtual camera parameters. In this embodiment, the yaw angle is in the range of [-60°, 60°], the pitch angle is in the range of [-40°, 40°], and the roll angle is in the range of [-40°, 40°]. A new point cloud face is generated every 20°, and the rotation angle of each point cloud face is saved. The final dataset consists of point cloud face images of 10,000 identities as the training dataset for the first stage of the model. Lock3DFace and KinectFaces are used for fine-tuning. The nose tip is detected by a key point coordinate recognition network, and a 224×224 image is cropped and adjusted to 128×128 before being input into the model for initial training. Among them, all data of 340 categories from 509 individuals are randomly selected from the Lock3DFace dataset for training, and the remaining 169 categories are used as test data. The experiments are carried out on the Bosphorus dataset and the Lock3DFace dataset. Bosphorus is a 3D face dataset collected by a high-precision camera, covering face images of 381 people with various expressions and poses. The dataset provides manually annotated 24 facial feature points for each scanned image, such as the tip of the nose, the center of the chin, and the corners of the eyes. In the Lock3DFace test set, images of the natural face of each individual with yaw angles of ±60°, ±40°, and ±20°, as well as pitch images, are calculated. The pose categories of the original dataset are merged to form the Lock3Dface pose recognition test set, and the rank-one test results are calculated and statistically analyzed.

[0108] Experiments were conducted to verify the effectiveness of the loss function.

[0109] The hyperparameter λ plays a crucial role in adjusting convergence within the network. If it's too large, the model is prone to overfitting; if it's too small, it may fail to converge. An appropriate λ value can balance the influence of the two loss functions, thereby improving the model's prediction accuracy. In this embodiment, six experiments were conducted on the Bosphorus dataset with λ values ​​ranging from 0 to 0.01. The results are as follows... Figure 5 As shown, the experimental results indicate that the model achieves the highest accuracy when λ is set to 0.006. Therefore, in this embodiment, λ is ultimately set to 0.006.

[0110] To verify the effectiveness of pose loss, this embodiment uses LightCNN29 V2 as the backbone network, adds a depth residual identity mapping module, and keeps other parameters unchanged. Three models were trained using different loss functions: (a) Softmax loss + euclidean distance loss; (b) Sofmax loss + center loss; and (c) Sofmax loss + pose loss. The Bosphorus experiment results are shown in Table 1. The proposed scheme in this embodiment improves performance by 2.89 percentage points compared to (a) and 0.77 percentage points compared to (b).

[0111] Table 1. Experimental results of Bosphorus

[0112]

[0113] ablation experiment

[0114] To investigate the impact of each step in the proposed algorithm on the model performance gain, a direction-adaptive deep residual identity mapping (DA-REM) module and a K3 residual block were gradually introduced, and performance metrics were evaluated on the Bosphorus dataset. To ensure the fairness of the experiment, all parameter settings were kept the same, and the experimental results are shown in Table 2. Each module contributed to the improvement of the model accuracy to different degrees, while the direction-adaptive deep residual identity mapping module showed the greatest improvement, with a performance gain of 10.96%. This is mainly due to the deep residual identity mapping module and the KC3 module, which allow the network to extract more useful information. By combining all modules, the average recognition accuracy reached 90.12%.

[0115] Table 2 Comparison of performance gains (%) for each improvement strategy

[0116]

[0117] The proposed method was compared with the 3D face recognition models FR3Dnet and Led3D algorithm on the Lock3DFace dataset. The experimental results are shown in Table 3. The method in this embodiment outperforms the Led3D algorithm by 6.64%. FR3Dnet outperforms the proposed method by 1.09%. Considering that FR3Dnet has 29.4M parameters, the proposed method maintains a high level of recognition accuracy with a lighter footprint, making it suitable for use on resource-constrained devices. In addition, the comparison test set uses depth map and depth map additive vector representations. The average recognition accuracy of images that fuse depth map additive vectors is significantly improved, indicating that these two features are complementary.

[0118] Table 3 shows the rank-one test (%) on the Lock3DFace dataset.

[0119]

[0120]

[0121] This embodiment proposes a direction-adaptive multi-pose point cloud face recognition algorithm. This algorithm uses residual blocks, more suitable for feature extraction from synthesized images of 3D face data, as the backbone network. The input to the residual network is controlled by the face's pose angle, achieving feature-level mapping from multi-pose faces to frontal faces. Furthermore, based on the center loss function, a pose loss function is proposed. This not only reduces the error in mapping from side-view faces to frontal faces but also widens the distance between mapped faces of different categories, improving the model's recognition ability and robustness for multi-pose face recognition.

[0122] Example 2

[0123] In one or more embodiments, a direction-adaptive multi-pose point cloud face recognition system is disclosed, including a feature extraction module, an angle estimation module, a feature mapping module, and a face recognition module:

[0124] The feature extraction module is configured to extract multi-pose features from the point cloud face to be identified based on the improved LightCNN network.

[0125] The angle estimation module is configured to estimate the rotation angle of the face pose under different rotation axes by using the positions of facial key points in the point cloud face to be identified.

[0126] The feature mapping module is configured to use the rotation angle of the face pose under different rotation axes as the rotation coefficient to control the depth residual identity mapping module, and map multi-pose features to approximate frontal face features.

[0127] The face recognition module is configured to perform face recognition using near-frontal facial features to obtain the recognition result.

[0128] Example 3

[0129] The purpose of this embodiment is to provide a computer-readable storage medium.

[0130] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a direction-adaptive multi-pose point cloud face recognition method as described in Embodiment 1 of this disclosure.

[0131] Example 4

[0132] The purpose of this embodiment is to provide an electronic device.

[0133] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of a direction-adaptive multi-pose point cloud face recognition method as described in Embodiment 1 of this disclosure.

[0134] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for multi-pose point cloud face recognition with orientation adaptation, characterized in that, The method comprises the following steps: extracting multi-pose features from a point cloud face to be identified based on an improved LightCNN network; The improved LightCNN network takes a three-channel image with a size of 3*128*128 as input, and then extracts features through four blocks. Each block has an MFM and a pooling layer to change the feature size. Block1 contains one KC3 module, and the output feature channel is 48. Block2 contains two KC3 modules, and the output channel is 96. Block3 contains three MFM residual modules, which are MFM blocks, and the output channel is 192. Block4 contains four MFM residual modules, and the output channel is 128. The feature sizes of each block are 48*64*64, 96*32*32, 192*16*16, and 128*16*16 respectively. Finally, a fully connected layer FC extracts features. The KC3 module is a module that selects a convolution kernel size suitable for three-dimensional synthetic images based on the C3 module. Estimating the rotation angles of the face pose under different rotation axes by the positions of the face key points in the point cloud face to be identified. Using the rotation angles of the face pose under different rotation axes as the rotation coefficients of the depth residual identity mapping module to map the multi-pose features to approximate front face features. The depth residual identity mapping module is composed of an activation layer and a fully connected layer. The input of the depth residual identity mapping module is the extracted multi-pose features. The activation layer is used to determine whether rotation reduction to the front face is needed according to the rotation coefficients. The fully connected layer is used to map the multi-pose features to approximate front face features. The depth residual identity mapping module increases the pose loss function based on the center loss function to optimize the distance between the approximate front face features obtained by the depth residual identity mapping module and the front face features. The center loss function is used to reduce the intra-class distance, and the pose loss function is used to expand the inter-class distance. Using the approximate front face features to perform face recognition to obtain the recognition result.

2. The method of claim 1, wherein the method further comprises: The point cloud face to be identified is preprocessed in the channel before being input into the improved LightCNN network to synthesize a three-channel image. The first channel is a surface z(x, y) fitted by the gridfit algorithm. The second and third channels are the azimuth and pitch angles of the surface normal vector in the spherical coordinate system calculated based on the point cloud.

3. The method of claim 1, wherein the method further comprises: The estimation of the rotation angles of the face pose under different rotation axes is as follows: A three-dimensional camera acquisition device is used to acquire the point cloud face to be identified. The spatial coordinates of the face key points are obtained by using the key point detection function. The different rotation axis rotation angles required to basically restore the front face are calculated based on the positions of the face key points in space.

4. The method of claim 1, wherein the method further comprises: The method also includes generating a point cloud face using the 3DMM technology, calculating the bending energy to fuse the faces with the largest rigid difference, and further optimizing the generated point cloud face to construct a training data set.

5. A directional adaptive multi-pose point cloud face recognition system using the method of claim 1, wherein, The method comprises a feature extraction module, an angle estimation module, a feature mapping module, and a face recognition module. The feature extraction module is configured to extract multi-pose features from the point cloud face to be recognized based on the improved LightCNN network. The angle estimation module is configured to estimate the rotation angle of the face pose under different rotation axes through the positions of the face key points in the point cloud face to be recognized. The feature mapping module is configured to use the rotation angle of the face pose under different rotation axes as the rotation coefficient of the depth residual identity mapping module to map the multi-pose features to the approximate front face features. The face recognition module is configured to perform face recognition using the approximate front face features to obtain the recognition result.

6. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; and a processor for executing the computer readable instructions, wherein the computer readable instructions, when executed by the processor, perform the method of any one of claims 1-4. Instructions that, when executed by a computer, perform the method of any one of claims 1-4.

7. A storage medium characterized by non-transitorily storing computer readable instructions, wherein, ​

Images