Lightweight UAV face recognition method based on ensemble learning strategy

Abstract

This invention belongs to the technical field of drone face recognition, specifically a lightweight drone face recognition method based on an ensemble learning strategy. First, drone face images are collected. Next, a lightweight face detection model is constructed and used to extract low-resolution face images from the drone face images. A face super-resolution reconstruction model is constructed and used to extract high-resolution face images from the low-resolution face images. A three-dimensional face reconstruction model is constructed and used to reconstruct face images from the high-resolution face images to generate three-dimensional face reconstructed images. Finally, a lightweight face recognition model (DAMobileFace) based on a dual-attention mechanism and a lightweight face recognition model (CAMFace) based on a CA-attention mechanism are constructed. Face recognition is performed on the high-resolution face images and the three-dimensional face reconstructed images using the DAMobileface and CAMface models, respectively. The recognition results of the two models on the two images are weighted based on an ensemble learning strategy to obtain the final face recognition result. This method solves the problems of low resolution, complex backgrounds, and limited computing resources that exist in drone face recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of face recognition, and in particular is a lightweight unmanned aerial vehicle face recognition method based on an ensemble learning strategy. Background Art

[0002] With the rapid development of science and technology, drone technology is closely integrated with facial recognition and is widely used in fields such as intelligent transportation, video surveillance, security tracking, and border security. However, drones need to maintain a certain distance from their targets during flight and are affected by complex environments. This results in low-quality facial images, complex backgrounds, and small face sizes. Furthermore, drones have limited computing resources, which seriously affects the accuracy of face detection and recognition.

[0003] Current research primarily relies on conventional face recognition models trained directly on drone face datasets. While achieving a certain level of accuracy on specific datasets, this approach fails to fully consider the unique challenges faced by drone face recognition. First, drone-captured facial images typically have low resolution, resulting in a loss of facial details and difficulty extracting effective features. Second, drone shooting angles vary, and facial poses and lighting environments are complex, increasing the difficulty of recognition. Third, drones themselves have limited computing resources, making it difficult to support high-complexity model inference. Conventional face recognition models, such as Facenet and ArcFace, while performing well on high-resolution face images in stable environments, fail to achieve satisfactory results on low-resolution drone face images. Therefore, this paper proposes a lightweight drone face recognition method based on an ensemble learning strategy. Summary of the Invention

[0004] In view of the shortcomings of existing face recognition technology, the technical problem to be solved by the present invention is to propose a lightweight UAV face recognition method based on ensemble learning strategy.

[0005] The present invention solves the technical problem by adopting the following technical solutions:

[0006] A lightweight UAV face recognition method based on an ensemble learning strategy, characterized in that the method comprises the following steps:

[0007] Step 1: Collect drone face images;

[0008] Step 2: Build a lightweight face detection model and use it to extract low-resolution face images from drone face images;

[0009] Step 3: Build a super-resolution face reconstruction model and use it to extract a high-resolution face image from a low-resolution face image.

[0010] Step 4: Build a 3D face reconstruction model, and use the 3D face reconstruction model to reconstruct the face image from the high-resolution face image to generate a 3D face reconstructed image;

[0011] Step 5: Based on the FaceNet network, the backbone network of the FaceNet network is replaced with the MobileNet V2 network. The CBAM module is embedded after the first convolutional block and the last bottleneck block of the MobileNet V2 network. A SENet module is embedded after the depthwise convolution of each bottleneck block to obtain the lightweight face recognition model DAMobileFace based on the dual attention mechanism.

[0012] Based on the MixNets network, a channel shuffle module is embedded after the element-by-element addition operation of each 3×3, 5×5, 7×7MixConv and 3×3, 5×5, 7×7MixConv-d in the MixNets network. A CA attention module is embedded after the first and second 3×3, 5×5, 7×7MixConv and the first 3×3, 5×5, 7×7, 9×9MixConv respectively, to obtain a lightweight face recognition model CAMFace based on the CA attention mechanism;

[0013] The DAMobileface and CAMface models are used to perform face recognition on high-resolution face images and 3D face reconstruction images respectively. It is assumed that the accuracy of the DAMobileface model on the high-resolution face image dataset and the 3D face reconstruction image dataset are A and B respectively. super,DAM and A recon,DAM The accuracy of the CAMface model on the high-resolution face image dataset and the 3D face reconstruction image dataset are A super,CAM and A recon,CAM , the accuracy of each model is standardized according to the following formula;

[0014]

[0015] In the formula, max(·) means taking the maximum value, W super,DAM 、W recon,DAM are the weights of the DAMobileface model on the high-resolution face image dataset and the 3D face reconstruction image dataset, respectively. super,CAM 、W recon,CAM are the weights of the CAMface model on the high-resolution face image dataset and the 3D face reconstruction image dataset;

[0016] Assume that the recognition results of the DAMobileface model on high-resolution face images and 3D face reconstruction images are Rsuper,DAM and R recon,DAM The recognition results of CAMface model on high-resolution face images and 3D face reconstruction images are R super,CAM and R recon,CAM , then the final face recognition result R is calculated by the following formula;

[0017] R=W super,DAM ·R super,DAM +W recon,DAM ·R recon,DAM +W super,CAM ·R super,CAM +W recon,CAM ·R recon,CAM (6)

[0018] Furthermore, the lightweight face detection model is obtained by replacing each CSP module of the YOLO V5 model backbone network one by one using a lightweight cross-scale feature fusion module; the lightweight cross-scale feature fusion module is divided into a main path and a residual path, the main path includes two convolution modules, multiple RepConv modules and a convolution module connected in sequence, and the residual path performs 1×1 convolution on the input features of the module. The output features of the main path and the residual path are added and then subjected to 1×1 convolution to obtain the output features of the lightweight cross-scale feature fusion module.

[0019] Furthermore, the face super-resolution reconstruction model is obtained by replacing each basic residual block of the generator of the SRGAN network one by one with the residual dense block of the fusion attention mechanism; the residual dense block of the fusion attention mechanism introduces the attention module CBAM on the basis of the residual dense block RDB, and the attention module CBAM is located after the 1×1 convolution of the residual dense block RDB, and the output features of the attention module CBAM are residually connected with the input features of the dense residual block RDB to obtain the output features of the residual dense block of the fusion attention mechanism.

[0020] Furthermore, the three-dimensional face reconstruction model performs three-dimensional face reconstruction based on 3DMM, in which a face instance is represented by a pair of shape vectors and texture vectors, and a PCA algorithm is used to describe the spatial distribution of the shape vectors and texture vectors;

[0021] t=t0+B t δ(1)

[0022] s=s0+B id α+B exp β(2)

[0023] Where t and s are shape vector and texture vector respectively, s0 and t0 are average shape vector and average texture vector respectively, B id 、B exp and Bt is the basis matrix of the PCA algorithm, α, β and δ represent the face reconstruction coefficients;

[0024] The face reconstruction coefficient also includes the illumination parameter γ and the posture matrix p. The face reconstruction coefficient is represented by the vector v = (α, β, δ, γ, p). The ResNet-50 network is used to regress the face reconstruction coefficient vector v to obtain a three-dimensional face reconstructed image.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] In the face recognition task based on drone images, due to the complex and changeable environment (for example, different flight altitudes, shooting angles, face angles, lighting conditions, etc.), traditional single recognition methods are difficult to maintain high accuracy in complex and changeable scenes. Therefore, the present invention adopts an ensemble learning strategy to significantly improve the accuracy of face recognition from the drone's perspective by weighting the recognition results of multiple face recognition models on different face images. To address the low resolution and complex background problems of drone images, face super-resolution reconstruction is used to effectively improve the clarity of drone face images; 3D face reconstruction technology is used to restore the 3D face structure from a single image, effectively enhancing facial features. High-resolution face images provide rich 2D face features for face recognition and 3D face reconstruction, while 3D face reconstruction images provide rich 3D face features for face recognition. The synergistic and complementary 2D and 3D face features further improve the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the overall flow chart of the present invention;

[0028] Figure 2 This is a structural diagram of the lightweight cross-scale feature fusion module of the present invention;

[0029] Figure 3 This is a structural diagram of the face super-resolution reconstruction model of the present invention;

[0030] Figure 4 This is a structural diagram of the lightweight face recognition model DAMobileFace based on the dual attention mechanism of the present invention;

[0031] Figure 5 This is a structural diagram of the lightweight face recognition model CAMFace based on the CA attention mechanism of the present invention;

[0032] Figure 6 A comparison chart of face recognition accuracy of different methods. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described in detail below with reference to specific drawings and embodiments, which does not limit the protection scope of the present application.

[0034] The present invention provides a lightweight UAV face recognition method based on ensemble learning strategy (hereinafter referred to as the method, see Figures 1 to 6 ), including the following steps:

[0035] Step 1: Collect drone face images;

[0036] We used DJI drones to capture videos of people in various situations, such as playgrounds, basketball courts, streets, parking lots, and teaching buildings. The drones hovered at a height of 4 to 8 meters. Each video was approximately one minute long, and the videos were processed at 30 FPS, yielding approximately 6,000 drone face images. We used annotation software to manually annotate the face target boxes in each image, and then mapped the annotation files to the faces, generating a drone face image dataset.

[0037] The drone face images collected from the same video are uniformly divided into training sets or test sets, resulting in a training set containing approximately 4,800 images and a test set containing 1,200 images. The face scale of the dataset is concentrated below 60 pixels to reflect the characteristics of face images from the drone's perspective.

[0038] Step 2: Improve the YOLOV5 model, build a lightweight face detection model, and train the model. Use the trained lightweight face detection model to extract low-resolution face images from drone face images.

[0039] A lightweight cross-scale feature fusion module is introduced into the YOLO V5 model. This module replaces each CSP module in the YOLO V5 model's backbone network one by one to further extract and enhance features at each layer and process more abstract semantic information, resulting in a lightweight face detection model. The lightweight cross-scale feature fusion module consists of a main path and a residual path. The main path consists of two sequentially connected convolutional modules, multiple RepConv modules, and a convolutional module. The residual path performs a 1×1 convolution on the input features of this module. The output features of the main and residual paths are added and then subjected to a 1×1 convolution to obtain the output features of the lightweight cross-scale feature fusion module. The lightweight cross-scale feature fusion module can effectively integrate detailed features and contextual information, enhancing the model's feature extraction capabilities for small-scale faces.

[0040] The model training loss is calculated using the Normalized Wasserstein Distance (NWD) loss function. NWD uses Wasserstein distance and normalization design to achieve smoother optimization effects and greater sensitivity to target box scale in object detection tasks. Therefore, the NWD loss function is used instead of the original IoU loss. Compared with the IoU loss, NWD is more sensitive to the relative position and scale differences between the predicted box and the target box, making the optimization process more stable and further improving the accuracy of the face detection model.

[0041] Step 3: Improve the SRGAN network by introducing a residual dense block with an attention mechanism into the SRGAN network. Use the residual dense block with an attention mechanism to replace each basic residual block of the SRGAN network generator one by one to build a face super-resolution reconstruction model; train the face super-resolution reconstruction model and use the trained face super-resolution reconstruction model to extract high-resolution face images from low-resolution face images;

[0042] The residual dense block with fusion attention mechanism introduces the attention module CBAM on the basis of the residual dense block RDB. The attention module CBAM is located after the 1×1 convolution of the residual dense block RDB. The output features of the attention module CBAM are residually connected with the input features of the dense residual block RDB to obtain the output features of the residual dense block with fusion attention mechanism.

[0043] The attention module (CBAM) assigns dynamic weights to different areas of the face image and focuses on the key parts of the face to solve problems such as uneven illumination, noise and resolution differences in drone face images. The attention module CBAM consists of two parts: spatial attention and channel attention. The input features of the dynamic attention module Perform global average pooling and global maximum pooling operations in the spatial dimension to generate channel average pooling features and channel maximum pooling features Channel average pooling features and channel maximum pooling features The attention weight matrix W is calculated by two shared fully connected layers respectively avg and W max ;in, W1 and W2 are learnable weight matrices, σ(·) represents the Sigmoid activation function; the two attention weights W avg and W max Weighted with the input features to obtain the channel attention feature F c ; Channel attention feature F c Perform global average pooling and global maximum pooling operations on the channel dimension to generate spatial average pooling features and spatial maximum pooling features Spatial average pooling features and spatial maximum pooling features After splicing, the spatial attention weight matrix W is obtained through convolution and Sigmoid activation function. s ;in, Conv(·) represents the convolution operation, [;] represents the concatenation operation; the channel attention feature F c and the spatial attention weight matrix W s Perform element-by-element multiplication to obtain the output feature F of the attention module out ;in, Represents an element-wise multiplication operation.

[0044] When training the face super-resolution reconstruction model, we introduce perceptual loss, edge loss, and adversarial loss to optimize the generator's performance. The perceptual loss enhances structural consistency, the edge loss improves contour clarity, and the adversarial loss optimizes the generator's ability to capture high-frequency details. The high-resolution images generated by the face super-resolution reconstruction model have a fourfold increase in resolution compared to the original low-resolution images, enabling better visualization of detailed features and improving image quality.

[0045] Step 4: Build a 3D face reconstruction model, train the 3D face reconstruction model, and use the trained 3D face reconstruction model to reconstruct a face image from the high-resolution face image to generate a 3D face reconstructed image;

[0046] The 3D face reconstruction model is based on the 3DMM (3D face deformation statistical model) to perform 3D face reconstruction. It achieves 3D face reconstruction by regressing the face reconstruction coefficient. In the 3DMM, a complete 3D face is accurately described by a pair of shape vectors t and texture vectors s. The PCA algorithm is used to describe the spatial distribution of shape vectors t and texture vectors s. A face instance can be expressed as:

[0047] t=t0+B t δ(1)

[0048] s=s0+B id α+B exp β(2)

[0049] Where s0 and t0 represent the average shape vector and average texture vector respectively; B id 、B exp and B t is the basis matrix of the PCA algorithm, representing identity, expression, and texture respectively; α, β, and δ represent the 3DMM coefficients, which are the coefficients of face reconstruction to be regressed, and different coefficients represent different faces;

[0050] The parameters to be regressed also include the illumination parameter γ and the posture matrix p. The face reconstruction coefficients to be regressed are represented by the vector v = (α, β, δ, γ, p). The face reconstruction coefficients are regressed using the ResNet-50 network. The face reconstruction coefficients are obtained to obtain a 3D face reconstructed image.

[0051] Image-level loss function and perception-level loss function are introduced to perform weakly supervised learning training on the 3D face reconstruction model to improve the accuracy and robustness of face reconstruction. The image-level loss function focuses on the color information of each pixel in the face image and sparse landmarks, which helps the model effectively handle face reconstruction tasks under occlusion and extreme posture conditions. The image-level loss function is as follows:

[0052]

[0053] Where, L photo (x) is the image-level loss, A i represents the skin attention mask of the i-th image, I i Represents the i-th original image, I′ i (x) represents the i-th 3D face reconstructed image, is the obtainable reprojected face area, ||.|| represents the L2 norm;

[0054] However, relying solely on image-level loss functions may cause the model to fall into a local minimum during the reconstruction process, resulting in inaccurate reconstruction results. To overcome this challenge, a perceptual-level loss function is introduced during training. By calculating the cosine distance between features, the perceptual-level loss function can effectively optimize the texture details and shape accuracy of the reconstructed face, ensuring that the 3D face reconstructed image is more visually realistic and accurate. The perceptual-level loss function is as follows:

[0055]

[0056] Where, L per (x) is the perceptual level loss, f(I) represents the depth feature vector of the original image I, f(I′(x)) represents the depth feature vector of the reconstructed image I′(v), and <·> represents the vector inner product.

[0057] Step 5: Construct a lightweight face recognition model DAMobileFace based on the dual attention mechanism and a lightweight face recognition model CAMface based on the CA attention mechanism. The two models perform face recognition on high-resolution face images and three-dimensional face reconstructed images, respectively, and use an integrated learning strategy to fuse the recognition results of the two models, thereby further improving the recognition accuracy while maintaining a low number of parameters.

[0058] like Figure 4As shown in the figure, the FaceNet network is used as the basis, and the backbone network of the FaceNet network is replaced with the MobileNet V2 network. The CBAM module and SENet module are introduced into the MobileNet V2 network to improve feature extraction capabilities. The CBAM module is embedded after the first convolution block and the last bottleneck block of the MobileNet V2 network, and an SENet module is embedded after the depthwise convolution of each bottleneck block. This results in the lightweight face recognition model DAMobileFace based on the dual attention mechanism. The CBAM module enables the model to focus more on the salient features of the facial area, and the SENet module further improves the network's ability to focus on valid features.

[0059] like Figure 5 As shown in the figure, based on the MixNets network, a channel shuffle module (Channel-shuffle) and a CA attention module are introduced into the MixNets network. A channel shuffle module is embedded after each element-by-element addition operation of 3×3, 5×5, 7×7MixConv and 3×3, 5×5, 7×7MixConv-d. A CA attention module is embedded after the first and second 3×3, 5×5, 7×7MixConv and the first 3×3, 5×5, 7×7, 9×9MixConv, respectively. This results in a lightweight face recognition model CAMFace based on the CA attention mechanism. The channel shuffle module enhances feature expression capabilities, and the CA attention module further highlights key facial features by dynamically adjusting channel weights, allowing the network to focus more on effective information, such as salient areas and detailed features of the face.

[0060] The DAMobileface and CAMface models were trained on high-resolution face image datasets and 3D face reconstruction image datasets at the same time, and the accuracy of the DAMobileface model on the high-resolution face image dataset and 3D face reconstruction image dataset were A and B, respectively. super,DAM and A recon,DAM The accuracy of the CAMface model on the high-resolution face image dataset and the 3D face reconstruction image dataset are A super,CAM and A recon,CAM ;

[0061] Each accuracy rate is normalized according to the following formula and mapped to the interval [0,1] to obtain the weight of each model on the high-resolution face image dataset and the 3D face reconstruction image dataset;

[0062]

[0063] In the formula, max(·) means taking the maximum value, W super,DAM 、W recon,DAM are the weights of the DAMobileface model on the high-resolution face image dataset and the 3D face reconstruction image dataset, respectively. super,CAM 、W recon,CAM are the weights of the CAMface model on the high-resolution face image dataset and the 3D face reconstruction image dataset;

[0064] Assume that the recognition results of the DAMobileface model on high-resolution face images and 3D face reconstruction images are R super,DAM and R recon,DAM The recognition results of CAMface model on high-resolution face images and 3D face reconstruction images are R super,CAM and R recon,CAM , then the final face recognition result R is calculated by the following formula:

[0065] R=W super,DAM ·R super,DAM +W recon,DAM ·R recon,DAM +W super,CAM ·R super,CAM +W recon,CAM ·R recon,CAM (6)

[0066] The final face recognition result R is the weighted combination of the recognition results of the two lightweight face recognition models for different face images. Figure 6The effectiveness of the method of the present invention and the existing method (see "Rostami M, Farajollahi A, Parvin H. Deeplearning-based face detection and recognition on drones [J]. Journal of Ambient Intelligence and Humanized Computing, 2024, 15 (1): 373-387.") in face recognition on drone images is compared. The results show that the face recognition accuracy of the method of the present invention is significantly higher than that of the existing method, improving the face recognition performance from the perspective of drones. The recognition accuracy on three-dimensional face reconstructed images is higher than that on high-resolution face images alone, proving the effectiveness of adding a three-dimensional face reconstruction step after extracting high-resolution face images. The method of the present invention combines the two stages of face super-resolution reconstruction and three-dimensional face reconstruction. Face super-resolution reconstruction provides rich two-dimensional face features for the subsequent three-dimensional face reconstruction and ensemble learning stages, and three-dimensional face reconstruction provides rich three-dimensional face features for subsequent face recognition. Through the synergistic complementarity of two-dimensional face features and three-dimensional face features, the accuracy of drone face recognition is significantly improved.

[0067] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A lightweight UAV face recognition method based on ensemble learning strategy, characterized by: The method comprises the following steps: Step 1: Collect drone face images; Step 2: Build a lightweight face detection model and use it to extract low-resolution face images from drone face images; Step 3: Build a super-resolution face reconstruction model and use it to extract a high-resolution face image from a low-resolution face image. Step 4: Build a 3D face reconstruction model, and use the 3D face reconstruction model to reconstruct the face image from the high-resolution face image to generate a 3D face reconstructed image; Step 5: Based on the FaceNet network, the backbone network of the FaceNet network is replaced with the MobileNetV2 network. The CBAM module is embedded after the first convolution block and the last bottleneck block of the MobileNetV2 network. A SENet module is embedded after the depthwise convolution of each bottleneck block to obtain the lightweight face recognition model DAMobileFace based on the dual attention mechanism. Based on the MixNets network, a channel shuffle module is embedded after the element-by-element addition operation of each 3×3, 5×5, 7×7MixConv and 3×3, 5×5, 7×7MixConv-d in the MixNets network. A CA attention module is embedded after the first and second 3×3, 5×5, 7×7MixConv and the first 3×3, 5×5, 7×7, 9×9MixConv respectively, to obtain a lightweight face recognition model CAMFace based on the CA attention mechanism; The DAMobileface and CAMface models are used to perform face recognition on high-resolution face images and 3D face reconstruction images respectively. It is assumed that the accuracy of the DAMobileface model on the high-resolution face image dataset and the 3D face reconstruction image dataset are A and B respectively. super,DAM and A recon,DAM The accuracy of the CAMface model on the high-resolution face image dataset and the 3D face reconstruction image dataset are A super,CAM and A recon,CAM , the accuracy of each model is standardized according to the following formula; In the formula, max(·) means taking the maximum value, W super,DAM 、W recon,DAM are the weights of the DAMobileface model on the high-resolution face image dataset and the 3D face reconstruction image dataset, respectively. super,CAM 、W recon,CAM are the weights of the CAMface model on the high-resolution face image dataset and the 3D face reconstruction image dataset; Assume that the recognition results of the DAMobileface model on high-resolution face images and 3D face reconstruction images are R super,DAM and R recon,DAM The recognition results of CAMface model on high-resolution face images and 3D face reconstruction images are R super,CAM and R recon,CAM , then the final face recognition result R is calculated by the following formula; R=W super,DAM ·R super,DAM +W recon,DAM ·R recon,DAM +W super,CAM ·R super,CAM +W recon,CAM ·R recon,CAM (6) 2. The lightweight UAV face recognition method based on ensemble learning strategy according to claim 1 is characterized in that: The lightweight face detection model is obtained by replacing each CSP module of the YOLO V5 model backbone network one by one with a lightweight cross-scale feature fusion module. The lightweight cross-scale feature fusion module is divided into a main path and a residual path. The main path includes two convolution modules, multiple RepConv modules and a convolution module connected in sequence. The residual path performs a 1×1 convolution on the input features of the module. The output features of the main path and the residual path are added and then subjected to a 1×1 convolution to obtain the output features of the lightweight cross-scale feature fusion module.

3. The lightweight UAV face recognition method based on ensemble learning strategy according to claim 1 or 2 is characterized in that: The face super-resolution reconstruction model is obtained by replacing each basic residual block of the generator of the SRGAN network one by one with the residual dense block of the fusion attention mechanism; the residual dense block of the fusion attention mechanism introduces an attention module CBAM on the basis of the residual dense block RDB, and the attention module CBAM is located after the 1×1 convolution of the residual dense block RDB. The output features of the attention module CBAM are residually connected with the input features of the dense residual block RDB to obtain the output features of the residual dense block of the fusion attention mechanism.

4. The lightweight UAV face recognition method based on ensemble learning strategy according to claim 3 is characterized in that: The three-dimensional face reconstruction model is based on 3DMM to perform three-dimensional face reconstruction. In the 3DMM, a face instance is represented by a pair of shape vectors and texture vectors, and the PCA algorithm is used to describe the spatial distribution of the shape vectors and texture vectors. t=t0+B t δ(1) s=s0+B id α+B exp β(2) Where t and s are shape vector and texture vector respectively, s0 and t0 are average shape vector and average texture vector respectively, B id 、B exp and B t is the basis matrix of the PCA algorithm, α, β and δ represent the face reconstruction coefficients; The face reconstruction coefficient also includes the illumination parameter γ and the posture matrix p. The face reconstruction coefficient is represented by the vector v = (α, β, δ, γ, p). The ResNet-50 network is used to regress the face reconstruction coefficient vector v to obtain a three-dimensional face reconstructed image.

Images