Lightweight multi-task face key point detection method and system based on linear vision

Abstract

The invention discloses a multi-task face key point detection method based on a linear vision lightweight model. The method comprises the following steps: acquiring an input image; quickly positioning a face area by using a lightweight face measurement and calculation method, and obtaining a face bounding box BBox; the detected face is preprocessed; the preprocessed image is input into a light weight model FaceMama module; predicting a visibility probability in combination with the coordinates of the key points, and dynamically filtering shielding and low-confidence regions; and outputting the face key point structure description. According to the invention, through deep fusion of a linear vision lightweight architecture and multi-task collaborative optimization, through innovation of three dimensions of a model structure, a training strategy and a loss function, the contradiction of an existing lightweight face recognition algorithm in precision, efficiency and multi-task compatibility is solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and specifically relates to a multi-task facial key point detection method based on a linear vision lightweight model, which is suitable for low-power real-time face analysis scenarios on mobile terminals and vehicle-mounted platforms. Background Art

[0002] Face recognition technology, a key research area in computer vision, has been widely used in recent years in scenarios such as security surveillance, identity verification, and intelligent driving. Traditional face recognition algorithms primarily rely on deep convolutional neural networks (CNNs), such as VGG and ResNet. While these models demonstrate high accuracy in face recognition tasks, their large computational and parameter-intensive nature makes them challenging to implement efficiently on resource-constrained mobile platforms (such as smartphones and tablets) and in-vehicle platforms. To address this issue, researchers have proposed a variety of lightweight models, such as MobileNet and ShuffleNet. These models utilize techniques such as depthwise separable convolutions and channel shuffling to reduce computational complexity, thereby adapting to the computing power of mobile devices. However, these lightweight models often face a significant trade-off between accuracy and speed: on the one hand, over-compressing the model can lead to reduced recognition accuracy; on the other hand, complex training strategies and loss function designs can increase training time and computational overhead. Furthermore, existing face recognition algorithms typically employ single-task training when handling facial landmark detection and visibility classification tasks, making it difficult to simultaneously optimize the performance of multiple tasks. This results in insufficient generalization and robustness in practical applications. Therefore, how to further improve model training efficiency and inference speed while maintaining high accuracy, and achieve multi-task collaborative optimization, has become a key challenge that currently needs to be addressed in lightweight face recognition algorithms.

[0003] In the field of face recognition, existing deep convolutional neural networks (CNNs), such as VGG and ResNet, can achieve high recognition accuracy in face recognition tasks, but they are computationally intensive and require a large number of parameters. This makes it difficult for traditional CNNs to operate efficiently on resource-constrained mobile platforms such as smartphones, tablets, and in-vehicle platforms, and they cannot meet the real-time and low-power requirements of these devices.

[0004] To address these issues, existing technologies have proposed lightweight models such as MobileNet and ShuffleNet. These models, through techniques like depthwise separable convolution and channel shuffling, have reduced their computational complexity to a certain extent, making them suitable for the computing power of mobile devices. However, these lightweight models still have significant drawbacks: First, excessive compression during model structure compression limits the model's learning ability, preventing it from fully extracting complex facial features. This leads to a significant decrease in recognition accuracy in complex scenarios such as those with varying lighting and poses. Second, the use of complex training strategies and loss function designs to improve recognition accuracy significantly increases training time and computational overhead, making it difficult to meet the real-time and low-power requirements of mobile devices. This also increases the hardware performance requirements, severely limiting the expansion of application scenarios.

[0005] Furthermore, existing technologies typically employ a single-task training approach for facial keypoint detection and visibility classification. This approach prevents the model from effectively exploring the correlations and complementarities between different tasks during learning, making it difficult to simultaneously optimize the performance of multiple tasks. Specifically, when dealing with occluded faces, models trained solely for keypoint detection are unable to accurately determine the visibility of keypoints. This results in insufficient generalization and robustness in practical applications, making it difficult to adapt to the complex and ever-changing demands of real-world scenarios. Summary of the Invention

[0006] In order to solve the defects in the prior art, the present invention discloses a multi-task face key point detection method based on a linear visual lightweight model, and its technical solution is as follows:

[0007] A lightweight multi-task face key point detection method based on linear vision is characterized by:

[0008] Step 1: Input image acquisition;

[0009] Step 2: Use a lightweight face measurement method to quickly locate the face area and obtain the face bounding box BBox;

[0010] Step 3: Preprocess the detected face;

[0011] Step 4: Input the preprocessed image into the lightweight model FaceMamba module;

[0012] Step 5: Combine the key point coordinates to estimate the visibility probability and dynamically filter out occlusion and low confidence areas;

[0013] Step 6 outputs the facial key point structure description.

[0014] The present invention also discloses a lightweight multi-task face key point detection system based on linear vision, which is characterized by:

[0015] Preprocessing module: This module performs standardization on the detected face images. It first crops the face area based on the BBox coordinates output by the face detection module and aligns the face based on key reference points through affine transformation to ensure uniform posture. It then scales the cropped and aligned face images to a uniform size of 112×112 pixels and outputs a standardized face image through color space conversion and pixel value normalization.

[0016] Keypoint Detection Module: The FaceMamba face detection module relies on a multi-stage network design and the FSR-OPT feature interaction module. It uses wavelet transform enhancement (WTE-Mamba) to capture high-frequency edge details (such as eyelashes and wrinkles) and multi-kernel deep convolution (MK-DeConv) to extract multi-scale texture features in parallel, while retaining the original feature branches to reduce redundant computation.

[0017] The post-processing module focuses on optimizing model-based outputs. It first receives the raw output of the FaceMamba model, such as keypoint coordinates and visibility probabilities, as a data source. It then uses the "keypoint coordinates - visibility probability" relationship to identify occluded areas and low-confidence prediction points, and then uses a threshold to filter out invalid keypoints. Finally, it integrates the filtered valid keypoint coordinates and outputs a structured description with location and visibility annotations based on the facial topology.

[0018] Beneficial effects

[0019] (1) Based on the BBox coordinates output by the face detection module, the face area is cropped and aligned through affine transformation, and the image is scaled to 112×112 pixels. After color space conversion and pixel normalization, environmental interference is eliminated and a standardized image is output to improve the consistency and robustness of subsequent model inputs.

[0020] (2) The FaceMamba state space model is adopted, and through a multi-stage network and FSR-OPT feature interaction module, it integrates WTE-Mamba high-frequency detail capture, MK-DeConv multi-scale texture extraction and original feature retention to achieve long-distance dependency modeling with low computational complexity and accurately detect facial key points in occluded scenes.

[0021] (3) Receive the coordinates and visibility probability output by the key point detection module, determine the occlusion and low confidence areas through the "coordinate-probability" association, dynamically filter invalid key points, and output structured coordinate data with visibility annotations according to the face topology structure, which is convenient for downstream applications to call.

[0022] (4) Enhance high-frequency details (such as edge texture) through the FSR-Opt module to achieve improved accuracy;

[0023] (5) FLOPs are reduced by 18%, and ARM chip inference reaches 50 FPS (640×480) to achieve efficiency optimization;

[0024] (6) Dynamic filtering strategy integrates key point coordinates and visibility probability to achieve multi-task compatibility;

[0025] (7) Phased training + knowledge distillation shortens the convergence cycle and accelerates training. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0027] FIG2 is a structural diagram of the model of the present invention;

[0028] Figure 3 is an example of feature segmentation of FST-OPT. DETAILED DESCRIPTION

[0029] Example 1

[0030] See also Figure 1 As shown in the figure, the lightweight multi-task face key point detection method based on linear vision includes the following steps:

[0031] Step 1: Input image acquisition: Obtain the original input image (such as RGB format) through a camera, image sensor, or storage device. The supported resolution range is from 640×480 to 1920×1080 and is suitable for various hardware devices such as mobile terminals and vehicle-mounted cameras.

[0032] Step 2: Use a lightweight face measurement method to quickly locate the face area and obtain the face bounding box BBox;

[0033] a) A lightweight face detection algorithm based on YOLOv5s is used to reduce the amount of computation through spatial pyramid pooling (SPP) and depth-wise separable convolution.

[0034] b) Outputs face bounding box (BBox) coordinates and confidence, with a detection speed of 50 FPS on an ARM Cortex-A77 chip (640×480 resolution).

[0035] Step 3: Preprocess the detected face;

[0036] c) Cropping and alignment: Cropping the face area according to the BBox coordinates and using affine transformation to perform face alignment (based on the preset benchmark key point template).

[0037] d) Normalization: The image size is uniformly scaled to 112 × 112 pixels and normalized (pixel values ​​are mapped to the range [-1, 1]) to eliminate the influence of lighting differences.

[0038] Step 4: Input the preprocessed image into the face detection model, see Figure 2 The face detection framework shown is based on the lightweight model FaceMamba module;

[0039] For the input face image, it first passes through the PatchEmbed module, which includes four convolution layers (Conv) (s=2) to obtain the image's embedding vector information. This feature is then sent to the FaceMamba module (D1, D2, D3) in sequence to finally generate the key point heatmap and visibility classification results.

[0040] A detailed explanation of the framework of the face detection model:

[0041] Figure 2 The face detection model framework adopts a coarse-grained network framework with a three-stage hierarchical architecture: it is built with the three-layer FaceMamba backbone modules D1, D2, and D3. By reducing redundant layers, the computational complexity is significantly reduced (FLOPs are reduced by approximately 18%). At the same time, cross-stage feature reuse mechanisms (such as skip connections) are used to compensate for the loss of semantic information caused by layer compression.

[0042] The structure of the core module of FaceMamba is explained in detail:

[0043] Each FaceMamba module contains the following subcomponents:

[0044] DWConv+BN: Depthwise Separable Convolution (DWConv, k=3) extracts local features. This depthwise separable convolution operation is different from traditional convolution. Traditional convolution uses a convolution kernel to convolve all channels of the input feature map at the same time, which is computationally intensive and requires large parameters, and the convolution kernel parameters are not shared. DWConv, on the other hand, uses a separate convolution kernel for each input channel, and is often combined with 1x1 convolution (point convolution) to fuse channel information. Its parameter and computational complexity are mainly related to the number of input channels and the size of the convolution kernel. It can reduce redundant calculations and floating-point operations, reduce the number of parameters and computational complexity, and improve computational efficiency while maintaining certain performance. It is more suitable for resource-constrained mobile and edge devices. It is then combined with batch normalization (BN layer) to accelerate convergence.

[0045] FeedForwardNet: A lightweight feedforward network, a fundamental and widely used artificial neural network architecture, extracts and deepens features, enabling the model to better understand and process input data. Input data first enters the input layer and is then passed to the hidden layer via weights and biases. Nodes in the hidden layer perform a weighted summation of the input and perform nonlinear transformations using activation functions. Finally, the output layer receives the processed signal from the hidden layer and produces the final output. The FeedForward Layer increases the model's expressiveness while also improving training efficiency.

[0046] Frequency-Spatial Receptive Field Optimizer FSR-OPT (FSR-Opt): The proposed efficient multi-receptive field feature interaction (FSR-Opt) module is placed between the symmetric local convolution information perception and each FeedForwardNet.

[0047] The input features of FSR_OPT are divided into three parts along the channel dimension, as follows Figure 3 Shown: Feature input for FSR_OPT , we follow The channel decomposes its dimensions and obtains The features are fed into the WTE-Mamba module and obtained The features are fed into the MK-DeConv module, and the remaining features Feed it into the identity mapping module.

[0048] FSR_OPT achieves global-local feature fusion and high-frequency detail enhancement through channel division and multi-branch collaboration. It includes the following core components:

[0049] 1) Long-range Wavelet Transform Enhanced Mamba (WTE-Mamba): This module uses the wavelet transform to decompose input features into multiple frequency bands. Combined with the long-range modeling capabilities of the state-space model (SSM), it enhances the capture of high-frequency details such as edges and textures, addressing the smoothing effect of traditional convolution in global feature extraction. This module enhances the extraction of high-frequency edge information while performing global modeling. This aims to enhance the ability to extract fine-grained information (such as high-frequency edge details) based on global modeling. Furthermore, convolution operations on WT feature maps achieve larger ERFs and lower computational complexity compared to conventional scales.

[0050] 2) Efficient Multi-kernel Depthwise Separable Convolution (MK-DeConv): This approach uses multi-scale convolution kernels (e.g., 3×3 and 5×5) to extract local features in parallel. This reduces the number of parameters through a depthwise separable structure while expanding the model's ability to perceive occlusion and pose changes. MK-DeConv performs processing, using multiple convolution kernels of different sizes or parameters to extract different features, enhancing perception across different receptive fields.

[0051] 3) De-redundant Identity Mapping: Based on channel pruning and dynamic gating mechanisms, it automatically filters redundant channels in high-dimensional features, reducing feature map computation by 30%-40% while preserving key information transmission paths. This reduces feature redundancy in high-dimensional space and computational complexity, thereby improving processing speed.

[0052] Step 5: Combine key point coordinates to estimate visibility probability and dynamically filter occlusion and low confidence areas

[0053] The lightweight model FaceMamba module is set up with two head outputs, namely the key point regression branch and the visibility classification branch. The output head needs to handle two different types of tasks: regression (key point coordinates) and classification (visibility).

[0054] Keypoint regression branch: Outputs the coordinates of 106 keypoints (continuous values) and uses L2 loss optimization. This branch uses a fully connected layer (FC layer) to output the 2D coordinates of the 106 keypoints (a total of 212 output nodes). A pre-Global Average Pooling (GAP) layer compresses the feature map dimensions to reduce the risk of overfitting.

[0055] The spatial weighted L2 loss function is:

[0056]

[0057] in, is the key point weight, which is dynamically adjusted according to the importance of the region (the key points of the eyes, nose, and corners of the mouth are more important and the weight is set to 1, and the rest of the key points are set to 0.8).

[0058] Visibility classification branch: Outputs the binary classification probability (visible / invisible) for each keypoint, using Focal Loss optimization. A lightweight convolutional layer (1×1 Conv) is used to output the binary classification probability (visible / invisible) for 106 keypoints, using the Sigmoid activation function.

[0059] The Focal Loss optimization loss function is:

[0060]

[0061] Where α=0.8 (negative sample weight) and γ=2 (focus parameter).

[0062] Dynamic filtering strategy: Combine visibility classification probability (threshold > 0.8) with key point confidence to filter key points in low confidence or occluded areas.

[0063] Structural description generation: Outputs the coordinates of facial key points with visibility annotations to form structured data (JSON format), which can be directly used for downstream tasks such as identity verification and posture analysis.

[0064] Step 6 outputs the facial key point structure description.

[0065] Step 7: Lightweight training strategy of Facemamba framework.

[0066] Example 2

[0067] The lightweight multi-task face key point detection system based on linear vision includes the following modules:

[0068] Preprocessing module: This module performs standardization on the detected face images. It first crops the face area based on the BBox coordinates output by the face detection module and aligns the face based on key reference points through affine transformation to ensure uniform posture. It then scales the cropped and aligned face images to a uniform size of 112×112 pixels and outputs a standardized face image through color space conversion and pixel value normalization.

[0069] Keypoint Detection Module: The FaceMamba face detection module relies on a multi-stage network design and the FSR-OPT feature interaction module. It uses wavelet transform enhancement (WTE-Mamba) to capture high-frequency edge details and combines it with multi-kernel deep convolution (MK-DeConv) to extract multi-scale texture features in parallel, while retaining the original feature branches to reduce redundant computations.

[0070] The post-processing module focuses on optimizing model-based outputs. It first receives the raw results, such as keypoint coordinates and visibility probabilities, from the lightweight FaceMamba module as a data source. It then uses the "keypoint coordinates - visibility probability" relationship to identify occluded areas and low-confidence prediction points, filtering out invalid keypoints based on threshold judgment. Finally, it integrates the filtered valid keypoint coordinates and outputs a structured description with location and visibility annotations based on the facial topology.

[0071] Example 3

[0072] A non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein when the program is run, it controls the device where the non-volatile storage medium is located to execute the linear vision-based lightweight multi-task face key point detection method described in Example 1.

[0073] Example 4

[0074] A terminal device, characterized in that the terminal device includes: a processor, a memory, a communication interface and a bus; the processor, the memory and the communication interface are connected via the bus and communicate with each other; the memory stores executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the linear vision-based lightweight multi-task face key point detection method described in Example 1

[0075] The training strategy of this invention achieves efficient model training through multi-task phased collaborative learning optimization, solving the pain points of low training efficiency, weak generalization ability and high inference latency of lightweight models. The specific design is as follows:

[0076] Multi-task phased collaborative learning: To address the gradient conflicts and convergence instability issues that are common in lightweight models in multi-task scenarios, this paper proposes a phased progressive training framework that breaks down complex tasks into ordered optimization goals to achieve efficient collaborative learning. The specific process is divided into two stages:

[0077] Basic task pre-training phase: Prioritize training the model backbone network with a single task (locating 106 key points of the face). By dynamically adjusting regional weights (for example, strengthening the loss weights of easily occluded areas such as the eyes and mouth corners), ensure that the model initially masters highly robust feature extraction capabilities.

[0078] During the multi-task joint fine-tuning phase, a visibility classification branch is introduced based on the pre-trained model. A dynamic task weight allocation strategy is used to balance the optimization of keypoint regression and classification tasks. The weighted classification formula is as follows: L_all = a * L_ketpoint + b * L_visibal. The training tasks are balanced by dynamically adjusting the values ​​of the a and b parameters. a is set to 0.8 and b is set to 0.2.

[0079] The breakthrough of this invention lies in the deep integration of linear vision lightweight architecture and multi-task collaborative optimization. Through innovations in three dimensions: model structure, training strategy and loss function, it solves the contradictions in accuracy, efficiency and multi-task compatibility of existing lightweight face recognition algorithms.

[0080] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A lightweight multi-task facial key point detection method based on linear vision, characterized by: Step 1: Input image acquisition; Step 2: Use a lightweight face measurement method to quickly locate the face area and obtain the face bounding box BBox; Step 3: Preprocess the detected face; Step 4: Input the preprocessed image into the lightweight model FaceMamba module; Step 5: Combine the key point coordinates to estimate the visibility probability and dynamically filter out occlusion and low confidence areas; Step 6 outputs the facial key point structure description; The FaceMamba module structure contains the following subcomponents: a) DWConv+BN: Depthwise separable convolution extracts local features and combines batch normalization to accelerate convergence; b) FeedForwardNet: A lightweight feedforward network responsible for extracting and deepening features, enabling the model to better understand and process input data; through two linear transformations and a nonlinear activation function; c) FSR-OPT: The proposed efficient multi-receptive field feature interaction module is placed between the symmetric local convolution information perception and each FeedForwardNet.

2. The method for detecting facial key points based on linear vision lightweight multi-task method according to claim 1, characterized by: Step 2 further includes the following: using a lightweight face detection algorithm improved based on YOLOv5s, reducing the amount of computation through spatial pyramid pooling (SPP) and depth-wise separable convolution; outputting the face bounding box (BBox) coordinates and confidence level, and achieving a detection speed of 50 FPS on an ARM Cortex-A77 chip.

3. The method for detecting facial key points based on linear vision lightweight multi-task method according to claim 1, characterized by: The step 3 further includes the following contents: cropping and alignment: cropping the face area according to the BBox coordinates and aligning the face using affine transformation; normalization processing: scaling the image size to 112×112 pixels and normalizing it to eliminate the influence of lighting differences.

4. The method for detecting facial key points based on linear vision lightweight multi-task method according to claim 1, characterized by: Step 4 further includes the following: for the input face image, it first passes through four convolution layers (Conv) to obtain the image's embedding vector information, and then sends it to the lightweight model FaceMamba module to finally generate a key point heatmap and visibility classification results.

5. The method for detecting facial key points based on linear vision lightweight multi-task method according to claim 4, characterized by: The features after the four convolution layers Conv are input into the lightweight model FaceMamba module, and then after the depthwise separable convolution (DWConv) and the forward transmission network (FeedForwardNet), they are input into the FSR-opt module. The features of the FSR-opt module are as follows: Assume that the feature input of the FSR-opt module is H*W*C, and its input features are divided into three parts along the channel dimension: 1) The first part of the feature is H*W*ξC, where ξ is an artificial hyperparameter, 0<ξ<1; It has been enhanced with long-range wavelet transform Mamba to enhance the extraction of high-frequency edge information when performing global modeling, enhance the ability to extract fine-grained information based on global modeling, and at the same time, implement convolution operations on WT feature maps; 2) The second part of the feature is H*W*µC, where µ is an artificial hyperparameter and µ< 1-ξ. This is processed through multi-kernel depthwise convolution, using multiple convolution kernels of different sizes or parameters to extract different features. 3) The remaining feature is H*W*(1-ξ-µ)C, which performs identity mapping, reduces feature redundancy in high-dimensional space, reduces computational complexity, and improves processing speed.

6. The method for detecting facial key points based on linear vision lightweight multi-task method according to claim 1, characterized by: The step 5 further includes the following: 1) The lightweight FaceMamba module has two head outputs: a keypoint regression branch and a visibility classification branch. The output heads need to handle two different types of tasks: regression and classification. 2) Keypoint regression branch: Outputs the coordinates of 106 keypoints and uses L2 loss optimization. This branch uses a fully connected layer (FC) to output the two-dimensional coordinates of 106 keypoints and uses a pre-global average pooling layer to compress the feature map dimensions to reduce the risk of overfitting. The spatial weighted L2 loss function is: ； in, is the key point weight, which is dynamically adjusted according to the importance of the region; and are the true value of the key point and the predicted value of the key point respectively; Visibility classification branch: Outputs the binary classification probability of each key point, using Focal Loss optimization; uses a lightweight convolutional layer 1×1 Conv to output the binary classification probability of 106 key points, with a Sigmoid activation function; The Focal Loss optimization loss function is: ； Among them, α is the negative sample weight, γ is the focusing parameter; is the predicted probability of the visibility head classification result; 3) Dynamic filtering strategy: Combining visibility classification probability with keypoint confidence to filter keypoints in low confidence or occluded areas; 4) Structural description generation: Output the coordinates of facial key points with visibility annotations to form structured data.

7. A lightweight multi-task facial key point detection system based on linear vision, characterized by: Preprocessing module: This module performs standardization on the detected face images. It first crops the face area based on the BBox coordinates output by the face detection module and aligns the face based on key reference points through affine transformation to ensure uniform posture. It then scales the cropped and aligned face images to a uniform size of 112×112 pixels and outputs a standardized face image through color space conversion and pixel value normalization. Keypoint Detection Module: The FaceMamba face detection module relies on a multi-stage network design and the FSR-OPT feature interaction module. It uses wavelet transform enhancement (WTE-Mamba) to capture high-frequency edge details and combines it with multi-kernel deep convolution (MK-DeConv) to extract multi-scale texture features in parallel, while retaining the original feature branches to reduce redundant computations. The post-processing module focuses on optimizing model-based outputs. It first receives the raw output of the lightweight FaceMamba module, including keypoint coordinates and visibility probabilities, as a data source. Using the "keypoint coordinate-visibility probability" relationship, it identifies occluded areas and low-confidence prediction points, then filters out invalid keypoints using a threshold. Finally, it integrates the filtered valid keypoint coordinates and outputs a structured description with location and visibility annotations based on the facial topology.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is run, the device where the non-volatile storage medium is located is controlled to execute the linear vision-based lightweight multi-task face key point detection method according to any one of claims 1 to 6.

9. A terminal device, characterized in that: The terminal device includes: a processor, a memory, a communication interface and a bus; the processor, the memory and the communication interface are connected through the bus and communicate with each other; the memory stores executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the linear vision-based lightweight multi-task face key point detection method as described in any one of claims 1 to 6 above.

Images