Multi-task face sensing method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a multi-task face sensing method and device, electronic equipment, a storage medium and a computer program product. The multi-task face sensing method comprises the steps that a to-be-detected face picture is acquired; a to-be-detected face picture is input into the multi-task face perception model, face features and positioning reference points are extracted from the face picture through the multi-task face perception model, and the positioning reference points are used for executing a positioning task about face key point detection; semantic reference points are calculated based on the positioning reference points through a multi-task face perception model, and the semantic reference points are used for executing semantic tasks about face state detection; and executing a positioning task and a semantic task based on the face features, the positioning reference points and the semantic reference points through a multi-task face perception model. Therefore, by explicitly utilizing the physical representation of the face key points, effective supervision for the semantic reference points is realized, the guiding effect of the face key points on the face perception task is enhanced, and the prediction effect of the face perception model can be improved.

Description

Multi-task face perception methods, devices, electronic devices, storage media, and computer program products Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to multi-tasking face perception methods, devices, electronic devices, storage media, and computer program products. Background Technology

[0002] Face perception tasks mainly include "facial landmark detection," "face recognition," "gaze estimation," "head pose estimation," "expression recognition," "age estimation," and "gender estimation," among others. Some face perception-related systems may need to perform one or more of these tasks.

[0003] Because different systems have different timeliness requirements, the execution frequency of face perception tasks varies significantly between different systems. For example, face recognition tasks (such as unlocking a terminal based on face recognition), age estimation tasks, and gender estimation tasks often only need to be executed once within a certain period. For systems such as human-computer interaction systems and driver monitoring systems, which need to continuously perceive the target's psychological state, physiological state, and attention location, the execution frequency of face perception tasks in these systems is relatively high.

[0004] In related technologies, multi-task face perception methods mostly use human understanding of the connections between different perception tasks as priors, and enhance perception performance by strengthening these connections. Furthermore, since facial landmarks are essentially simple geometric representations of the face, the "facial landmark detection task" is often used as an auxiliary task in face perception. However, using the "facial landmark detection task" as an auxiliary task to optimize the model merely leverages the strong dependencies between facial landmarks and other tasks, which leads to poor prediction performance of the face perception model. Summary of the Invention

[0005] This disclosure provides a multi-task face perception method, apparatus, electronic device, storage medium, and computer program product to at least solve the problem of poor prediction performance of face perception models in the aforementioned related technologies.

[0006] According to a first aspect of the present disclosure, a multi-task face perception method is provided, comprising: acquiring a face image to be detected; inputting the face image to be detected into a multi-task face perception model, extracting face features and localization reference points from the face image through the multi-task face perception model, wherein the localization reference points are used to perform a localization task related to face key point detection; calculating semantic reference points based on the localization reference points through the multi-task face perception model, wherein the semantic reference points are used to perform a semantic task related to face state detection; and performing the localization task and the semantic task through the multi-task face perception model based on the face features, the localization reference points, and the semantic reference points.

[0007] Optionally, the multi-task face perception model includes a cross-landmark attention (CLD) module, which is used to measure the correlation between the localization task and the semantic task; the step of calculating semantic reference points based on the localization reference points through the multi-task face perception model includes: calculating the semantic reference points based on the localization reference points through the CLD module.

[0008] Optionally, the step of calculating the semantic reference point based on the positioning reference point by the CLD module includes: calculating a relevance matrix by the CLD module based on the positioning query vector and the semantic query vector contained in the query vector, wherein the positioning query vector is used to characterize the task content of the positioning task, and the semantic query vector is used to characterize the task content of the semantic task; and calculating the semantic reference point by the CLD module based on the relevance matrix and the positioning reference point.

[0009] Optionally, the query vector is the query vector obtained by updating the initial query vector during the training of the multi-task face perception model.

[0010] Optionally, the multi-task face perception model further includes a deformable dual attention (DeDA) module, a localization prediction module, and a semantic prediction module. The DeDA module includes a localization branch and a semantic branch, and each of the localization branch and the semantic branch includes a deformable attention module. The step of performing the localization task and the semantic task based on the face features, the localization reference point, and the semantic reference point using the multi-task face perception model includes: inputting the face features, the localization reference point, and the query vector into the localization branch to obtain an updated query vector; inputting the updated query vector into the localization prediction module to obtain a localization prediction result corresponding to the localization task; inputting the face features, the semantic reference point, and the semantic query vector into the semantic branch to obtain a semantic branch output; and inputting the semantic branch output into the semantic prediction module to obtain a semantic prediction result corresponding to the semantic task.

[0011] Optionally, the multi-task face perception model is trained using the following method: acquiring face sample images; extracting training face features and training localization reference points from the face sample images; inputting the training localization reference points and input query vector into the CLD module to obtain training semantic reference points; inputting the training face features, the training localization reference points, the training semantic reference points, and the input query vector into the DeDA module to obtain the training localization prediction result corresponding to the localization task, the training semantic prediction result corresponding to the semantic task, and the output query vector, wherein the output query vector serves as the input query vector for the next training iteration; calculating the localization loss based on the training localization prediction result and localization ground truth labels; calculating the semantic loss based on the training semantic prediction result and semantic ground truth labels; and adjusting the parameters of the multi-task face perception model according to the localization loss and the semantic loss, thereby training the multi-task face perception model.

[0012] Optionally, the multi-task face perception model includes multiple cascaded DeDA modules.

[0013] Optionally, the semantic task includes at least one of the following: human eye gaze detection task, head pose estimation task, and facial expression recognition task.

[0014] According to a second aspect of the present disclosure, a multi-task face perception device is provided, comprising: an image acquisition module configured to acquire a face image to be detected; a feature extraction module configured to input the face image to be detected into a multi-task face perception model, and extract face features and localization reference points from the face image through the multi-task face perception model, wherein the localization reference points are used to perform a localization task related to face key point detection; a reference point calculation module configured to calculate semantic reference points based on the localization reference points through the multi-task face perception model, wherein the semantic reference points are used to perform a semantic task related to face state detection; and a task execution module configured to execute the localization task and the semantic task through the multi-task face perception model based on the face features, the localization reference points, and the semantic reference points.

[0015] Optionally, the multi-task face perception model includes a cross-landmark attention (CLD) module, which is used to measure the correlation between the localization task and the semantic task; the reference point calculation module is configured to calculate the semantic reference point based on the localization reference point through the CLD module.

[0016] Optionally, the reference point calculation module is configured to: calculate a relevance matrix based on the location query vector and semantic query vector contained in the query vector through the CLD module, wherein the location query vector is used to characterize the task content of the location task, and the semantic query vector is used to characterize the task content of the semantic task; and calculate the semantic reference point based on the relevance matrix and the location reference point through the CLD module.

[0017] Optionally, the query vector is the query vector obtained by updating the initial query vector during the training of the multi-task face perception model.

[0018] Optionally, the multi-task face perception model further includes a deformable dual attention (DeDA) module, a localization prediction module, and a semantic prediction module. The DeDA module includes a localization branch and a semantic branch, and each of the localization branch and the semantic branch includes a deformable attention module. The task execution module is configured to: input the face features, the localization reference point, and the query vector into the localization branch to obtain an updated query vector; input the updated query vector into the localization prediction module to obtain a localization prediction result corresponding to the localization task; input the face features, the semantic reference point, and the semantic query vector into the semantic branch to obtain a semantic branch output; and input the semantic branch output into the semantic prediction module to obtain a semantic prediction result corresponding to the semantic task.

[0019] Optionally, the multi-task face perception model is trained using the following method: acquiring face sample images; extracting training face features and training localization reference points from the face sample images; inputting the training localization reference points and input query vector into the CLD module to obtain training semantic reference points; inputting the training face features, the training localization reference points, the training semantic reference points, and the input query vector into the DeDA module to obtain the training localization prediction result corresponding to the localization task, the training semantic prediction result corresponding to the semantic task, and the output query vector, wherein the output query vector serves as the input query vector for the next training iteration; calculating the localization loss based on the training localization prediction result and localization ground truth labels; calculating the semantic loss based on the training semantic prediction result and semantic ground truth labels; and adjusting the parameters of the multi-task face perception model according to the localization loss and the semantic loss, thereby training the multi-task face perception model.

[0020] Optionally, the multi-task face perception model includes multiple cascaded DeDA modules.

[0021] Optionally, the semantic task includes at least one of the following: human eye gaze detection task, head pose estimation task, and facial expression recognition task.

[0022] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a multi-tasking face perception method according to the present disclosure.

[0023] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a multi-task face perception method according to the present disclosure.

[0024] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the multi-tasking face perception method according to the present disclosure.

[0025] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: In this disclosure, by explicitly utilizing the physical representation of facial key points, effective supervision of semantic reference points is achieved, the guiding role of facial key points in the face perception task is strengthened, and thus the prediction effect of the face perception model can be improved.

[0026] According to an exemplary embodiment of this disclosure, the CLD module can calculate reference point coordinates for non-local tasks, i.e. semantic tasks, without introducing additional parameters, and the implementation process is simple, fast and efficient.

[0027] According to exemplary embodiments of this disclosure, the DeDA Block explicitly separates the localization task from the semantic task. Specifically, the DeDA Block explicitly splits the localization representation and semantic representation of the query vector, reducing interference between features from different tasks and thus improving the prediction accuracy of the multi-task face perception model. Experiments on publicly available datasets demonstrate that, with a significant reduction in the number of parameters and computational cost, the multi-task face perception model of this disclosure achieves prediction accuracy exceeding or approaching that of single-task tasks across all face perception tasks.

[0028] According to exemplary embodiments of this disclosure, pre-training provides a high-quality initial weight for subsequent tasks. These weights already contain a wealth of useful information, offering strong support for the tasks. This allows the model to converge to the optimal solution more quickly during training for subsequent tasks, thereby improving model performance. Furthermore, pre-training reduces the model's dependence on labeled data, lowering training costs.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0031] Figure 1 is a flowchart illustrating a multi-task face perception method according to an exemplary embodiment of the present disclosure; Figure 2 is a structural schematic diagram illustrating a CLD module according to an exemplary embodiment of the present disclosure; Figure 3 is a structural schematic diagram illustrating a DeDA module according to an exemplary embodiment of the present disclosure; Figure 4 is a structural schematic diagram illustrating a deformable attention module in the localization branch included in the DeDA module according to an exemplary embodiment of the present disclosure; Figure 5 is a structural schematic diagram illustrating a multi-task face perception model according to an exemplary embodiment of the present disclosure; Figure 6 is a block diagram illustrating a multi-task face perception device according to an exemplary embodiment of the present disclosure; Figure 7 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0033] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0034] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0035] In related technologies, multi-task face perception methods mostly use the understanding of the relationships between different perception tasks as priors and enhance perception performance by strengthening these relationships. Furthermore, since facial landmarks are simple geometric representations of the face, the "facial landmark detection task" is often used as an auxiliary task in face perception. However, for some perception tasks, not all facial regions contribute equally to the perception results. For example, for the "gaze estimation task," features around the eyes are clearly more important. Using the "facial landmark detection task" as an auxiliary task to optimize the model merely utilizes the strong dependencies between facial landmarks and other tasks. This leads to a lack of supervision regarding semantic-level reference points for face perception tasks, resulting in poor prediction performance of the face perception model.

[0036] To address the aforementioned problems in related technologies, the multi-task face perception method, apparatus, electronic device, storage medium, and computer program product provided in this disclosure achieve effective supervision of semantic reference points by explicitly utilizing the physical representation of facial key points, thereby strengthening the guiding role of facial key points in the face perception task and improving the prediction performance of the face perception model.

[0037] Figure 1 is a flowchart illustrating a multi-task face perception method according to an exemplary embodiment of the present disclosure.

[0038] Referring to Figure 1, in step 101, the face image to be detected can be obtained, that is, an image containing the complete face region can be obtained.

[0039] In step 102, the face image to be detected can be input into the multi-task face perception model, and then the multi-task face perception model can extract face features and localize reference points from the face image. The localization reference points can be used to perform a localization task (Landmark Detection) for face landmark detection.

[0040] As an example, this disclosure can extract multi-level facial features from face images using a backbone network. Exemplarily, the backbone network can progressively extract facial features using a deep convolutional neural network, which specifically includes: low-level feature extraction, i.e., capturing basic visual elements such as edges, corners, and textures through shallow convolutional operations; high-level feature extraction, i.e., progressively combining low-level features through deep networks to form a more semantic representation; and feature abstraction and dimensionality reduction, i.e., compressing high-dimensional image data into low-dimensional, highly expressive feature vectors through multi-layer nonlinear transformations and pooling operations, thereby reducing redundancy while preserving discriminative information.

[0041] In step 103, semantic reference points can be calculated based on the aforementioned localization reference points using a multi-task face perception model. These semantic reference points can be used to perform semantic tasks related to face state detection. For example, the semantic reference points can be facial location points learned autonomously by the network to serve the semantic tasks.

[0042] According to exemplary embodiments of this disclosure, the semantic task of face state detection described above may include at least one of: gaze estimation (GE), head pose estimation (HP), and expression recognition (ER); the localization task of face key point detection described above may include a task for determining the location of key features on the face, wherein the key features on the face may include, but are not limited to: features of facial features, features of the overall facial contour, and features of acupoints on the face. For example, features related to the eyes may include, but are not limited to: features of the corners of the eyes, and features near the eye contour; features related to the eyebrows may include, but are not limited to: features of the overall eyebrow contour, and features of key acupoints on the eyebrows (e.g., Zanzhu acupoint, located at the beginning of the eyebrow; Yuyao acupoint, located in the middle of the eyebrow; Sizhukong acupoint, located at the end of the eyebrow).

[0043] It should be noted that it can be used The query vector representing different face perception tasks can be defined as follows:

[0044] Each of them Both are learnable vectors, or vectors extracted from the model, and their dimensions can be 1xC. This disclosure uses C=256 as an example. The number of tasks in this disclosure . For the first The number of queries for each task. For example, the number of queries for the "localization task", i.e., the "facial landmark detection task", can be the number of landmarks that need to be estimated. For example, in the 300W dataset, the number of landmarks could be 68.

[0045] The number of queries for the "Gaze Detection Task" can be 2, representing the yaw angle and pitch angle of the gaze direction, respectively. Yaw can be used to describe the degree of left and right gaze turning; Pitch can be used to describe the degree of upward or downward gaze turning.

[0046] The "head pose estimation task" can have 3 queries, representing the head rotation pose: yaw, pitch, and roll. Yaw describes the degree of left and right turning of the head in the horizontal direction; pitch describes the degree of tilting the head up or down; and roll describes the degree of tilting the head left and right in the vertical direction.

[0047] The number of queries for the "facial expression recognition task" can be 7, which can include the following expressions: neutral, happy, sad, shocked, fearful, disgusted, and angry.

[0048] In addition, Before being fed into the Deformable Dual Attention (DeDA) module, it can also be... Positional and task encodings are embedded through element-wise addition, which can be specifically represented as follows:

[0049] in, For learnable location encoding, its shape and size are similar to Consistent; For learnable task encoding, its initial shape size can be This learnable task encoding can represent different tasks, and this learnable task encoding They can be shared within the same task. For example, as mentioned earlier, in a "facial landmark detection task," the number of landmarks can be 68, then the learnable task encoding... It can be used 68 times in the "Facial Keypoint Detection Task". For example, it can be used by copying... After dividing, the shape and size are obtained. Consistent Specifically, it can be expressed as:

[0050] in," "This represents the learnable task encoding corresponding to the first face perception task." Represents learnable task encoding It was replicated in the first face recognition task. Second-rate;" "This represents the learnable task encoding corresponding to the second face perception task." Represents learnable task encoding It was replicated in the second face perception task. Second-rate;" " indicates the first The learnable task encoding corresponding to the personal face perception task Represents learnable task encoding In the It was copied in the personal face perception task. Second-rate.

[0051] It's important to note that in Deformable Attention, the query fuses features by learning the biases of sampling points centered on a reference point and the attention weights. Therefore, the location of the reference point is crucial for feature selection and sampling. In localization tasks, the reference point is typically set as the coordinates that the task needs to regress; for example, the coordinates of a keypoint or the location of a bounding box can be set as the coordinates that the task needs to regress. This allows for effective supervision of the reference point using the corresponding ground truth labels, making it easier to sample key locations and thus better optimize the model.

[0052] Semantic tasks typically require regressing semantic labels that contain information about the entire image, such as gaze and facial expression. In these cases, reference points cannot be supervised, making it difficult for deformable attention to find suitable reference points, which limits the prediction performance of face perception models. To address the issue of unsupervised reference points in semantic tasks, this disclosure provides a Cross LanDmark attention layer (CLD) to compute reference points for semantic tasks.

[0053] According to exemplary embodiments of this disclosure, a multi-task face perception model may include a Cross-Landmark Attention (CLD) module, which can be used to measure the correlation between localization and semantic tasks. Semantic reference points can be calculated based on localization reference points using this CLD module.

[0054] According to an exemplary embodiment of this disclosure, a relevance matrix can be calculated by a CLD module based on the location query vector and the semantic query vector contained in the query vector, wherein the location query vector can be used to characterize the task content of the location task, and the semantic query vector can be used to characterize the task content of the semantic task. Then, a semantic reference point can be calculated by the CLD module based on the relevance matrix and the location reference point.

[0055] Figure 2 is a schematic diagram illustrating the structure of a CLD module according to an exemplary embodiment of the present disclosure. Referring to Figure 2, the CLD module can adaptively extract reference points for the semantic task from the reference point coordinates of the supervised localization task by utilizing the correlation between the query of the semantic task and the query of the localization task, wherein the "reference point coordinates of the localization task" are the specific location coordinates of key points on the face.

[0056] Specifically, the CLD module can use the semantic task query (sem_query) as the query vector Q and the localization task query (local_query) as the key vector K. Then, matrix multiplication can be used to obtain the relevance matrix between sem_query and loc_query. Furthermore, the predicted values ​​of the localization task's reference points (i.e., the supervised upper-layer landmark coordinates) can be treated as the value vector V. Matrix multiplication can be directly performed on the aforementioned relevance matrix and the localization task's reference points to obtain the semantic reference points for each sem_query. The semantic task query can include queries from the following tasks: human eye gaze detection, head pose estimation, and facial expression recognition.

[0057] Referring to Figure 2, Wq and Wk are learnable parameters that can be directly reused in the self-attention module of the encoder. "Matmul" represents matrix multiplication, "Scale" represents scaling by a certain ratio, and "Softmax" represents the activation function. It is evident that the CLD module can calculate reference point coordinates for semantic tasks without introducing additional parameters, making the implementation simple.

[0058] It should be noted that, in order to improve the estimation accuracy of the localization task (LD), this disclosure can introduce the query initialization module in Deformable Transformer Landmark Detection (DTLD) as a landmark generator (LDG), which can be specifically represented as follows:

[0059] in, This represents the last layer of features extracted by the backbone (its shape is...). , For the height of the last layer of features, For the width of the last layer feature, The result of flattening the spatial dimension (where the number of channels in the last layer of features) is sized as follows: , LDG in The dimension has R units of fully connected nodes to a landmark, therefore The shape is Furthermore, to minimize feature confusion between different face perception tasks in the backbone, the queries for other face perception tasks remain randomly initialized, learnable vectors.

[0060] According to an exemplary embodiment of this disclosure, the query vector can be a query vector obtained by updating the initial query vector during the training process of a multi-task face perception model, wherein, as mentioned above, the query vector can include a location query vector and a semantic query vector.

[0061] In step 104, a multi-task face perception model can be used to perform localization and / or semantic tasks based on face features, localization reference points, and semantic reference points.

[0062] This disclosure can utilize a multi-task face perception model to simultaneously complete face perception tasks such as localization, eye gaze detection, head pose estimation, and facial expression recognition.

[0063] According to an exemplary embodiment of this disclosure, as described above, the multi-task face perception model may include a deformable dual attention (DeDA) module, a location prediction module, and semantic prediction modules. The DeDA module may include a location branch and a semantic branch, and each of the location branch and the semantic branch may include a deformable attention module.

[0064] Facial features, localization reference points, and query vectors can be input into the localization branch to obtain an updated query vector. This updated query vector is then input into the localization prediction module to obtain the localization prediction result corresponding to the localization task. Conversely, facial features, semantic reference points, and semantic query vectors can be input into the semantic branch to obtain its output. This semantic branch output can then be input into the semantic prediction module to obtain the semantic prediction result corresponding to the semantic task.

[0065] Figure 3 is a schematic diagram illustrating the structure of a DeDA module according to an exemplary embodiment of the present disclosure. Referring to Figure 3, the DeDA module may include a localization branch and a semantic branch, wherein the localization branch is the left branch of the DeDA module, that is, the localization branch may include a deformable attention module and a feed-forward neural network (FFN); the semantic branch is the right branch of the DeDA module, that is, the semantic branch may include sDeformable attention and sFFN. Furthermore, the "s" in "sDeformable attention" and "sFFN" can represent "small," that is, compared to the deformable attention included in the localization branch, the "sDeformable attention" included in the semantic branch has fewer sampling points; compared to the FFN included in the localization branch, the "sFFN" included in the semantic branch has fewer hidden layer channels.

[0066] Furthermore, the localization branch and the semantic branch can share the self-attention module included in the DeDA module. After the query vectors mentioned above undergo the self-attention shared by the localization branch and the semantic branch, all query vectors, i.e., the localization query vectors (the number of which is...), become... ) and semantic query vectors (number of All queries will be sent to the left-hand positioning branch; furthermore, queries belonging to the semantic task, i.e., semantic query vectors (number of which are...) can also be... ) is sent to the semantic branch on the right.

[0067] It should be noted that the localization branch on the left can accept facial features, localization reference points, and query vectors as input. Thus, the deformable attention mechanism within the localization branch can first calculate sampling point biases and attention weights based on the query vector and localization reference points. Specifically, the deformable attention mechanism within the localization branch can calculate the sampling point bias relative to the localization reference point and the corresponding attention weights for each query. The number of sampling points can be multiple, and the "sampling point bias" is the positional bias between the sampling point and the localization reference point. Each sampling point bias also has its own attention weight. Next, the deformable attention mechanism within the localization branch can perform sampling and weighting processing on the facial features based on the calculated sampling point biases and corresponding attention weights. The sampling and weighting result can then be input into the FFN mechanism within the localization branch to obtain an updated query vector. .

[0068] Figure 4 is a schematic diagram illustrating the structure of deformable attention in the localization branch of the DeDA module according to an exemplary embodiment of the present disclosure. Referring to Figure 4, the deformable attention in the localization branch on the left can receive a face feature map, a reference point, and a query feature as input. Thus, the deformable attention can first calculate sampling offsets and attention weights based on the query vector and the reference point. Assuming the user sets three sampling points, each sampling point can have its own attention weight. For example, the three attention weights corresponding to the three sampling offsets in head1 of Figure 4 are 0.5, 0.3, and 0.2; the three attention weights corresponding to the three sampling offsets in head2 of Figure 4 are 0.4, 0.2, and 0.4; and the three attention weights corresponding to the three sampling offsets in head3 of Figure 4 are 0.3, 0.4, and 0.3.

[0069] Next, each of head1, head2 and head3 can perform weighted sampling of facial features in parallel based on the bias of its three sampling points and the corresponding attention weights, thereby obtaining a weighted fusion result (Aggregate). For example, for head1, it can perform weighted sampling of facial features based on the three sample point biases: a, b, and c, and the corresponding three attention weights: 0.5, 0.3, and 0.2, thereby obtaining a weighted fusion result (Aggregate1); for head2, it can perform weighted sampling of facial features based on the three sample point biases: d, e, and f, and the corresponding three attention weights: 0.4, 0.2, and 0.4, thereby obtaining a weighted fusion result (Aggregate2); for head3, it can perform weighted sampling of facial features based on the three sample point biases: g, h, and i, and the corresponding three attention weights: 0.3, 0.4, and 0.3, thereby obtaining a weighted fusion result (Aggregate3).

[0070] Then, the fully connected layer (Linear) contained in the deformable attention in the localization branch on the left can fully connect the above three weighted fusion results: Aggregate1, Aggregate2 and Aggregate3, and thus obtain the output of deformable attention: output.

[0071] Next, the output of deformable attention in the left localization branch can be input into the FFN contained in the left localization branch, thereby obtaining the updated query vector. Then, the updated query vector can be used ( By inputting the location prediction module, you can obtain the location prediction result corresponding to the location task.

[0072] It should be noted that the localization branch on the left is only supervised by the localization task, i.e., the facial landmark detection task. For the semantic task, the purpose of this left-hand localization branch is simply to update the query by regressing appropriate sampling points / attention weights near the reference point.

[0073] Thus, the Deformable attention mechanism included in the DeDA module of this disclosure can adaptively extract the required facial features near the reference point, effectively reducing the training difficulty and improving the facial key point detection effect.

[0074] Furthermore, the semantic branch on the right side of the DeDA module can accept semantic query vectors, i.e., semantic queries, semantic reference points calculated based on localization reference points, and facial features extracted by the backbone network as input. Moreover, the semantic branch can be supervised using only semantic tasks, specifically Gaze Estimation (GE), Head Pose Estimation (HP), and Expression Recognition (ER).

[0075] It should be noted that the regression of sampling points and attention weights in deformable attention prioritizes the accuracy of the regression results. Since the semantic branch on the right only involves semantic reasoning, i.e., the task is relatively simple, a smaller deformable attention and FFN can be used for the semantic branch compared to the localization branch on the left. For example, referring back to Figure 3, the number of sampling points in the sDeformable Attention included in the semantic branch on the right can be half the number of sampling points in deformable attention, and the number of hidden layer channels in the sFFN included in the semantic branch on the right can be one-quarter of the number in the FFN. For instance, the number of sampling points in sDeformable Attention can be 2, and the number of hidden layer channels in the sFFN can be 256.

[0076] Thus, in this disclosure, the DeDA Block explicitly separates the localization task from the semantic task; that is, the DeDA Block explicitly splits the localization representation and semantic representation of the query vector, which can reduce interference between features of different tasks and thus improve the prediction accuracy of the multi-task face perception model. Experiments on public datasets show that the multi-task face perception model of this disclosure, with a significant reduction in the number of parameters and computational cost, achieves prediction accuracy exceeding or approaching that of single-task tasks on all face perception tasks.

[0077] Figure 5 is a schematic diagram illustrating the structure of a multi-task face perception model according to an exemplary embodiment of the present disclosure. Referring to Figure 5, the multi-task face perception model may include a backbone network, a cross-landmark attention layer (CLD), a deformable dual attention block (DeDA), and a multilayer perceptron (MLP), wherein, in this disclosure, the MLP may also be referred to as a "prediction head".

[0078] In Figure 5, the different colored circles represent query vectors for different face perception tasks. For example, they can represent query vectors for four tasks: "facial landmark detection," "gaze estimation," "head pose estimation," and "expression recognition." (x, y) are the coordinates of the reference points calculated by the CLD layer, and these reference points can include both localization and semantic reference points. DeDA modules can exist in a cascaded manner, and the number of cascades can be N.

[0079] It should be noted that the "facial landmark detection task" takes a face image as input and outputs the coordinates of all landmarks in the face image in the image coordinate system. In Figure 5, ( , ), ( , The "Face Key Point Detection Task" takes a face image as input and outputs the category of facial expression, such as happiness, sadness, or surprise. In Figure 5, "Neutral" represents the expression detection result obtained from the "Face Recognition Task." The "Gaze Estimation Task" takes an image containing the eyes as input and outputs yaw and pitch angles representing the gaze direction. In Figure 5, "Yaw" and "Pitch" represent the gaze estimation result obtained from the "Gaze Estimation Task." The "Head Pose Estimation Task" takes a face image as input and outputs yaw, pitch, and roll angles representing the head pose. In Figure 5, "Yaw," "Pitch," and "Roll" represent the head pose estimation result obtained from the "Head Pose Estimation Task."

[0080] In this disclosure, a face image after being processed by the detector can be used as input to the multi-task face perception model, and its size can be 224×224×3. The face image can be processed by the backbone to extract multi-level features; for example, four levels of features can be extracted. The backbone can be based on a Convolutional Neural Network (CNN) or a Transformer Encoder. In this disclosure, a Residual Network (ResNet) is used as an example for the backbone.

[0081] According to an exemplary embodiment of this disclosure, the multi-task face perception model described above can be trained by: acquiring face sample images; then, extracting training face features and training localization reference points from the face sample images; next, inputting the training localization reference points and the input query vector for this training into the CLD module to obtain training semantic reference points. Specifically, the CLD module can be used to calculate the correlation matrix between the input localization query vector and the input semantic query vector contained in the input query vector for this training, and then the CLD module can calculate the training semantic reference points based on the correlation matrix and the training localization reference points.

[0082] It should be noted that the input query vector for this training can be the output query vector of the previous training iteration. A single face sample image can be used for one training iteration. Assuming there are 10,000 face sample images, 10,000 training iterations can be performed. In this case, the input query vector for the current training can be the output query vector of the previous training iteration. For example, if the current training iteration is the 100th iteration, the input query vector for that 100th training iteration can be the output query vector of the previous training iteration, i.e., the 99th training iteration. Furthermore, the output query vector of the first training iteration can be the query vector obtained by updating the initial training query vector using the DeDA module included in the multi-task face perception model. The "initial training query vector" can be a randomly generated, learnable query vector.

[0083] Then, the training face features, training localization reference points, training semantic reference points, and the input query vector for this training can be input into the DeDA module to obtain the training localization prediction result for the localization task, the training semantic prediction result for the semantic task, and the output query vector for this training. The output query vector for this training can be used as the input query vector for the next training.

[0084] Next, the localization loss can be calculated based on the training localization prediction results and localization ground truth labels; the semantic loss can also be calculated based on the training semantic prediction results and semantic ground truth labels. Then, the parameters of the multi-task face perception model can be adjusted according to the localization loss and semantic loss, thereby training the multi-task face perception model.

[0085] According to exemplary embodiments of this disclosure, a multi-task face perception model may include multiple cascaded DeDA modules.

[0086] Referring back to Figure 3, as previously described, the DeDA block can accept facial features, reference points, and a query vector as input. The query vector Q can be updated within each cascaded DeDA module. For example, assuming there are three cascaded DeDA modules, the query vector Q can be updated three times consecutively.

[0087] Specifically, the first DeDA module of the cascaded three DeDA modules receives facial features, localization reference points, semantic reference points, and a query vector. It then uses the localization branch on the left to calculate and update the query vector based on these facial features, localization reference points, and the query vector. And can update the query vector Input the localization prediction module to obtain the localization prediction result corresponding to the localization task. Furthermore, the semantic branch on the right side of the first DeDA module can calculate its output based on facial features, semantic reference points, and the semantic query vector contained in the query vector. This output can then be input into the semantic prediction module to obtain the semantic prediction result corresponding to the semantic task. At this point, the location prediction results can be used as a basis. The localization loss of the first DeDA module is calculated using the ground truth labels and the localization results, and can be based on the semantic prediction results. And the semantic loss of the first DeDA module is calculated using semantic truth labels.

[0088] Next, the location prediction results can be... This serves as a new localization reference point, upon which a new semantic reference point can be calculated. Thus, facial features, the new localization reference point, the new semantic reference point, and the updated query vector are all considered. This can then be used as input to the second DeDA module. In this way, the second DeDA module can use the localization branch on the left to update the query vector based on facial features, the new localization reference point, and the updated query vector. Calculate and update query vector And can update the query vector Input the localization prediction module to obtain the localization prediction result corresponding to the localization task. Furthermore, the semantic branch on the right side of the second DeDA module can be based on facial features, new semantic reference points, and updated query vectors. The included updated semantic query vector is used to calculate the semantic branch output, which can then be input into the semantic prediction module to obtain the semantic prediction result corresponding to the semantic task. At this point, the location prediction results can be used as a basis. The localization loss of the second DeDA module is calculated using the ground truth labels and the localization results, and can be based on the semantic prediction results. And the semantic loss of the second DeDA module is calculated using semantic truth labels.

[0089] Then, the location prediction results can be... This serves as a new localization reference point, upon which a new semantic reference point can be calculated. Thus, facial features, the new localization reference point, the new semantic reference point, and the updated query vector are all considered. This can then be used as input to the third DeDA module. In this way, the third DeDA module can use the localization branch on the left to update the query vector based on facial features, the new localization reference point, and the updated query vector. Calculate and update query vector And can update the query vector Input the localization prediction module to obtain the localization prediction result corresponding to the localization task. Furthermore, the semantic branch on the right side of the third DeDA module can be based on facial features, new semantic reference points, and updated query vectors. The included updated semantic query vector is used to calculate the semantic branch output, which can then be input into the semantic prediction module to obtain the semantic prediction result corresponding to the semantic task. At this point, the location prediction results can be used as a basis. The localization loss of the third DeDA module is calculated using the ground truth labels and the localization results, and can be based on semantic prediction results. And the semantic loss of the third DeDA module is calculated using semantic truth labels.

[0090] Next, the total localization loss can be calculated based on the three localization losses corresponding to the three cascaded DeDA modules; and the total semantic loss can be calculated based on the three semantic losses corresponding to the three cascaded DeDA modules. Then, the parameters of the multi-task face perception model can be adjusted based on the total localization loss and the total semantic loss, thereby training the multi-task face perception model. For example, the total localization loss and the total semantic loss can be directly added together, and the parameters of the multi-task face perception model can be adjusted based on the summation result.

[0091] It should be noted that for LD, GE, and HP, the mean absolute error (L1 loss) can be used to supervise the output of each DeDA module, and each DeDA module can output in a refined manner, that is, each DeDA module only outputs the residual, which needs to be added to the result of the previous DeDA module to obtain the result of this DeDA module. For ER, cross-entropy loss can be used to supervise each DeDA module, and the results of each DeDA module can be independent of each other. In addition, after training the multi-task face perception model, when actually using the trained multi-task face perception model, for each of the four face perception tasks, the result of the last DeDA module in the cascaded multiple DeDA modules can be used as the final output result of the multi-task face perception model.

[0092] Furthermore, since this disclosure utilizes a multi-task learning method guided by facial keypoints, the initial accuracy of these keypoints is crucial. Therefore, this disclosure employs a two-stage training strategy. In the first stage, pre-training can be performed using only LD (Layered Localization), and the modules involved in the pre-training can be a backbone network plus a deformable transformer; that is, the modules involved in the pre-training can be the left-side localization branch contained in the backbone network and the DeDA module. For example, this disclosure can use the LD dataset LaPa for pre-training in the first stage, which has 106 labeled points and an 18k training set. Experimental results demonstrate that better modeling of LD can improve performance across all tasks.

[0093] Next, in the second stage of training, the pre-trained backbone and the left-side localization branch contained in the DeDA module can be loaded first. Then, training can be performed using all face perception tasks and data, meaning that four face perception tasks and data can be used for the second stage of training. It should be noted that, for a fairer comparison, this disclosure uses only the ground truth values ​​for each face perception task as supervision. For example, 300WLP provides LD and HP, but this disclosure can use only the HP label for supervision.

[0094] In this way, pre-training provides a high-quality initial weight for subsequent tasks. These weights already contain a wealth of useful information, offering strong support for the tasks ahead. This allows the model to converge to the optimal solution more quickly during training for later tasks, thus improving model performance. Furthermore, pre-training reduces the model's dependence on labeled data, lowering training costs.

[0095] During the experiment, LD can be verified using the Normalized Mean Square Error (NMSE) metric, the specific formula of which is as follows: Where Y represents the location truth label, that is, the truth label of the key point. This represents the training localization prediction result, specifically the prediction result of key points. The number of key points is represented by D, which is a normalization index that can be calculated based on the location truth label Y.

[0096] In addition, GE can be evaluated using Mean Angular Error; HP can be evaluated using Mean Absolute Error; and ER can be evaluated using accuracy. Multi-task face perception model integrated results. It can be calculated using the following formula:

[0097] in, It represents the number of face recognition tasks. This represents the number of test sets corresponding to the face recognition task t. This is the evaluation result of the multi-task face perception model. This is the evaluation result of a single-task face perception model. Is it better to be as high as possible? ,otherwise, , It can be It can also be used as .

[0098] This disclosure also allows for experiments with different backbones to compare the performance of each module. For example, experimental results using ResNet-18 as the backbone can be compared to those using ResNet-50 as the backbone. Experimental results show that each additional module changes the model's metrics. For example, after adding a "Multi-Task Learning (MTL) baseline model" module to the "Single Task Learning (STL)" module, LD, GE, HP, ER, and... All the indicator values ​​will change; after adding the "CLD" module to the "MTL baseline" module, LD, GE, HP, ER and The values ​​of all indicators will change again, etc.

[0099] Furthermore, experimental results show that the CLD structure can bring certain performance improvements to semantic tasks; although LDG is effective in the original method, its performance improvement is not significant in this disclosure; the DeDA block improves the performance of both localization and semantic tasks, and its overall performance surpasses that of single-task models in different backbone settings without pre-training. After using two-stage training, accuracy is improved for all tasks except ER. Moreover, adding an additional LD ​​dataset (LaPa) for pre-training will bring even greater accuracy improvements, and this configuration can be selected as the default configuration in this disclosure.

[0100] Furthermore, visualization of the effects on DeDA and CLD shows that this disclosure can find relatively accurate reference points and sampling points. For example, in the GE task, it can find the two eyes and surrounding areas; in the ER task, it can find the facial features and surrounding areas; and in the HP task, it focuses more on the overall head region. Moreover, ablation experiments using different LD datasets show that this disclosure is more robust than different LD datasets, but a larger number of keypoints and a larger amount of LD data can still improve the overall accuracy.

[0101] It should be noted that ablation experiments can be conducted using different numbers of DeDA modules. Experimental results show that different face perception tasks have their own optimal number of DeDA modules for achieving the best performance for that task. For example, for LD, a 3-layer decoder yields the best results; however, for GE and HP, if a 3-layer decoder is used, both the reference points and sampling points appear near the image center, and the effective sampling points are few, meaning that no features related to GE / HP are learned when using a 3-layer decoder. Experimental results show that for GE and HP, a decoder layer of 2 yields the best results. Furthermore, experimental results also show that as the number of decoder layers increases, all tasks except ER experience varying degrees of performance degradation, with ER slightly improving, but overall the results remain comparable.

[0102] Figure 6 is a block diagram illustrating a multi-tasking face perception device according to an exemplary embodiment of the present disclosure.

[0103] Referring to Figure 6, the multi-task face perception device 600 may include an image acquisition module 601, a feature extraction module 602, a reference point calculation module 603, and a task execution module 604.

[0104] The image acquisition module 601 can acquire the face image to be detected, that is, to obtain an image containing the complete face region.

[0105] The feature extraction module 602 can input the face image to be detected into a multi-task face perception model, and then the multi-task face perception model can extract face features and localize reference points from the face image. The localization reference points can be used to perform a localization task related to face landmark detection. For example, this disclosure can extract multi-level face features from a face image using a backbone network.

[0106] The reference point calculation module 603 can calculate semantic reference points based on the above-mentioned localization reference points through a multi-task face perception model. The semantic reference points can be used to perform semantic tasks related to face state detection.

[0107] According to exemplary embodiments of this disclosure, the semantic task described above may include at least one of a human eye gaze detection task, a head pose estimation task, and a facial expression recognition task.

[0108] According to an exemplary embodiment of this disclosure, a multi-task face perception model may include a cross-landmark attention (CLD) module, which can be used to measure the correlation between localization tasks and semantic tasks. A reference point calculation module 603 can calculate semantic reference points based on localization reference points through this CLD module.

[0109] According to an exemplary embodiment of this disclosure, the reference point calculation module 603 can calculate a relevance matrix based on the location query vector and semantic query vector contained in the query vector through the CLD module. The location query vector can be used to represent the task content of the location task, and the semantic query vector can be used to represent the task content of the semantic task. Then, the reference point calculation module 603 can calculate semantic reference points based on the relevance matrix and the location reference points through the CLD module. It is evident that the CLD module can calculate reference point coordinates for non-location tasks, i.e., semantic tasks, without introducing additional parameters, and the implementation process is simple, fast, and efficient.

[0110] According to an exemplary embodiment of this disclosure, the query vector can be a query vector obtained by updating the initial query vector during the training of a multi-task face perception model.

[0111] The task execution module 604 can perform localization and / or semantic tasks based on facial features, localization reference points, and semantic reference points using a multi-task face perception model. That is, this disclosure can use a multi-task face perception model to simultaneously complete four face perception tasks: localization, eye gaze detection, head pose estimation, and facial expression recognition.

[0112] According to an exemplary embodiment of this disclosure, the multi-task face perception model described above may further include a deformable dual attention (DeDA) module, a localization prediction module, and a semantic prediction module. The DeDA module may include a localization branch and a semantic branch, and each of the localization branch and the semantic branch may include a deformable attention module.

[0113] The task execution module 604 can input facial features, localization reference points, and query vectors into the localization branch to obtain an updated query vector. Then, the task execution module 604 can input the updated query vector into the localization prediction module to obtain the localization prediction result corresponding to the localization task. Furthermore, the task execution module 604 can input facial features, semantic reference points, and semantic query vectors into the semantic branch to obtain the semantic branch output. Finally, the task execution module 604 can input the semantic branch output into the semantic prediction module to obtain the semantic prediction result corresponding to the semantic task.

[0114] Thus, in this disclosure, the DeDA Block explicitly separates the localization task from the semantic task; that is, the DeDA Block explicitly splits the localization representation and semantic representation of the query vector, which can reduce interference between features of different tasks and thus improve the prediction accuracy of the multi-task face perception model. Experiments on public datasets show that the multi-task face perception model of this disclosure, with a significant reduction in the number of parameters and computational cost, achieves prediction accuracy exceeding or approaching that of single-task tasks on all face perception tasks.

[0115] According to an exemplary embodiment of this disclosure, the multi-task face perception model described above can be trained by: acquiring face sample images; then, extracting training face features and training localization reference points from the face sample images; next, inputting the training localization reference points and the input query vector for this training into the CLD module to obtain training semantic reference points. Specifically, the CLD module can be used to calculate the correlation matrix between the input localization query vector and the input semantic query vector contained in the input query vector for this training, and then the CLD module can calculate the training semantic reference points based on the correlation matrix and the training localization reference points.

[0116] It should be noted that the input query vector for this training can be the output query vector of the previous training iteration. A single face sample image can be used for one training iteration. Assuming there are 10,000 face sample images, 10,000 training iterations can be performed. In this case, the input query vector for the current training can be the output query vector of the previous training iteration. For example, if the current training iteration is the 100th iteration, the input query vector for that 100th training iteration can be the output query vector of the previous training iteration, i.e., the 99th training iteration. Furthermore, the output query vector of the first training iteration can be the query vector obtained by updating the initial training query vector using the DeDA module included in the multi-task face perception model. The "initial training query vector" can be a randomly generated, learnable query vector.

[0117] Then, the training face features, training localization reference points, training semantic reference points, and the input query vector for this training can be input into the DeDA module to obtain the training localization prediction result for the localization task, the training semantic prediction result for the semantic task, and the output query vector for this training. The output query vector for this training can be used as the input query vector for the next training.

[0118] Next, the localization loss can be calculated based on the training localization prediction results and localization ground truth labels; the semantic loss can also be calculated based on the training semantic prediction results and semantic ground truth labels. Then, the parameters of the multi-task face perception model can be adjusted according to the localization loss and semantic loss, thereby training the multi-task face perception model.

[0119] According to exemplary embodiments of this disclosure, a multi-task face perception model may include multiple cascaded DeDA modules.

[0120] Furthermore, since this disclosure utilizes a multi-task learning method guided by facial keypoints, the initial accuracy of these keypoints is crucial. Therefore, this disclosure employs a two-stage training strategy. In the first stage, pre-training can be performed solely using the localization task (LD), and the modules involved in the pre-training can be a backbone network plus a deformable transformer (DeDA) module; that is, the modules involved in the pre-training can be the left-side localization branch contained in the backbone network and the DeDA module. For example, this disclosure can use the LD dataset LaPa for pre-training in the first stage, which has 106 labeled points and an 18k training set. Experimental results demonstrate that better modeling of LD can improve performance across all tasks.

[0121] Next, in the second stage of training, the pre-trained backbone and the left-side localization branch contained in the DeDA module can be loaded first. Then, training can be performed using all face perception tasks and data, meaning that four face perception tasks and data can be used for the second stage of training. It should be noted that, for a fairer comparison, this disclosure uses only the ground truth values ​​for each face perception task as supervision. For example, 300WLP provides LD and HP, but this disclosure can use only the HP label for supervision.

[0122] In this way, pre-training provides a high-quality initial weight for subsequent tasks. These weights already contain a wealth of useful information, offering strong support for the tasks ahead. This allows the model to converge to the optimal solution more quickly during training for later tasks, thus improving model performance. Furthermore, pre-training reduces the model's dependence on labeled data, lowering training costs.

[0123] Figure 7 is a block diagram illustrating an electronic device 700 according to an exemplary embodiment of the present disclosure.

[0124] Referring to FIG7, the electronic device 700 includes at least one memory 701 and at least one processor 702. The at least one memory 701 stores instructions that, when executed by the at least one processor 702, execute a multi-task face perception method according to an exemplary embodiment of the present disclosure.

[0125] As an example, electronic device 700 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 700 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 700 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0126] In electronic device 700, processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0127] The processor 702 can execute instructions or code stored in the memory 701, which can also store data. Instructions and data can also be sent and received via a network through a network interface device, which can employ any known transmission protocol.

[0128] The memory 701 can be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 701 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 701 and the processor 702 can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 702 to read files stored in the memory.

[0129] In addition, the electronic device 700 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 700 can be interconnected via a bus and / or network.

[0130] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the aforementioned multi-task face recognition method. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0131] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements the multi-tasking face perception method according to the present disclosure.

[0132] According to the multi-task face perception method, apparatus, electronic device, storage medium and computer program product disclosed herein, by explicitly utilizing the physical representation of facial key points, effective supervision of semantic reference points is achieved, the guiding role of facial key points on face perception tasks is strengthened, and thus the prediction effect of face perception models can be improved.

[0133] According to an exemplary embodiment of this disclosure, the CLD module can calculate reference point coordinates for non-local tasks, i.e. semantic tasks, without introducing additional parameters, and the implementation process is simple, fast and efficient.

[0134] According to exemplary embodiments of this disclosure, the DeDA Block explicitly separates the localization task from the semantic task. Specifically, the DeDA Block explicitly splits the localization representation and semantic representation of the query vector, reducing interference between features from different tasks and thus improving the prediction accuracy of the multi-task face perception model. Experiments on publicly available datasets demonstrate that, with a significant reduction in the number of parameters and computational cost, the multi-task face perception model of this disclosure achieves prediction accuracy exceeding or approaching that of single-task tasks across all face perception tasks.

[0135] According to exemplary embodiments of this disclosure, pre-training provides a high-quality initial weight for subsequent tasks. These weights already contain a wealth of useful information, offering strong support for the tasks. This allows the model to converge to the optimal solution more quickly during training for subsequent tasks, thereby improving model performance. Furthermore, pre-training reduces the model's dependence on labeled data, lowering training costs.

[0136] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0137] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A multi-task face perception method, characterized in that, include: A face image to be detected is acquired; the face image is input into a multi-task face perception model, and the multi-task face perception model extracts face features and localization reference points from the face image, wherein the localization reference points are used to perform a localization task related to face key point detection; the multi-task face perception model calculates semantic reference points based on the localization reference points, wherein the semantic reference points are used to perform a semantic task related to face state detection; the multi-task face perception model performs the localization task and the semantic task based on the face features, the localization reference points, and the semantic reference points.

2. The multi-task face perception method as described in claim 1, characterized in that, The multi-task face perception model includes a cross-landmark attention (CLD) module, which is used to measure the correlation between the localization task and the semantic task; the step of calculating semantic reference points based on the localization reference points using the multi-task face perception model includes: calculating the semantic reference points based on the localization reference points using the CLD module.

3. The multi-task face perception method as described in claim 2, characterized in that, The step of calculating the semantic reference point based on the positioning reference point using the CLD module includes: calculating a relevance matrix using the CLD module based on the positioning query vector and the semantic query vector contained in the query vector, wherein the positioning query vector is used to characterize the task content of the positioning task, and the semantic query vector is used to characterize the task content of the semantic task; and calculating the semantic reference point using the CLD module based on the relevance matrix and the positioning reference point.

4. The multi-task face perception method as described in claim 3, characterized in that, The query vector is the query vector obtained by updating the initial query vector during the training process of the multi-task face perception model.

5. The multi-task face perception method as described in claim 4, characterized in that, The multi-task face perception model also includes a deformable dual attention DeDA module, a localization prediction module, and a semantic prediction module. The DeDA module includes a localization branch and a semantic branch, and each of the localization branch and the semantic branch includes a deformable attention module. The step of performing the localization task and the semantic task based on the facial features, the localization reference point and the semantic reference point through the multi-task facial perception model includes: inputting the facial features, the localization reference point and the query vector into the localization branch to obtain the updated query vector; The updated query vector is input into the localization prediction module to obtain the localization prediction result corresponding to the localization task; the facial features, the semantic reference point, and the semantic query vector are input into the semantic branch to obtain the semantic branch output; the semantic branch output is input into the semantic prediction module to obtain the semantic prediction result corresponding to the semantic task.

6. The multi-task face perception method as described in claim 5, characterized in that, The multi-task face perception model is trained using the following method: acquiring face sample images; extracting training face features and training localization reference points from the face sample images; inputting the training localization reference points and input query vector into the CLD module to obtain training semantic reference points; inputting the training face features, the training localization reference points, the training semantic reference points, and the input query vector into the DeDA module to obtain the training localization prediction result corresponding to the localization task, the training semantic prediction result corresponding to the semantic task, and the output query vector, wherein the output query vector serves as the input query vector for the next training iteration; calculating the localization loss based on the training localization prediction result and localization ground truth labels; calculating the semantic loss based on the training semantic prediction result and semantic ground truth labels; and adjusting the parameters of the multi-task face perception model according to the localization loss and the semantic loss, thereby training the multi-task face perception model.

7. The multi-task face perception method as described in claim 1, characterized in that, The multi-task face perception model contains multiple cascaded DeDA modules.

8. The multi-task face perception method as described in claim 1, characterized in that, The semantic task includes at least one of the following: human eye gaze detection task, head pose estimation task, and facial expression recognition task.

9. A multi-tasking face sensing device, characterized in that, include: The image acquisition module is configured to acquire images of the faces to be detected. The feature extraction module is configured to input the face image to be detected into a multi-task face perception model, and extract face features and localization reference points from the face image through the multi-task face perception model, wherein the localization reference points are used to perform a localization task related to face key point detection; the reference point calculation module is configured to calculate semantic reference points based on the localization reference points through the multi-task face perception model, wherein the semantic reference points are used to perform a semantic task related to face state detection; the task execution module is configured to execute the localization task and the semantic task based on the face features, the localization reference points, and the semantic reference points through the multi-task face perception model.

10. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the multi-tasking face perception method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the multi-task face perception method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-task face perception method as described in any one of claims 1 to 8.

Images