Facial pose estimation method, device, electronic device, and storage medium

Abstract

This disclosure provides a face pose estimation method, device, electronic device, and storage medium. The method includes the steps of obtaining a target image including face information, inputting the target image into a pre-constructed pose estimation model, performing feature extraction on the target image using a shallow dense-connection layer to obtain a plurality of first feature maps including shallow feature information, fusing deep feature information into shallow feature information by performing information fusion operations on the plurality of first feature maps using a deep feature reuse layer respectively to obtain a second feature map, extracting face pose information in the second feature map using an attention layer, obtaining a third feature map including face pose information, predicting the third feature map using a classifier, obtaining a prediction result of the face pose corresponding to the third feature map, and identifying the face pose in the target image based on the prediction result. This disclosure can fully obtain important information of the face pose and improve the accuracy of the face pose estimation result.

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer technology, and more particularly to a face pose estimation method, device, electronic device, and storage medium. [Background technology]

[0002] Facial pose estimation is an important research direction in the field of computer vision, and facial pose change is also an important factor affecting the effectiveness of face recognition. Effective pose estimation for facial images cannot further reduce the impact of facial pose on face recognition. Facial pose estimation is widely used in computer vision, such as live recognition, human-computer interaction, virtual reality, and smart surveillance.

[0003] Currently, conventional face pose estimation methods only capture pose angle information and ignore irrelevant facial features, so they can optimize the model using face and surrounding information, and conventional face pose recognition methods using general recognition models cannot fully capture important face pose information. Conventional face pose estimation methods use the detected face frame as input without considering the uncertainty of the face frame. Therefore, when directly regressing or classifying the pose angle, the face pose estimation method has difficulty achieving optimal prediction results, which reduces the accuracy of the face pose estimation result. Summary of the Invention

[0004] In view of this, the embodiments disclosed herein provide a face pose estimation method, device, electronic device, and storage medium to solve the problem existing in the prior art that important information of the face pose cannot be sufficiently acquired and the accuracy of the face pose estimation result is low.

[0005] In a first aspect of an embodiment of the present disclosure, a face pose estimation method is provided, including the steps of acquiring a target image including face information and inputting the target image into a pre-constructed pose estimation model; performing feature extraction on the target image using a shallow densely connected layer in the pose estimation model to acquire a plurality of first feature maps including shallow feature information; using the plurality of first feature maps as inputs to a deep feature reuse layer, and performing an information fusion operation on each of the plurality of first feature maps using the deep feature reuse layer to acquire a second feature map, thereby fusing the shallow feature information with the deep feature information; extracting face pose information in the second feature map using an attention layer, acquiring a third feature map including the face pose information, predicting the third feature map using a classifier, acquiring a prediction result of the face pose corresponding to the third feature map, and identifying the face pose in the target image based on the prediction result.

[0006] A second aspect of the disclosed embodiment provides a face pose estimation device including: an acquisition module configured to acquire a target image including facial information and input the target image into a pre-constructed pose estimation model; an extraction module configured to perform feature extraction on the target image using a shallow densely connected layer in the pose estimation model to acquire a plurality of first feature maps including the shallow feature information; a fusion module configured to use the plurality of first feature maps as inputs to a deep feature reuse layer, and to perform an information fusion operation on each of the plurality of first feature maps using the deep feature reuse layer to acquire a second feature map, thereby fusing the shallow feature information with the deep feature information; and a prediction module configured to extract facial pose information in the second feature map using an attention layer, acquire a third feature map including the facial pose information, predict the third feature map using a classifier, acquire a prediction result of a face pose corresponding to the third feature map, and identify a face pose in the target image based on the prediction result.

[0007] In a third aspect of an embodiment of the present disclosure, there is provided an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the electronic device implementing the steps of the above method when the processor executes the computer program.

[0008] In a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the above method when executed by a processor.

[0009] The at least one technical solution used in the embodiments of the present disclosure has the following beneficial effects:

[0010] A target image containing facial information is acquired, and the target image is input into a pre-constructed pose estimation model. In the pose estimation model, a shallow densely connected layer is used to extract features from the target image, and a plurality of first feature maps containing shallow feature information are obtained. The plurality of first feature maps are used as inputs to a deep feature reuse layer, and the deep feature reuse layer is used to perform an information fusion operation on each of the plurality of first feature maps to obtain a second feature map, thereby fusing the shallow feature information with the deep feature information. Face pose information in the second feature map is extracted using an attention layer, a third feature map containing the facial pose information is obtained, and a classifier is used to predict the third feature map, and a prediction result of the face pose corresponding to the third feature map is obtained. The present disclosure can fully acquire important information about the facial pose, resulting in more accurate face pose estimation results. [Brief explanation of the drawings]

[0011] In order to more clearly explain the technical solutions in the embodiments of the present disclosure, the following briefly introduces drawings necessary for explaining the embodiments or prior art. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without the need for creative work. [Figure 1] FIG. 1 is a schematic diagram of the network structure of a posture estimation model provided in an embodiment of the present disclosure. [Figure 2] 1 is a flowchart of a face pose estimation method provided in an embodiment of the present disclosure. [Figure 3]FIG. 1 is a structural schematic diagram of a face pose estimation device provided in an embodiment of the present disclosure. [Figure 4] 1 is a structural schematic diagram of an electronic device provided in an embodiment of the present disclosure; DETAILED DESCRIPTION OF THE INVENTION

[0012] In the following description, for purposes of explanation, not limitation, specific details, such as particular system structures and techniques, are provided to thoroughly understand the embodiments of the present disclosure. However, those skilled in the art should understand that the present disclosure can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present disclosure with unnecessary details.

[0013] In the task of face pose estimation, factors such as the diversity of face positions and sizes in images, the difference in the angles of different faces, and complex background environments have adverse effects on the algorithm's ability to accurately estimate face pose angles.

[0014] Conventional face pose estimation algorithms use the detected face frame as input without considering the uncertainty of the face frame, and are unable to fully utilize face and surrounding information to optimize the model. Pose estimation requires capturing pose angle information and ignoring irrelevant facial features. The backbone network and neck module of conventional general-purpose recognition models lack effective design and are therefore unable to fully capture important pose information. Furthermore, conventional methods that directly regress or classify angles are too straightforward and difficult to achieve optimal results. Therefore, when face pose estimation methods directly regress or classify pose angles, it is difficult to achieve optimal prediction results, reducing the accuracy of face pose estimation results.

[0015] In view of the problems in the prior art, this disclosure proposes a new neural network-based face pose estimation method. The neural network model in this disclosure mainly includes a backbone network, a neck module, and a head module. The training set used to train the pose estimation model includes a face frame and multiple pose labels, including pose label information consisting of multiple face frame label points and multiple pose angles. The original image is retained and data augmentation is performed based on the original image to achieve data augmentation training for the pose estimation model. The pose estimation model uses a backbone network consisting of a shallow densely connected layer and a deep feature reuse layer to fuse shallow and deep features in the face image, and estimates the face pose angle in the photo using an attention module and a grouping angle regression module.

[0016] Hereinafter, the network structure of the posture estimation model in this disclosure will be described in detail with reference to the drawings. FIG. 1 is a schematic diagram of the network structure of the posture estimation model provided in the embodiments of this disclosure. As shown in FIG. 1, the network structure of the posture estimation model may specifically include the following:

[0017] The backbone network includes shallow densely connected layers (corresponding to C1 to C5) and deep feature reuse layers (corresponding to P3 to P5). The shallow densely connected layers are mainly used to extract forward features from facial images. Densely connected means that the output of each module is used as input for the next module as well as for other modules that follow. The deep feature reuse layer fuses the extracted shallow feature information with the deep feature information during feature representation to obtain a feature map that contains more semantic information (i.e., pose information).

[0018] The Neck module consists of an SE attention module and a Transformer feature transformation module. The SE attention module weights the features of different channels to enhance important features while suppressing less important features. The Transformer has a strong feature extraction ability due to its clever structural design, and uses a multi-head attention mechanism to extract effective features, highlight important features in the feature map, and suppress interfering features.

[0019] The Head module corresponds to a grouping angle regression module (i.e., classifier), which sets up multiple classifiers for each face pose, and each classifier predicts the corresponding probability values ​​of multiple angles, calculates the prediction results of the classifiers according to the corresponding probability values ​​of each angle, and finally adds up the prediction results of all the classifiers to obtain the final estimation result, thereby identifying the corresponding angle values ​​of all poses in the face image.

[0020] 2 is a flowchart of a face pose estimation method provided in an embodiment of the present disclosure. The face pose estimation method of FIG. 2 may be executed by a server. As shown in FIG. 2, the face pose estimation method specifically includes: S201: acquiring a target image including face information and inputting the target image into a pre-constructed pose estimation model; In a posture estimation model, a shallow densely connected layer is used to perform feature extraction on the target image, thereby obtaining a plurality of first feature maps including shallow feature information (S202); S203: fusing deep feature information with shallow feature information by using the plurality of first feature maps as inputs of a deep feature reuse layer, and performing an information fusion operation on the plurality of first feature maps using the deep feature reuse layer to obtain a second feature map; The method may include S204 of extracting facial pose information in the second feature map using an attention layer, obtaining a third feature map including the facial pose information, predicting the third feature map using a classifier, obtaining a prediction result of the facial pose corresponding to the third feature map, and identifying the facial pose in the target image based on the prediction result.

[0021] Specifically, the target image in the present disclosure refers to the face image after face detection has been performed on the collected original image, i.e., the input of the pose estimation model is not the original image but the face image after face detection. Before predicting the pose angle using the pose estimation model, it is necessary to first construct and train the pose estimation model, and then use the trained pose estimation model as the model for actual use.

[0022] Furthermore, in the face pose estimation method of the disclosed embodiment, the detected face image (i.e., the target image) is input to the pose estimation model, and prediction is performed using the pose estimation model to output three pose angles, namely, pitch angle, polar angle, and roll angle. Due to differences in different faces and face detection models, there is no standard for perfect integration of face boundaries, so the pose estimation model obtained by training the detected face image has better robustness.

[0023] According to the technical solution provided in the embodiments of the present disclosure, the present disclosure obtains a target image containing facial information, inputs the target image into a pre-constructed pose estimation model, and in the pose estimation model, performs feature extraction on the target image using a shallow densely connected layer to obtain a plurality of first feature maps containing shallow feature information. The plurality of first feature maps are input to a deep feature reuse layer, and the deep feature reuse layer performs information fusion operations on the plurality of first feature maps respectively to obtain a second feature map, thereby fusing the shallow feature information with the deep feature information. An attention layer is used to extract facial pose information from the second feature map, obtains a third feature map containing the facial pose information, predicts the third feature map using a classifier, obtains a predicted result of the facial pose corresponding to the third feature map, and identifies the facial pose in the target image based on the prediction result. This disclosure can fully obtain important information about the facial pose and make the facial pose estimation result more accurate.

[0024] In some embodiments, the pose estimation model is constructed in the following manner, namely, by obtaining an original image containing face information, detecting the original image using a face detection model, obtaining a face image and face frame corresponding to the original image, and obtaining face pose information in the original image, and generating a first dataset using the face image, position coordinates of the face frame, and face pose information; cropping the original image using a default cropping method based on the original image and the position coordinates of the face frame, obtaining a cropped face image, and generating a second dataset using the cropped face image, position coordinates of the face frame, and face pose information; and combining the first dataset and the second dataset to obtain a training set, training the pose estimation model using the training set, and obtaining the trained pose estimation model.

[0025] Specifically, the training of the pose estimation model is mainly divided into two parts: dataset labeling and training data augmentation. The contents of these two parts will be described in detail below with reference to specific examples, and may include the following:

[0026] In the dataset labeling process, for each batch of dataset to be labeled, a conventional algorithm is used to obtain the three corresponding pose angles of the face in each original image X, which are denoted as p (pitch angle), y (polar angle), and r (roll angle). Then, the RetinaFace model is used to detect faces, obtain a face frame, and identify the coordinates (x1, y1) of the upper left corner of the face frame, as well as the width (w) and height (h) of the face frame. The original image X is retained, and the image tag is recorded as Y = [p, y, r, x1, y1, w, h].

[0027] Furthermore, after face detection, the acquired face frame can use the upper left corner coordinate, width, and height as the four corresponding label points of the detection frame (i.e., face frame), or the four corresponding vertex coordinates of the face frame rectangle can be used as the four label points. In this way, the label information in the training set includes seven pose labels, i.e., four detection frame label points and three pose angles. The face image after detection and the label information consisting of the seven pose labels are defined as the first data set.

[0028] In the data augmentation operation during training, a method of data augmentation using random cropping is used to crop the facial image in the original image within a random range based on the original image according to a certain cropping ratio (for example, 0.5 times). For example, by randomly cropping the facial image to a size slightly larger (1.0 to 1.2 times) than the face detection frame, a facial image similar to the facial image in the first dataset is generated, and the newly generated facial image and label information consisting of the seven pose labels corresponding to the facial image are used as the second dataset.

[0029] In the disclosed embodiment, the data augmentation method is used to increase the size of the training dataset, and when the pose estimation model is trained using the data-augmented training set, not only is the impact of inaccurate label boxes reduced, but the increased data volume also improves the robustness of the model.

[0030] In some embodiments, the training set includes face images and label information, and the label information is used as a tag during model training. The label information includes a plurality of label points corresponding to face frames and a plurality of pose angles, where the label points of the face frame include the coordinates of the corner points corresponding to the face frame, the width and height of the face frame, and the pose angles include the pitch angle, polar angle, and roll angle.

[0031] In some embodiments, the step of performing feature extraction on the target image using a shallow densely connected layer to obtain a plurality of first feature maps including shallow feature information includes a step in which the shallow densely connected layer includes a plurality of convolution modules connected in sequence, and each convolution module sequentially performs a convolution operation on the feature maps input to the convolution module, using the output of each convolution module as the input of the next convolution module, whose input further includes the output of the previous convolution module, and using the output of the last of the plurality of convolution modules in the shallow densely connected layer as the first feature map.

[0032] Specifically, since facial pose classification requires consideration of the positions of facial features and the overall angle of the avatar, some shallow information and deep abstract information in the facial image both play important roles. Therefore, the disclosed embodiment proposes a backbone network consisting of a shallow densely connected layer and a deep feature reuse layer. The backbone network extracts shallow feature information and deep feature information from the facial image, and then merges the shallow feature information with the deep feature information to obtain a feature map with more semantic information.

[0033] Furthermore, when extracting forward features in the shallow densely connected layer, the shallow densely connected layer in the pose estimation model is divided into five convolutional modules C1, C2, C3, C4, and C5. "Densely connected" means that the output of each convolutional module is used as input for the next convolutional module, and also for other subsequent convolutional modules. This allows the deep feature reuse layer to receive more shallow information, while improving the ability to represent pose information through features and improving feature utilization efficiency, allowing the extracted features to be used by more modules.

[0034] In some embodiments, the step of fusing deep feature information with shallow feature information by using a deep feature reuse layer to perform an information fusion operation on each of a plurality of first feature maps to obtain a second feature map includes the step of: the deep feature reuse layer includes convolution modules corresponding to the number of first feature maps; fusing the deep feature information with the second feature map including shallow feature information by using the convolution modules of the deep feature reuse layer to perform a convolution transformation on the first feature maps to obtain a second feature map; and performing global average pooling on the second feature map to obtain a corresponding second feature map after global average pooling.

[0035] Specifically, when using a deep feature reuse layer to perform feature representation, shallow features are fused with deep features, and three different levels of feature representation, namely P3, P4, and P5, are generated using the outputs of convolutional modules C3, C4, and C5, respectively, as shown in Figure 1, so that deep features are fused with the feature maps output by C3, C4, and C5. Hereinafter, the implementation method of the deep feature reuse layer will be described in detail with reference to Figure 1 in the above embodiment, and specifically includes the following:

[0036] C3, C4, and C5 correspond to convolutional modules in the shallow densely connected layer. The outputs of C3, C4, and C5 are used as inputs to P3, P4, and P5. The convolutional modules in the deep feature reuse layer perform convolution operations on the feature maps output by C3, C4, and C5, outputting feature maps containing semantic information (i.e., pose information). The following rule is used when performing convolution operations on P3, P4, and P5: when calculating P3, 1x1 convolution is performed on C3, C4, and C5, respectively, to convert them into a (28,28,512) feature map, thereby fusing the information of C3, C4, and C5 into P3; when calculating P4, 1x1 convolution is performed on C4 and C5, respectively, to convert them into a (14,14,512) feature map, thereby fusing the information of C4 and C5; when calculating P5, 1x1 convolution is performed on C5, to convert it into a (7,7,512) feature map, thereby using only the information of C5. Finally, global average pooling is performed on P3, P4, and P5, respectively, to obtain feature vectors of lengths 128, 256, and 512, respectively, which are input to the attention layer.

[0037] In some embodiments, the attention layer includes an SE attention module and a feature transformation module, and the step of using the attention layer to extract facial pose information in the second feature map and obtain a third feature map including the facial pose information includes the steps of: using the SE attention module to perform weight calculations on feature channels in the second feature map and weighting the feature channels based on the channel weights to obtain a weighted second feature map; and using the feature transformation module to perform feature extraction on the weighted second feature map to obtain a third feature map including valid feature information, where the valid feature information includes the facial pose information.

[0038] Specifically, to effectively extract pose information from the feature information extracted from the backbone network, this publication provides an SE attention module and a Transformer-based neck module. The SE attention module weights the features of different channels to enhance important features while suppressing less important features. The Transformer has strong feature extraction capabilities, highlighting important features and suppressing interfering features.

[0039] Furthermore, the SE attention module includes two fully connected layers and one sigmoid layer (i.e., normalization layer). Assuming that the number of channels of the second feature map input to the SE attention module is c, the second feature map with c channels is processed by two fully connected layers and one sigmoid layer to obtain channel weights for each feature channel, and then each channel weight is multiplied by the corresponding feature channel to obtain a weighted feature map.

[0040] Furthermore, the Transformer (feature transformation module) processes the input weighted feature maps using multi-head attention and a multi-layer fully connected MLP to obtain a set of transformed feature maps, c. The Transformer's multi-head attention mechanism can extract effective features from the feature maps and further highlight important features.

[0041] In some embodiments, the step of predicting a third feature map using a classifier, obtaining a predicted result of a face pose corresponding to the third feature map, and identifying a face pose in the target image based on the prediction result includes: each face pose corresponds to a plurality of third feature maps, each third feature map corresponds to a plurality of classifiers, each classifier predicts a number of angle values ​​based on the third feature map, calculates a predicted pose angle for each classifier based on the number of angle values, sums up the predicted pose angles of all the classifiers, obtains a pose angle corresponding to each face pose, and uses the pose angles corresponding to the three face poses to be an estimated result for the face pose in the target image.

[0042] Specifically, to obtain accurate posture angles, this disclosure adds a grouped angle regression module (also referred to as a classifier module) to the posture estimation model, i.e., the Head module in FIG. 1 . For each posture angle, each set of features may include three classifiers, each of which predicts 10 angle values. For example, the first classifier predicts probability values ​​from 0° to 9°, and the model predicts the probability values ​​of these 10 angles (p00, p01, ..., p09). The predicted result of this classifier is angle0 = p01*0 + p02*1 + ... + p09*9. Finally, the results of the nine classifiers are added together to obtain the final posture angle. Performing the above calculation for each of the three posture angles yields the three final predicted posture angles.

[0043] Furthermore, each third feature map input to the Head module corresponds to three classifiers, each of which predicts 10 different angle values. Based on the prediction results of each classifier, the final prediction results of all classifiers (a total of nine classifiers) are calculated, and the final prediction result of the Head module is used as the corresponding angle value of the current pose angle. For each pose angle, the model outputs one angle value, and the angle values ​​corresponding to all pose angles are used as the estimation results for the face pose in the target image.

[0044] According to the technical proposals provided in the embodiments of this disclosure, this disclosure uses a dataset format with 7 pose labels (4 detection frame label points and 3 pose angles) and designs facial image data augmentation based on this labeling scheme to expand the data volume of the training set. This disclosure analyzes based on facial image pose features and designs a backbone network with shallow dense connection and deep feature reuse to improve feature utilization efficiency and expressive power. This disclosure also discloses SE attention and a neck module using Transformer to further extract important pose information. This disclosure adds a grouping angle regression module to the model and uses 27 classifiers to more accurately predict the three pose angles.

[0045] The following are apparatus embodiments of the present disclosure for carrying out the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0046] 3 is a structural schematic diagram of a face pose estimation device provided in an embodiment of the present disclosure. As shown in FIG. 3, the face pose estimation device includes: an acquisition module 301 configured to acquire a target image including face information and input the target image into a pre-constructed pose estimation model; an extraction module 302 configured to perform feature extraction on the target image using shallow densely connected layers in the pose estimation model to obtain a plurality of first feature maps including shallow feature information; a fusion module 303 configured to fuse deep feature information with shallow feature information by taking the plurality of first feature maps as inputs of a deep feature reuse layer, and using the deep feature reuse layer to perform an information fusion operation on the plurality of first feature maps respectively to obtain a second feature map; and a prediction module 304 configured to extract facial pose information in the second feature map using an attention layer, obtain a third feature map including the facial pose information, predict the third feature map using a classifier, obtain a prediction result of the facial pose corresponding to the third feature map, and identify the facial pose in the target image based on the prediction result.

[0047] In some embodiments, the acquisition module 301 in FIG. 3 constructs a pose estimation model in the following manner, namely, acquiring an original image containing face information, detecting the original image using a face detection model, acquiring a face image and face frame corresponding to the original image, and acquiring face pose information in the original image, and generating a first dataset using the face image, position coordinates of the face frame, and face pose information; cropping the original image using a default cropping manner based on the original image and the position coordinates of the face frame, acquiring a cropped face image, and generating a second dataset using the cropped face image, position coordinates of the face frame, and face pose information; combining the first dataset and the second dataset to obtain a training set, and using the training set to train a pose estimation model and obtain the trained pose estimation model.

[0048] In some embodiments, the training set includes face images and label information, and the label information is used as a tag during model training. The label information includes a plurality of label points corresponding to face frames and a plurality of pose angles, where the label points of the face frame include the coordinates of the corner points corresponding to the face frame, the width and height of the face frame, and the pose angles include the pitch angle, polar angle, and roll angle.

[0049] In some embodiments, the shallow densely connected layer includes multiple convolution modules connected in sequence, and the extraction module 302 in FIG. 3 uses each convolution module to perform a convolution operation on the feature maps input to the convolution modules in sequence, and the output of each convolution module is used as the input of the next convolution module, and the input of each convolution module further includes the output of the previous convolution module, and the output of the last multiple convolution modules in the shallow densely connected layer is used as the first feature map.

[0050] In some embodiments, the deep feature reuse layer includes convolution modules corresponding to the number of first feature maps, and the fusion module 303 in FIG. 3 uses the convolution modules of the deep feature reuse layer to perform convolution transformation on the first feature maps to obtain second feature maps, thereby fusing deep feature information into second feature maps containing shallow feature information, and performing global average pooling on the second feature maps to obtain corresponding second feature maps after global average pooling.

[0051] In some embodiments, the attention layer includes an SE attention module and a feature transformation module, and the prediction module 304 in FIG. 3 includes the steps of: using the SE attention module to perform weight calculations on feature channels in the second feature map, and weighting the feature channels based on the channel weights to obtain a weighted second feature map; and using the feature transformation module to perform feature extraction on the weighted second feature map to obtain a third feature map containing valid feature information, where the valid feature information includes face pose information.

[0052] In some embodiments, each face pose corresponds to a plurality of third feature maps, each third feature map corresponds to a plurality of classifiers, each classifier predicts a number of angle values ​​based on the third feature map, calculates a predicted pose angle for each classifier based on the number of angle values, sums up the predicted pose angles of all the classifiers to obtain a pose angle corresponding to each face pose, and uses the pose angles corresponding to the three face poses as an estimation result for the face pose in the target image.

[0053] It should be understood that the magnitude of the numbers of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and does not arbitrarily limit the implementation process of the embodiments disclosed herein.

[0054] 4 is a structural schematic diagram of an electronic device 4 provided in an embodiment of the present disclosure. As shown in FIG. 4, the electronic device 4 of the embodiment includes a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable by the processor 401. When the processor 401 executes the computer program 403, it realizes the steps in each of the above method embodiments. Alternatively, when the processor 401 executes the computer program 403, it realizes the functions of each module / unit in each of the above device embodiments.

[0055] For example, the computer program 403 may be divided into one or more modules / units, and the one or more modules / units may be stored in the memory 402 and executed by the processor 401 to accomplish the present disclosure. The one or more modules / units may be a series of computer program command sections capable of performing a specific function, and the command sections are intended to explain the process of the computer program 403 being executed in the electronic device 4.

[0056] The electronic device 4 may be an electronic device such as a desktop computer, a laptop computer, a palmtop computer, or a cloud server. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. As will be understood by those skilled in the art, FIG. 4 is merely an example of the electronic device 4 and is not intended to limit the electronic device 4, which may include more or fewer components than those shown, or may combine certain components or different components; for example, the electronic device may include input / output devices, network access devices, buses, etc.

[0057] Processor 401 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor, any common processor, etc.

[0058] The memory 402 may be an internal storage unit of the electronic device 4, such as a hard disk or RAM of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash card, etc., that is provided in the electronic device 4. Furthermore, the memory 402 may include not only the internal storage unit of the electronic device 4 but also an external storage device. The memory 402 is intended to store computer programs and other programs and data required by the electronic device. The memory 402 may also be used to temporarily store data that has been output or that is to be output.

[0059] Those skilled in the art will understand that, for convenience and brevity, only the division of the above functional units and modules has been used as an example. However, in actual applications, the above functions can be assigned to different functional units or modules as needed, i.e., all or part of the above-described functions can be achieved by dividing the internal structure of the device into different functional units or modules. The functional units and modules in the embodiments may be integrated into a single processing unit, each unit may exist physically independently, or two or more units may be integrated into a single unit. The integrated unit may be implemented in the form of hardware or software functional units. The specific names of the functional units and modules are provided solely for the purpose of distinguishing them from one another and do not limit the scope of protection of the present application. For the specific operating processes of the units and modules in the above system, reference may be made to the corresponding processes in the above-described method embodiments, and further description will be omitted here.

[0060] In the above embodiments, the description of each embodiment has its own emphasis, and for the details or parts not described in an embodiment, reference can be made to the relevant descriptions of other embodiments.

[0061] Those skilled in the art can recognize that the units and algorithm steps of each example described in the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software is determined by the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to realize the described functions for each specific application, but such realization should not be considered beyond the scope of this disclosure.

[0062] It should be understood that the disclosed devices / computer devices and methods in the embodiments provided in this disclosure can be implemented in other ways. For example, the device / computer device embodiments described above are merely illustrative, and the division into modules or units is merely a logical division of functions. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into other systems, or some features may be omitted or not implemented. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through several interfaces, devices, or units, and may be electrical, mechanical, or other types.

[0063] Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, i.e., located in one location or distributed across multiple network units, some or all of which may be selected according to actual needs to achieve the objectives of the solutions of this embodiment.

[0064] Note that the functional units in this disclosure may be integrated into one processing unit, each unit may exist physically independently, or two or more units may be integrated into one unit. The integrated unit may be realized in the form of hardware or in the form of a software functional unit.

[0065] The integrated module / unit may be realized in the form of a software functional unit and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the present disclosure provides that the realization of all or part of the processes in the above-described method embodiments can be accomplished by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-described method embodiments can be realized. The computer program may include computer program code, which may be in source code format, object code format, an executable file, or some intermediate format. The computer-readable storage medium may include any entity or device capable of carrying computer program code, such as a recording medium, a U-disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier wave signal, an electrical communication signal, and a software distribution medium. Furthermore, the content contained on a computer-readable storage medium may be increased or decreased as required by the legislation and patent practice of a jurisdiction. For example, in some jurisdictions, the legislation and patent practice may require that a computer-readable storage medium not include electrical carrier signals and telecommunications signals.

[0066] The above-described examples are merely for the purpose of illustrating the technical solutions of the present disclosure, and are not intended to limit the same. Although the present disclosure has been described in detail with reference to the above-described examples, those skilled in the art may still amend the technical solutions described in the above-described examples or equivalently replace some of the technical features therein, and it should be understood that such amendments or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and all of them should be included in the protection scope of the present disclosure.

Claims

1. A method for face pose estimation executed by a processor of an electronic device, comprising: acquiring a target image including face information and inputting the target image into a pre-constructed pose estimation model; performing feature extraction on the target image using a shallow densely connected layer in the pose estimation model to obtain a plurality of first feature maps including shallow feature information; Fusing deep feature information with the shallow feature information by using the plurality of first feature maps as inputs of a deep feature reuse layer, and performing an information fusion operation on the plurality of first feature maps using the deep feature reuse layer to obtain a second feature map; extracting face pose information in the second feature map using an attention layer to obtain a third feature map including the face pose information; predicting the third feature map using a classifier to obtain a prediction result of a face pose corresponding to the third feature map; and identifying a face pose in the target image based on the prediction result; The step of obtaining a plurality of first feature maps including the shallow feature information includes: the target image is input to a first convolution module of a plurality of convolution modules included in the shallow densely connected layer, which are connected in sequence and each perform a convolution operation; an output of the operation result of the first convolution module is input to a next convolution module and also to other convolution modules subsequent to the next convolution module; outputs of the next convolution module and subsequent convolution modules are also input to the next convolution module and also to other convolution modules subsequent to the next convolution module; convolution operations are performed by each of the convolution modules; and output of the operation results of a final predetermined number of convolution modules among the plurality of convolution modules connected in sequence is set as the first feature map, The step of fusing the shallow feature information with the deep feature information comprises: using convolution modules included in the deep feature reuse layer corresponding to the number of the first feature maps, inputting a predetermined one of the plurality of first feature maps to one of the convolution modules corresponding to the number of the first feature maps, inputting two or more predetermined number of the plurality of first feature maps to another of the convolution modules, outputting results of the convolution operation performed by each of the convolution modules as second feature maps, and performing global average pooling on each of the second feature maps to obtain a plurality of second feature maps after the global average pooling; The step of identifying a face pose in the target image based on the prediction result includes: performing weight calculations on the feature channels in the second feature maps using an SE attention module included in the attention layer to obtain channel weights, and weighting the corresponding feature channels using the channel weights to obtain a plurality of weighted second feature maps; extracting features including facial pose information from the weighted second feature maps by a feature transformation module included in the attention layer, to obtain a plurality of third feature maps including facial pose information; The classifiers corresponding to the plurality of third feature maps respectively predict a probability for each angle value of the face pose based on the corresponding third feature map, and the sum of the products of each angle value and the probability is used as a predicted result of the pose angle. The predicted results of the pose angles of all the classifiers are summed up to be used as the angle of the face pose in the target image. A face pose estimation method comprising:

2. constructing the pose estimation model in the following manner: acquiring an original image including face information, detecting the original image using a face detection model, acquiring a face image and a face frame corresponding to the original image, and acquiring face pose information in the original image, and generating a first data set using the face image, position coordinates of the face frame, and the face pose information; trimming the original image using a default trimming method based on the original image and the position coordinates of the face frame to obtain a trimmed face image; and generating a second data set using the trimmed face image, the position coordinates of the face frame, and the face pose information; combining the first data set and the second data set to obtain a training set, and training a pose estimation model using the training set to obtain a trained pose estimation model.

2. The method of claim 1 .

3. the training set includes face images and label information, the label information is used as tags during model training, and the label information includes a plurality of label points corresponding to the face frame and a plurality of pose angles; The label points of the face frame include the coordinates of the corresponding corner points of the face frame, and the width and height of the face frame, and the posture angles include a pitch angle, a polar angle, and a roll angle.

3. The method of claim 2.

4. A face pose estimation device, an acquisition module configured to acquire a target image including facial information and input the target image into a pre-constructed pose estimation model; an extraction module configured to perform feature extraction on the target image using a shallow densely connected layer in the pose estimation model to obtain a plurality of first feature maps including shallow feature information; a fusion module configured to fuse deep feature information with the shallow feature information by using a plurality of the first feature maps as inputs of a deep feature reuse layer and performing an information fusion operation on the plurality of first feature maps respectively using the deep feature reuse layer to obtain a second feature map; a prediction module configured to extract facial pose information in the second feature map using an attention layer to obtain a third feature map including the facial pose information, predict the third feature map using a classifier to obtain a prediction result of a facial pose corresponding to the third feature map, and identify a facial pose in the target image based on the prediction result; The shallow densely connected layer includes a plurality of convolution modules connected in series, each of which performs a convolution operation; the extraction module inputs the output of the operation result of the first convolution module of the plurality of convolution modules in the shallow densely connected layer to the next convolution module, and also inputs it to other convolution modules subsequent to the next convolution module, and also inputs the outputs of the next convolution module and subsequent convolution modules to the next convolution module and also inputs it to other convolution modules subsequent to that, causing each of the convolution modules to perform a convolution operation, and sets the output of the operation result of the last plurality of convolution modules among the plurality of convolution modules connected in order as the first feature map, the deep feature reuse layer includes convolution modules corresponding to the number of the first feature maps; the fusion module inputs only one predetermined first feature map among the plurality of first feature maps to one convolution module among the convolution modules corresponding to the number of the first feature maps of the deep feature reuse layer, and inputs two or more predetermined number of first feature maps among the plurality of first feature maps to another convolution module, outputs a result of the convolution operation performed by each convolution module as a second feature map, and performs global average pooling on each of the second feature maps to obtain a plurality of second feature maps after the global average pooling; The attention layer includes an SE attention module and a feature transformation module; The prediction module: using the SE attention module to perform weight calculations on feature channels in the plurality of second feature maps to obtain channel weights; and weighting the corresponding feature channels using the channel weights to obtain a plurality of weighted second feature maps; the feature transformation module extracts features including face pose information from the weighted second feature maps to obtain a plurality of third feature maps including face pose information; The classifiers corresponding to the plurality of third feature maps respectively predict the probability for each angle value of the face pose based on the corresponding third feature maps, and the sum of the products of each angle value and the probability is used as the predicted result of the pose angle. The predicted results of the pose angles of all the classifiers are summed up to obtain the angle of the face pose in the target image. A face pose estimation device characterized by:

5. 10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the computer program implementing the method of claim 1 when executed by the processor. An electronic device characterized by:

6. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method of claim 1 when executed by a processor. A computer-readable storage medium comprising:

Images