Face pose estimation method and device, face input method and device, equipment and medium

By screening face key points and significance detection results, combining the three-dimensional face model and camera parameters, iteratively optimizes the rotation matrix, the problems of low accuracy of face pose estimation and high consumption of computing resources in the existing technology are solved, and the problem of low face pose estimation is realized. Efficient and accurate facial posture estimates.

CN119048590APending Publication Date: 2024-11-29BEIJING XIAOMI MOBILE SOFTWARE CO LTD

Patent Information

Application Number
CN202310620382.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-29
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The prior art has problems in the estimation of face poses with low accuracy, high computing resources, time-consuming and labor-intensive training data, and insufficient prediction results.

Method used

By obtaining the face key points and significance detection results in the face image, the target face key points that meet the significance screening conditions are selected, the face posture is estimated using the three-dimensional face model and in-camera parameters, and the projection error is optimized through iteratively To improve the accuracy of the rotation matrix.

Benefits of technology

It improves the accuracy and efficiency of facial pose estimation, reduces the consumption of computing resources, and enhances the estimation accuracy of facial poses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048590A_ABST
    Figure CN119048590A_ABST
Patent Text Reader

Abstract

The invention provides a face pose estimation method and device, a face input method and device, equipment and a medium. The method comprises the following steps: acquiring face key points in a face image and a saliency detection result corresponding to the face image; based on the saliency detection result, selecting target face key points meeting a first saliency screening condition from the face key points; determining a corresponding three-dimensional coordinate point of the target face key point in a preset three-dimensional face model; determining a target rotation matrix of a camera based on the three-dimensional coordinate points corresponding to the target face key points and internal parameters of the camera shooting the face image; and estimating a face pose in the face image based on the target rotation matrix of the camera. Through the method, the accuracy of face pose estimation can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a method, device, equipment, and medium for estimating and recording facial postures. Background Art

[0002] Head posture (i.e., facial posture) can provide important information in some scenarios and has broad application prospects. For example, in the face unlocking system of the mobile phone lock screen, when entering a face, it can be determined whether the head deflection angle meets the entry requirements to ensure that the template image with normal posture is entered; when unlocking a face, it can eliminate the problem of mis-locking caused by excessive head deflection angle, so as to unlock safely and accurately; in the security verification of payment systems and face smart door locks, the head posture of nodding and shaking can be used to determine whether the user is alive; in intelligent driving systems, the deflection of head posture can be used to determine whether the driver is distracted or fatigued; in virtual reality systems, the correct view of the scene can be rendered through head posture information.

[0003] At the same time, head posture estimation can also be used as an upstream task in computer vision research fields such as face recognition, iris recognition, gaze estimation, and expression analysis, and plays an important role in face-related image tasks. How to accurately estimate face posture has always attracted much attention. Summary of the invention

[0004] The present invention provides a face posture estimation method, a face entry method and a device, equipment and medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for estimating a face posture is provided, comprising:

[0006] Acquire facial key points in a facial image and a saliency detection result corresponding to the facial image; based on the saliency detection result, select target facial key points from the facial key points that meet a first saliency screening condition;

[0007] Determine the three-dimensional coordinate points corresponding to the target facial key points in the preset three-dimensional face model;

[0008] Determine a target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image;

[0009] The facial pose in the facial image is estimated based on the target rotation matrix of the camera.

[0010] In some embodiments, selecting, based on the saliency detection result, target facial key points from among the facial key points that meet a first saliency screening condition includes:

[0011] Based on the saliency detection result, selecting points that meet the first saliency screening condition other than the predetermined key points among the facial key points to obtain first saliency key points;

[0012] The predetermined key points and the first salient key points are used as the target face key points.

[0013] In some embodiments, determining the target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image includes:

[0014] Estimate the rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target face key points and the intrinsic parameters of the camera;

[0015] Calculate the projection point of each three-dimensional coordinate point on the two-dimensional plane according to the intrinsic parameters of the camera and the estimated rotation matrix of the camera;

[0016] Determine the difference between the target face key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, and determine the projection error based on the difference;

[0017] A target rotation matrix of the camera is determined according to the projection error.

[0018] In some embodiments, determining the target rotation matrix of the camera according to the projection error includes:

[0019] If the projection error is less than a preset error threshold, using the estimated rotation matrix of the camera as the target rotation matrix of the camera;

[0020] If the projection error is greater than or equal to the preset error threshold, the target facial key points are updated, and the rotation matrix of the camera is estimated based on the updated target facial key points to determine the projection error, until the projection error is less than the preset error threshold to obtain the target rotation matrix of the camera.

[0021] In some embodiments, updating the target face key points includes:

[0022] Based on the difference between the target face key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, select the point in the target face key point that meets the preset difference condition as the key point to be replaced;

[0023] Based on the saliency analysis result, selecting points among the facial key points other than the target facial key points that meet the second saliency screening condition as second saliency key points;

[0024] The second salient key point replaces the key point to be replaced in the target face key point to obtain an updated target face key point.

[0025] In some embodiments, determining the difference between the target facial key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, and determining the projection error based on the difference includes:

[0026] Determine the Euclidean distance between each of the target face key points and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane;

[0027] Based on the Euclidean distances corresponding to the target facial key points, an average value of the Euclidean distances is determined, and the average value is determined as the projection error.

[0028] In some embodiments, obtaining facial key points in a facial image includes:

[0029] Extracting features from the face image using a feature extraction layer of a preset face key point detection model to obtain face features;

[0030] Processing the facial features using the heat map output network of the preset facial key point detection model to obtain a downsampled feature heat map;

[0031] Processing the feature heat map using the offset coordinate output network of the preset facial key point detection model to obtain the offset coordinates corresponding to each feature point of the feature heat map;

[0032] The facial key points in the facial image are determined based on the feature heat map and each of the offset coordinates.

[0033] According to a second aspect of an embodiment of the present disclosure, a face entry method is provided, comprising:

[0034] Obtain a face image of the target object;

[0035] Obtaining the face posture of the target object using a preset face posture estimation method; wherein the preset face posture estimation method is any one of the methods in the first aspect;

[0036] In response to the facial posture satisfying a preset posture condition, saving the facial image of the target object;

[0037] In response to the facial posture not satisfying the preset posture condition, outputting prompt information.

[0038] In some embodiments, in response to the facial posture not satisfying a preset posture condition, outputting prompt information includes:

[0039] In response to the face posture not satisfying the preset posture condition, outputting prompt information indicating that the face image is unqualified;

[0040] and / or,

[0041] In response to the facial posture not satisfying the preset posture condition, outputting facial posture adjustment information corresponding to satisfying the preset posture condition.

[0042] According to a third aspect of an embodiment of the present disclosure, there is provided a face pose estimation device, comprising:

[0043] A first acquisition module is configured to acquire facial key points in a facial image and a saliency detection result corresponding to the facial image; a selection module is configured to select target facial key points that meet a first saliency screening condition from among the facial key points based on the saliency detection result;

[0044] A first determination module is configured to determine the three-dimensional coordinate points corresponding to the target face key points in a preset three-dimensional face model;

[0045] A second determination module is configured to determine a target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image;

[0046] An estimation module is configured to estimate the facial posture in the facial image based on the target rotation matrix of the camera.

[0047] In some embodiments, the selection module is further configured to select points that meet the first significance screening condition other than predetermined key points among the facial key points based on the significance detection result to obtain first significant key points; and use the predetermined key points and the first significant key points as the target facial key points.

[0048] In some embodiments, the second determination module is configured to estimate the rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera; calculate the projection points of each three-dimensional coordinate point on the two-dimensional plane according to the intrinsic parameters of the camera and the estimated rotation matrix of the camera; determine the difference between the target facial key points and the projection points of the corresponding three-dimensional coordinate points on the two-dimensional plane, and determine the projection error based on the difference; determine the target rotation matrix of the camera according to the projection error.

[0049] In some embodiments, the second determination module is further configured to use the estimated rotation matrix of the camera as the target rotation matrix of the camera if the projection error is less than a preset error threshold; if the projection error is greater than or equal to the preset error threshold, update the target facial key points, and estimate the rotation matrix of the camera based on the updated target facial key points to determine the projection error, until the projection error is less than the preset error threshold to obtain the target rotation matrix of the camera.

[0050] In some embodiments, the second determination module is further configured to select points in the target facial key points that meet a preset difference condition as key points to be replaced based on the difference between the target facial key points and the projection points of the corresponding three-dimensional coordinate points on the two-dimensional plane; based on the significance analysis result, select points in the facial key points that meet a second significance screening condition other than the target facial key points as second significance key points; replace the key points to be replaced in the target facial key points with the second significance key points to obtain updated target facial key points.

[0051] In some embodiments, the second determination module is further configured to determine the Euclidean distance between each of the target facial key points and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane; based on the Euclidean distance corresponding to each of the target facial key points, determine the average value of the Euclidean distance, and determine the average value as the projection error.

[0052] In some embodiments, the first acquisition module is further configured to use the feature extraction layer of a preset facial key point detection model to perform feature extraction on the facial image to obtain facial features; use the heat map output network of the preset facial key point detection model to process the facial features to obtain a downsampled feature heat map; use the offset coordinate output network of the preset facial key point detection model to process the feature heat map to obtain the offset coordinates corresponding to each feature point of the feature heat map; determine the facial key points in the facial image based on the feature heat map and each offset coordinate.

[0053] According to a fourth aspect of an embodiment of the present disclosure, a face recording device is provided, comprising:

[0054] A second acquisition module is configured to acquire a face image of a target object;

[0055] An acquisition module, configured to obtain the face posture of the target object by using a preset face posture estimation method; wherein the preset face posture estimation method is any one of the methods in the first aspect;

[0056] A saving module, configured to save the face image of the target object in response to the face posture satisfying a preset posture condition;

[0057] The output module is configured to output prompt information in response to the face posture not satisfying the preset posture condition.

[0058] In some embodiments, the output module is further configured to output prompt information indicating that the facial image is unqualified in response to the facial posture not satisfying the preset posture condition; and / or, in response to the facial posture not satisfying the preset posture condition, output facial posture adjustment information corresponding to satisfying the preset posture condition.

[0059] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0060] processor;

[0061] a memory for storing processor-executable instructions;

[0062] The processor is configured to execute the method as described in the first aspect or the second aspect above.

[0063] According to a sixth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0064] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in the first aspect or the second aspect above.

[0065] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0066] In the embodiments of the present disclosure, not only the facial key points of the facial image are obtained, but also the saliency detection result of the facial image is obtained, and the target facial key points that meet the saliency screening conditions are screened out from the facial key points through the saliency detection result. By using the saliency detection result to screen the facial key points, the quality of the selected facial key points can be improved, thereby helping to improve the accuracy of facial posture estimation based on the selected two-dimensional facial key points and the corresponding three-dimensional coordinate points.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0069] Figure 1 It is a flow chart of a method for estimating facial posture shown in an embodiment of the present disclosure.

[0070] Figure 2 An example diagram of facial key points provided in an embodiment of the present disclosure.

[0071] Figure 3 It is an example diagram of a saliency heat map provided in an embodiment of the present disclosure.

[0072] Figure 4 A flow chart of a face entry method provided in an embodiment of the present disclosure.

[0073] Figure 5 An example diagram of an electronic device outputting prompt information provided by an embodiment of the present disclosure.

[0074] Figure 6 The figure is a diagram of a face pose estimation device according to an exemplary embodiment.

[0075] Figure 7 The figure is a diagram of a face recording device according to an exemplary embodiment.

[0076] Figure 8 A frame of an electronic device according to an exemplary embodiment is shown Figure 1 .

[0077] Fig. 9 A frame of an electronic device according to an exemplary embodiment is shown Figure 2 . DETAILED DESCRIPTION

[0078] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0079] The terms "first / second / third" etc. involved in the present disclosure are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first / second / third" etc. can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the present disclosure and are not intended to limit the present disclosure.

[0081] The commonly used head pose estimation algorithms currently include methods based on appearance images, methods based on template matching, methods based on detector arrays, and methods based on general face models. (1) The appearance image-based method uses deep learning to obtain a model that maps appearance face images to three-dimensional head poses. The training of the model requires a huge amount of data, and the deep learning model consumes a lot of computing power when performing inference and prediction, which is not suitable for transplantation to ordinary terminal products. (2) In the template matching-based method, the head pose of the sample image to be estimated is obtained by comparing the similarity metric with a series of labeled head pose images. The collection of a series of labeled head pose images is time-consuming and labor-intensive, and the prediction results are not accurate enough. (3) The detector array method trains multiple head pose detectors, each of which predicts different discrete poses. Its disadvantage is that the process of training each discrete head pose is cumbersome, requires a large amount of labeled training data, and can only roughly estimate the discrete head pose. (4) The method based on the general face model uses a general average face three-dimensional (3D) key point model to fit the face key points on the two-dimensional (2D) image. This method requires the use of the location information of facial key points, which allows the model to estimate at the level of facial features rather than the pixel level of the image. This method is fast and efficient, but the accuracy of its prediction results is heavily dependent on the accuracy of 2D facial key points.

[0082] In this regard, the present disclosure provides a method for estimating face posture. Figure 1 is a flow chart of a method for estimating face posture shown in an embodiment of the present disclosure, such as Figure 1 As shown, the face pose estimation method includes the following steps:

[0083] S11, obtaining facial key points of a facial image and a saliency detection result of the facial image;

[0084] S12, based on the saliency detection result, selecting target facial key points that meet a first saliency screening condition from among the facial key points;

[0085] S13, determining the three-dimensional coordinate points corresponding to the target face key points in the preset three-dimensional face model;

[0086] S14, determining a target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image;

[0087] S15. Estimate the facial posture in the facial image based on the target rotation matrix of the camera.

[0088] In the embodiments of the present disclosure, the face pose estimation method can be implemented by the terminal / server alone; it can also be implemented by the terminal and the server in collaboration. For example, the terminal alone undertakes the face pose estimation method described below, or the terminal sends a face pose estimation request carrying a face image to the server, and the server executes the face pose estimation method on the face image according to the received face pose estimation request, and after determining the face pose in the face image, the face pose information is fed back to the terminal.

[0089] The electronic device for face posture estimation provided by the embodiments of the present disclosure may be various types of terminal devices or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the terminal may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart car connection, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the present disclosure.

[0090] Taking the face posture estimation method in the embodiment of the present disclosure applied to the above-mentioned electronic device as an example, the electronic device can perform face posture estimation after collecting face images through the built-in camera, and the electronic device can also perform face posture estimation after receiving face images sent by other devices.

[0091] In step S11, the electronic device can obtain facial key points by performing key point detection on the facial image. The purpose of facial key point detection is to automatically identify the position of facial key points in the facial image. These key points are either vantage points that describe the unique position of facial parts (such as the corners of the eyes and the corners of the mouth), or interpolation points that connect these vantage points to facial parts and contours. Among them, the key point detection method can be a traditional active shape model (Active Shape Model, ASM) method, a production model (Active Appearance Model, AAM) method or a cascade regression learning model CPR method, or a multi-level regression method based on deep learning, such as a cascade convolutional neural network (Deep Convolutional Network, DCNN) or a PIPNet method, etc., which is not limited by the embodiments of the present disclosure.

[0092] Usually, key point detection on a face image can obtain 21 key points, 29 key points, or 68 key points, etc. Figure 2 An example diagram of key points of a face provided in an embodiment of the present disclosure is shown in FIG. Figure 2As shown, 68 key points including eyebrows, eyes, nose, mouth and facial contour are marked on each person's face.

[0093] In addition, electronic devices will also perform saliency detection on facial images to obtain saliency detection results. The goal of saliency detection is to quickly find the most eye-catching area in the image. The saliency detection results include the saliency values ​​corresponding to each pixel in the facial image. Saliency detection methods are generally divided into two types: one is a bottom-up data-driven saliency detection method, which determines the saliency value of each pixel by the difference between each pixel and other pixels; the other is a top-down task-driven saliency detection method, which calculates the saliency of the image area through some specific features of the image. The saliency value obtained by saliency detection is used to represent the degree of saliency. The saliency value can be represented by different grayscale values ​​or by a heat map.

[0094] In step S12, the electronic device screens the facial key points based on the saliency detection result, and selects the target facial key points that meet the first saliency screening condition. In the embodiment of the present disclosure, the target facial key points that meet the first saliency screening condition may be a first preset number of facial key points with a high saliency value ranking, or may be facial key points with a saliency value greater than a first preset saliency threshold value, which is not limited in the embodiment of the present disclosure. In addition, in some embodiments, the electronic device may select the target facial key points based on the saliency values ​​of all facial key points. In other embodiments, the electronic device may also select the target facial key points based on the saliency values ​​of some facial key points.

[0095] Figure 3 is an example of a saliency heat map provided by an embodiment of the present disclosure, such as Figure 3 As shown in Figure 1, for the face image marked by A1, the higher the brightness of the point in the saliency heat map marked by A2, the greater the corresponding saliency value. A3 is the effect of superimposing A1 and A2. It can be seen from A3 that not all facial key points are points with higher brightness in the saliency heat map.

[0096] In step S13, the electronic device determines the three-dimensional coordinate points corresponding to the target face key points in the preset three-dimensional face model. The preset three-dimensional face model can be a general three-dimensional face model or a three-dimensional face model corresponding to the user of the face image, which is not limited in the embodiment of the disclosure.

[0097] The preset three-dimensional face model includes three-dimensional coordinate points corresponding to each two-dimensional face key point, so the electronic device can determine the three-dimensional coordinate points corresponding to each target face key point based on the preset three-dimensional face model. For example, if the target face key point includes the key point of the left eye corner, the three-dimensional coordinate point corresponding to the left eye corner can be found in the preset three-dimensional face model.

[0098] In step S14, the electronic device determines the target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image, for example, by solving the affine transformation matrix from 3D to 2D points based on the PnP (Perspective-n-Point algorithm, thereby obtaining the rotation matrix of the camera.

[0099] In the disclosed embodiment, the intrinsic parameters of the camera include the focal length, distortion parameters, optical center coordinates, etc. of the camera, which can be obtained through calibration. The target rotation matrix of the camera belongs to the extrinsic parameters of the camera, which is used to characterize the rotation required to transform a point in the world coordinate system to the camera coordinate system. In addition, the extrinsic parameters may also include a translation matrix, which is used to characterize the translation required to transform a point in the world coordinate system to the camera coordinate system.

[0100] It should be noted that the reason why the electronic device can determine the target rotation matrix of the camera based on the above method is that the camera coordinate system and the world coordinate system satisfy the mapping relationship, and the camera corresponds to an imaging model, as shown in the following formula (1):

[0101]

[0102] Where s is the scale factor, A is the 3×3 camera internal parameter matrix, R and T are the 3×3 rotation matrix and 3×1 translation vector of the camera external parameters, respectively, p is the 2D key point of the face in the image pixel coordinate system, and P is the 3D key point in the world coordinate system.

[0103] Assume that in the above formula (1), p = [u, v, 1] T P = [X, Y, Z, 1] T , then the mapping from the modeled face 3D key points to the face image 2D key points can be expressed as the following formula (2):

[0104]

[0105] Among them, α and β represent the focal length of the camera, u C 、v C represents the optical center coordinates of the camera, which are all internal parameters of the camera. In formula (2), a4, a8 and a 12 belongs to the translation vector T, and the rest belongs to the rotation matrix R.

[0106] The following two constraints can be obtained by calculating formula (2):

[0107]

[0108] For the convenience of representation, the row vectors b1, b2 and b3 of [R|T] are defined as follows:

[0109]

[0110] Therefore, the above formula (3) can also be expressed as:

[0111]

[0112] Formula (5) is a linear constraint obtained from one facial key point, then N facial key points can obtain a linear square group as shown in the following formula (6):

[0113]

[0114] Among them, the 2D key point of the face in the Nth image is represented by p i =[u i ,v i ,1] T , the Nth face 3D key point is represented as

[0115] From formula (6), we can see that since [b1, b2, b3] T There are 12 dimensions in total, or 12 unknowns, so at least 6 pairs of facial key point matches are required to achieve a linear solution of the matrix [R|T].

[0116] Based on the above, in the embodiment of the present disclosure, the number of target facial key points can be 6. Based on the three-dimensional coordinate points corresponding to the six target facial key points in the preset three-dimensional face model, the values ​​of b1, b2 and b3 can be obtained by solving the linear equation of formula (6), and the R part thereof is used as the target rotation matrix of the embodiment of the present disclosure. It should be noted that in this embodiment, the number of target facial key points is 6.

[0117] However, considering that the more facial key points provided, the higher the accuracy of facial pose estimation may be, the disclosed embodiment may also perform face estimation using more than 6 target facial key points. In addition, in order to take into account the speed of facial pose estimation, face estimation using less than 6 target facial key points may also be considered. It is understandable that when the number of target facial key points exceeds 6 or is less than 6, then solving 12 unknowns based on formula (6) through more or less than 12 equations may result in b1, b2, and b3 being unsolvable. In this case, an optimal solution may be found based on a fitting algorithm, such as finding an approximate optimal solution using the least squares method.

[0118] In the embodiment of the present disclosure, the R part of the above approximate optimal solution can also be used as the target rotation matrix of the embodiment of the present disclosure, and a more accurate target rotation matrix can be further determined based on the above approximate optimal solution.

[0119] In step S15, after determining the target rotation matrix of the camera, the electronic device can estimate the posture of the face in the face image according to the target rotation matrix. For example, the electronic device can calculate the Euler angle according to the target rotation matrix, and the Euler angle calculated can also know the deflection angle of the face relative to the camera.

[0120] For example, assuming that the target rotation matrix is ​​R, in order to make the rotation matrix R unit orthogonal to calculate the Euler angle, the F-norm can be minimized to obtain the rotation matrix R that satisfies the orthogonal constraint and is closest to the rotation matrix R. based on You can get three face posture angles (i.e. Euler angles), and use θ x ,θ y ,θ z It is expressed as the following formula (7):

[0121]

[0122] Among them, θ x is the pitch angle (rotation around the camera's X axis), θ y is the yaw angle (rotation around the camera's Y axis), θ z is the roll angle (rotation around the camera's Z axis).

[0123] It can be understood that in the embodiments of the present disclosure, not only the facial key points of the facial image are obtained, but also the saliency detection result of the facial image is obtained, and the target facial key points that meet the saliency screening conditions are screened out from the facial key points through the saliency detection result. By using the saliency detection result to screen the facial key points, the quality of the selected facial key points can be improved, thereby helping to improve the accuracy of facial posture estimation based on the selected two-dimensional facial key points and the corresponding three-dimensional coordinate points.

[0124] As mentioned above, the electronic device can select the target face according to the saliency value of some facial key points. In this embodiment, the step of selecting the target facial key points that meet the first saliency screening condition from the facial key points based on the saliency detection result includes:

[0125] Based on the saliency detection result, selecting points that meet the first saliency screening condition other than the predetermined key points among the facial key points to obtain first saliency key points;

[0126] The predetermined key points and the first salient key points are used as the target face key points.

[0127] In the embodiment of the present disclosure, the predetermined key points may be key points at six positions of the left and right eye corners, left and right mouth corners, nose tip and chin, etc., which are associated with the facial features. Of course, they may also be other key points including left and right eyebrow corners, or key points less than six, which are not limited in the embodiment of the present disclosure. However, it should be noted that the predetermined key points are usually selected from some representative key points that best reflect the facial features.

[0128] It should be noted that the electronic device selects points that meet the first significance screening condition other than the predetermined key points among the facial key points according to the significance detection result as the first significance key points. The number of the first significance key points can be a preset number, such as 2 or 3, or a number determined according to the significance value. For example, the first significance key point is a facial key point other than the predetermined key point whose significance value is greater than the preset first significance threshold. Figure 4 As shown, the first significant key point obtained after screening is a key point at a position with higher brightness other than the predetermined key point.

[0129] It can be understood that, on the basis of screening the facial key points in combination with the saliency detection results, because the predetermined key points are predetermined representative points that best reflect the facial features, in the embodiment of the present disclosure, the electronic device pre-screens the representative predetermined key points, and based on the saliency detection results, screens the facial key points other than the predetermined key points, so as to fuse the predetermined key points and the first saliency key points after screening to obtain the target facial key points, thereby further improving the quality of the target facial key points, and thus also helping to further improve the accuracy of facial posture estimation.

[0130] In some embodiments, determining the target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image includes:

[0131] Estimate the rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target face key points and the intrinsic parameters of the camera;

[0132] Calculate the projection point of each three-dimensional coordinate point on the two-dimensional plane according to the intrinsic parameters of the camera and the estimated rotation matrix of the camera;

[0133] Determine the difference between the target face key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, and determine the projection error based on the difference;

[0134] A target rotation matrix of the camera is determined according to the projection error.

[0135] As mentioned above, when the number of target face key points is not 6, the rotation matrix obtained by the fitting algorithm is an estimated value, which can be further optimized to obtain the target rotation matrix. In this regard, in the disclosed embodiment, after the electronic device first estimates the rotation matrix of the camera, it further calculates the projection points of each three-dimensional coordinate point on the two-dimensional plane based on the estimated rotation matrix and the intrinsic parameters of the camera, and calculates the difference between the target face key and the corresponding three-dimensional coordinate point on the two-dimensional plane to determine the projection error, thereby further determining the target rotation matrix of the camera based on the projection error.

[0136] Among them, when calculating the projection point of each three-dimensional coordinate point on the two-dimensional plane, it can be calculated based on the above formula (3), and when determining the difference between the target face key point and the corresponding projection point, it can be calculated by, for example, Euclidean distance, cosine similarity, etc., which is not limited by the embodiment of the present disclosure. In addition, when the electronic device determines the projection error based on the difference between each target face key point and the corresponding projection point, it can be determined based on the mean of each difference, or it can be determined based on the maximum value of each difference, which is not limited by the embodiment of the present disclosure.

[0137] It is understandable that if the accuracy of the rotation matrix is ​​high, then the projection points calculated based on the rotation matrix should theoretically be consistent with the target key points, and the corresponding projection error is small; if they are inconsistent, it means that the accuracy of the rotation matrix is ​​not high enough, and the corresponding projection error is large. Therefore, the embodiment of the present disclosure can be further optimized based on the projection error to obtain a target rotation matrix with higher accuracy. By calculating the projection point verification in this way, the accuracy of the target rotation matrix can be improved.

[0138] In some embodiments, determining the target rotation matrix of the camera according to the projection error includes:

[0139] If the projection error is less than a preset error threshold, using the estimated rotation matrix of the camera as the target rotation matrix of the camera;

[0140] If the projection error is greater than or equal to the preset error threshold, the target facial key points are updated, and the rotation matrix of the camera is estimated based on the updated target facial key points to determine the projection error, until the projection error is less than the preset error threshold to obtain the target rotation matrix of the camera.

[0141] In the embodiment of the present disclosure, if the projection error is less than a preset error threshold, it means that the estimated rotation matrix has high accuracy, and thus the rotation matrix can be used as the target rotation matrix; if the projection error is greater than or equal to the preset error threshold, it means that the accuracy of the rotation matrix is ​​insufficient. In this regard, the electronic device re-estimates the rotation matrix by updating the target facial key points, and estimates the rotation matrix and further calculates the projection error in an iterative manner, for example, after updating the target facial key points one or more times, the target rotation matrix of the camera is obtained when the projection error is less than the preset error threshold.

[0142] It should be noted that the electronic device updates the target face key points, which can be determined by re-detecting the face key points using a new key point detection method, or by updating the target face key points based on points other than the target face key points in the original face key points, which is not limited here. In addition, the process of the electronic device estimating the rotation matrix of the camera based on the updated target face key points is that the electronic device searches for the corresponding three-dimensional coordinate points in the preset three-dimensional face model based on the updated target face key points, and estimates the rotation matrix in combination with the updated target face key points and the corresponding three-dimensional coordinate points and the intrinsic parameters of the camera. Moreover, the electronic device determines the projection error based on the re-estimated rotation matrix by first calculating the projection points again using the re-estimated rotation matrix and the updated three-dimensional coordinate point set camera intrinsic parameters, and calculates the differences based on the updated target face key points and the re-calculated projection points, and determines the projection error based on the differences, which will not be described in detail here.

[0143] It can be understood that in the embodiments of the present disclosure, when the projection error is greater than or equal to the preset error threshold, the rotation matrix is ​​re-determined and the projection error is calculated after updating the target facial key points, and the target rotation matrix is ​​further determined in an iterative manner based on the projection error. Through this iterative optimization method, the accuracy of determining the target rotation matrix can be improved, which is beneficial to the accuracy of facial posture estimation.

[0144] In some embodiments, updating the target face key points includes:

[0145] Based on the difference between the target face key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, selecting a point among the face key points that meets a preset difference condition as the key point to be replaced;

[0146] Based on the saliency analysis result, selecting points among the facial key points other than the target facial key points that meet the second saliency screening condition as second saliency key points;

[0147] The second salient key point replaces the key point to be replaced in the target face key point to obtain an updated target face key point.

[0148] As mentioned above, the target face key points can be updated based on the points other than the target face key points in the face key points. In the disclosed embodiment, the electronic device selects the points in the target face key points that meet the preset difference condition as the key points to be replaced based on the difference between the target face key points and the projection points of the corresponding three-dimensional coordinate points on the two-dimensional plane, and then selects the points that meet the second significance screening condition from the candidate key points as the second significance key points to replace the key points to be replaced based on the significance analysis result, thereby updating the target face key points, wherein the points that meet the second significance screening condition can be the second preset number of target face key points with the top significance value ranking, or the target face key points with the significance value greater than the second preset significance threshold; the points that meet the preset difference condition can refer to the third preset number of points in the target face key points with the top difference value ranking, or the points in the target face key points with the difference value greater than the preset difference threshold; the candidate key points are points other than the target face key points in the face key points.

[0149] It can be understood that in the embodiments of the present disclosure, points that meet the second saliency screening condition are screened based on the saliency detection result to replace the points in the target face key points that meet the preset difference condition. That is, on the one hand, the key points in the target face key points that have a large impact on the projection error are eliminated, and on the other hand, possible more interesting face key points are introduced to update the target face key points, so that the quality of the updated target face key points is better, which can be beneficial to the estimation of facial posture.

[0150] In some embodiments, determining the difference between the target facial key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, and determining the projection error based on the difference includes:

[0151] Determine the Euclidean distance between each of the target face key points and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane;

[0152] Based on the Euclidean distances corresponding to the target facial key points, an average value of the Euclidean distances is determined, and the average value is determined as the projection error.

[0153] In this embodiment, the electronic device determines the difference based on the Euclidean distance, and determines the projection error based on the average value of the Euclidean distance. Assuming that there are 6+M key points in the target face, where 6 is the number of predetermined key points and M is the number of first significant key points, the projection error can be expressed by the following formula (8):

[0154]

[0155] Among them, p i '=[ui ',v i ',1] T is the projection point of the three-dimensional coordinate point corresponding to the key point on the two-dimensional plane, and d is the projection error.

[0156] It can be understood that in the embodiment of the present disclosure, determining the projection error based on the average value of the Euclidean distance can more comprehensively integrate the conditions of each target facial key point, thereby helping to more accurately determine the target rotation matrix based on the projection error to perform facial posture estimation.

[0157] In some embodiments, obtaining facial key points in a facial image includes:

[0158] Extracting features from the face image using a feature extraction layer of a preset face key point detection model to obtain face features;

[0159] Processing the facial features using the heat map output network of the preset facial key point detection model to obtain a downsampled feature heat map;

[0160] Processing the feature heat map using the offset coordinate output network of the preset facial key point detection model to obtain the offset coordinates corresponding to each feature point of the feature heat map;

[0161] The facial key points in the facial image are determined based on the feature heat map and each of the offset coordinates.

[0162] In this embodiment, the preset face key point detection model is a trained model, and the model includes a feature extraction layer, a heat map output network layer, and an offset coordinate output network layer, wherein the heat map output network layer and the offset coordinate output network layer are nested, and the offset coordinate output network layer performs local coordinate regression on the low-resolution feature heat map to determine the offset coordinates corresponding to each feature point of the feature heat map. When the preset face key point detection model is trained, the local coordinate regression is performed on the basis of the heat map regression of the heat map output network layer. The above-mentioned preset face key point detection model is also called a PIPNet model.

[0163] Since the heat map regression method can achieve higher accuracy and the robustness of coordinate regression is relatively reliable, the embodiments of the present disclosure can be combined with coordinate regression to process low-resolution feature heat maps. It can be understood that the electronic device can obtain key point detection results with higher accuracy based on the trained preset face key point detection model while greatly reducing the amount of calculation.

[0164] Figure 4 A flow chart of a face entry method provided in an embodiment of the present disclosure, as shown in the figure, includes the following steps:

[0165] S21, obtaining a face image of a target object;

[0166] S22, obtaining the face posture of the target object by using a preset face posture estimation method; wherein the preset face posture estimation method is any of the aforementioned face posture estimation methods;

[0167] S23a, in response to the facial posture satisfying a preset posture condition, saving the facial image of the target object;

[0168] S23b: In response to the face posture not satisfying the preset posture condition, outputting prompt information.

[0169] In the embodiment of the present disclosure, the electronic device that performs the face entry method and the electronic device that performs face posture estimation can be the same device or different devices. When they are different devices, the electronic device that performs the face entry method can send the face image of the target object to the electronic device that performs face posture estimation, and receive the face posture estimation result returned by the electronic device for face posture estimation for further judgment. In step S21, taking the electronic device as a smart phone as an example, the target object is the mobile phone user; taking the electronic device as a bank's information entry device as an example, the target object is the bank user; taking the electronic device as an access control device as an example, the target object is the user of the access control device, and the embodiment of the present disclosure does not limit this.

[0170] In the disclosed embodiment, the electronic device can obtain the face pose of the target object based on any of the aforementioned face pose estimation methods, and save the face image of the target object when the face pose meets the preset pose condition, and output prompt information when the face pose does not meet the preset pose condition. The preset pose condition can be a range of pitch angle and / or yaw angle set manually, and the face under the preset pose condition is usually a frontal face.

[0171] It can be understood that in the embodiments of the present disclosure, the electronic device of the face entry method can obtain the face posture of the target object more accurately based on the aforementioned face estimation method, and thus can more accurately enter the face image with normal posture when the face posture meets the preset posture conditions, so as to facilitate the subsequent unlocking after feature comparison based on the entered face image, thereby improving the security of face unlocking; in addition, when the face posture does not meet the preset posture conditions, a prompt message is output, which can facilitate the target object to enter the face immediately or in an intelligent manner.

[0172] In some embodiments, in response to the facial posture not satisfying a preset posture condition, outputting prompt information includes:

[0173] In response to the face posture not satisfying the preset posture condition, outputting prompt information indicating that the face image is unqualified;

[0174] and / or,

[0175] In response to the facial posture not satisfying the preset posture condition, outputting facial posture adjustment information corresponding to satisfying the preset posture condition.

[0176] In this embodiment, when the face posture of the target object does not meet the preset posture condition, a prompt message indicating that the face image is unqualified may be broadcasted by voice or displayed in text, and face posture adjustment information corresponding to meeting the preset posture condition may also be output in the above voice or text form. It should be noted that when the electronic device outputs the face posture adjustment information, it may also output prompt information indicating the acquisition time.

[0177] It should be noted that, in the embodiments of the present disclosure, if the electronic device fails to capture the face image of the target object, it may also output corresponding prompt information. For example, Figure 5 An example diagram of an electronic device outputting prompt information provided by an embodiment of the present disclosure, such as Figure 5 If the electronic device has captured the face image of the target object, the prompt information "Please keep your face in the viewfinder" in L0 is output. After the electronic device has captured the face image of the target object and compared the face posture of the target object with the preset posture condition, if it is determined that the current target object raises its head at an angle exceeding the elevation angle range corresponding to the preset posture condition, it is determined that the target object raises its head excessively, and the electronic device outputs the prompt information "Please lower your head" in L1; if it is determined that the current target object lowers its head at an angle exceeding the depression angle range corresponding to the preset posture condition, it is determined that the target object lowers its head excessively, and the electronic device outputs the prompt information "Please raise your head" in L2; and if it is determined that the current target object's left head tilt angle exceeds the left yaw angle range corresponding to the preset posture condition, it is determined that the target object is excessively tilted to the left, and the electronic device outputs the prompt information "Please turn your head to the right" in L3; if it is determined that the current target object's right head tilt angle exceeds the right yaw angle range corresponding to the preset posture condition, it is determined that the target object is excessively tilted to the right, and the electronic device outputs the prompt information "Please turn your head to the left" in L4. As shown in Figures L2, L3 and L4, the electronic device also outputs information marked with B to dynamically bold the viewfinder to indicate the acquisition time.

[0178] It can be understood that in the embodiment of the present disclosure, the face entry method outputs prompt information when the face posture does not meet the preset posture conditions, and has good intelligence. In particular, if the output is face posture adjustment information corresponding to the preset posture conditions, it is convenient for the target object to adjust the face posture accurately and immediately according to the prompt, which can improve the efficiency of face entry.

[0179] Figure 6 FIG. 1 is a diagram of a face posture estimation device according to an exemplary embodiment. Figure 6 , the device comprises:

[0180] A first acquisition module 101 is configured to acquire facial key points in a facial image and a saliency detection result corresponding to the facial image;

[0181] A selection module 102 is configured to select target facial key points that meet a first saliency screening condition from among the facial key points based on the saliency detection result;

[0182] A first determination module 103 is configured to determine the three-dimensional coordinate points corresponding to the target facial key points in a preset three-dimensional facial model;

[0183] A second determination module 104 is configured to determine a target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image;

[0184] The estimation module 105 is configured to estimate the facial posture in the facial image based on the target rotation matrix of the camera.

[0185] In some embodiments, the selection module 102 is further configured to select points that meet the first significance screening condition other than predetermined key points among the facial key points based on the significance detection result to obtain first significant key points; and use the predetermined key points and the first significant key points as the target facial key points.

[0186] In some embodiments, the second determination module 104 is configured to estimate the rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera; calculate the projection points of each three-dimensional coordinate point on the two-dimensional plane according to the intrinsic parameters of the camera and the estimated rotation matrix of the camera; determine the difference between the target facial key points and the projection points of the corresponding three-dimensional coordinate points on the two-dimensional plane, and determine the projection error based on the difference; determine the target rotation matrix of the camera according to the projection error.

[0187] In some embodiments, the second determination module 104 is further configured to use the estimated rotation matrix of the camera as the target rotation matrix of the camera if the projection error is less than a preset error threshold; if the projection error is greater than or equal to the preset error threshold, update the target facial key points, and estimate the rotation matrix of the camera based on the updated target facial key points to determine the projection error, until the projection error is less than the preset error threshold, the target rotation matrix of the camera is obtained.

[0188] In some embodiments, the second determination module 104 is further configured to select points in the target facial key points that meet a preset difference condition as key points to be replaced based on the difference between each facial key point in the target key point set and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane; based on the significance analysis result, select points in the facial key points that meet a second significance screening condition other than the target facial key points as second significance key points; replace the key points to be replaced in the target facial key points with the second significance key points to obtain updated target facial key points.

[0189] In some embodiments, the second determination module 104 is further configured to determine the Euclidean distance between each of the target facial key points and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane; based on the Euclidean distance corresponding to each of the target facial key points, determine the average value of the Euclidean distance, and determine the average value as the projection error.

[0190] In some embodiments, the first acquisition module 101 is further configured to perform feature extraction on the face image using a feature extraction layer of a preset face key point detection model to obtain face features; process the face features using a heat map output network of the preset face key point detection model to obtain a downsampled feature heat map; process the feature heat map using an offset coordinate output network of the preset face key point detection model to obtain offset coordinates corresponding to each feature point of the feature heat map; and determine the face key points in the face image based on the feature heat map and each offset coordinate.

[0191] Figure 7 FIG. 1 is a diagram showing a face recording device according to an exemplary embodiment. Figure 7 , the device comprises:

[0192] The second acquisition module 201 is configured to acquire a face image of a target object;

[0193] The obtaining module 202 is configured to obtain the face posture of the target object by using a preset face posture estimation method; wherein the preset face posture estimation method is any one of the aforementioned face posture estimation methods;

[0194] A saving module 203a, configured to save the face image of the target object in response to the face posture satisfying a preset posture condition;

[0195] The output module 203b is configured to output prompt information in response to the face posture not satisfying the preset posture condition.

[0196] In some embodiments, the output module 204 is further configured to output a prompt message indicating that the facial image is unqualified in response to the facial posture not satisfying the preset posture condition; and / or, in response to the facial posture not satisfying the preset posture condition, output facial posture adjustment information corresponding to satisfying the preset posture condition.

[0197] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0198] Figure 8 A frame of an electronic device according to an exemplary embodiment is shown Figure 1 For example, the electronic device (device 800) may be a mobile phone or the like for executing the face pose estimation method or the face entry method.

[0199] Reference Figure 8 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0200] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0201] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0202] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0203] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0204] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0205] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0206] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an infrared light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0207] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0208] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0209] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0210] Fig. 9 A frame of an electronic device according to an exemplary embodiment is shown Figure 2For example, the electronic device (device 900) may be a server for executing a face pose estimation method or a face entry method.

[0211] Reference Fig. 9 The apparatus 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as an application, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the above method.

[0212] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0213] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 932 including instructions, and the instructions can be executed by the processing component 922 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0214] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a face pose estimation method, the method comprising:

[0215] Obtaining key points of a face image and a saliency detection result corresponding to the face image;

[0216] Based on the saliency detection result, selecting target facial key points that meet a first saliency screening condition from among the facial key points;

[0217] Determine the three-dimensional coordinate points corresponding to the target facial key points in the preset three-dimensional face model;

[0218] Determine a target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image;

[0219] The facial pose in the facial image is estimated based on the target rotation matrix of the camera.

[0220] Alternatively, a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a face recording method, the method comprising:

[0221] Obtain a face image of the target object;

[0222] Obtaining the face posture of the target object using a preset face posture estimation method; wherein the preset face posture estimation method is any of the aforementioned face posture estimation methods;

[0223] In response to the facial posture satisfying a preset posture condition, saving the facial image of the target object;

[0224] In response to the facial posture not satisfying the preset posture condition, outputting prompt information.

[0225] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims above.

[0226] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for estimating a face posture, characterized in that: The method comprises: Obtaining facial key points in a facial image and a saliency detection result corresponding to the facial image; Based on the saliency detection result, selecting target facial key points that meet a first saliency screening condition from among the facial key points; Determine the three-dimensional coordinate points corresponding to the target facial key points in the preset three-dimensional face model; Determine a target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image; The facial pose in the facial image is estimated based on the target rotation matrix of the camera.

2. The method according to claim 1, characterized in that The selecting, based on the saliency detection result, target facial key points from among the facial key points that meet a first saliency screening condition comprises: Based on the saliency detection result, selecting points that meet the first saliency screening condition other than the predetermined key points among the facial key points to obtain first saliency key points; The predetermined key points and the first salient key points are used as the target face key points.

3. The method according to claim 1 or 2, characterized in that: The step of determining the target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image comprises: Estimate the rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target face key points and the intrinsic parameters of the camera; Calculate the projection point of each three-dimensional coordinate point on the two-dimensional plane according to the intrinsic parameters of the camera and the estimated rotation matrix of the camera; Determine the difference between the target face key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, and determine the projection error based on the difference; A target rotation matrix of the camera is determined according to the projection error.

4. The method according to claim 3, characterized in that Determining the target rotation matrix of the camera according to the projection error includes: If the projection error is less than a preset error threshold, using the estimated rotation matrix of the camera as the target rotation matrix of the camera; If the projection error is greater than or equal to the preset error threshold, the target facial key points are updated, and the rotation matrix of the camera is estimated based on the updated target facial key points to determine the projection error, until the projection error is less than the preset error threshold to obtain the target rotation matrix of the camera.

5. The method according to claim 4, characterized in that The updating of the target face key points includes: Based on the difference between the target face key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, select the point in the target face key point that meets the preset difference condition as the key point to be replaced; Based on the saliency analysis result, selecting points among the facial key points other than the target facial key points that meet the second saliency screening condition as second saliency key points; The second salient key point replaces the key point to be replaced in the target face key point to obtain an updated target face key point.

6. The method according to claim 3, characterized in that Determining the difference between the target face key point and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane, and determining the projection error based on the difference, includes: Determine the Euclidean distance between each of the target face key points and the projection point of the corresponding three-dimensional coordinate point on the two-dimensional plane; Based on the Euclidean distances corresponding to the target facial key points, an average value of the Euclidean distances is determined, and the average value is determined as the projection error.

7. The method according to claim 1, characterized in that The step of obtaining key facial points in a facial image includes: Extracting features from the face image using a feature extraction layer of a preset face key point detection model to obtain face features; Processing the facial features using the heat map output network of the preset facial key point detection model to obtain a downsampled feature heat map; Processing the feature heat map using the offset coordinate output network of the preset facial key point detection model to obtain the offset coordinates corresponding to each feature point of the feature heat map; The facial key points in the facial image are determined based on the feature heat map and each of the offset coordinates.

8. A face recording method, characterized in that: The method comprises: Obtain a face image of the target object; Obtaining the face posture of the target object using a preset face posture estimation method; wherein the preset face posture estimation method is any one of claims 1-7; In response to the facial posture satisfying a preset posture condition, saving the facial image of the target object; In response to the facial posture not satisfying the preset posture condition, outputting prompt information.

9. The method according to claim 8, characterized in that The step of outputting prompt information in response to the face posture not satisfying a preset posture condition comprises: In response to the face posture not satisfying the preset posture condition, outputting prompt information indicating that the face image is unqualified; and / or, In response to the facial posture not satisfying the preset posture condition, outputting facial posture adjustment information corresponding to satisfying the preset posture condition.

10. A face posture estimation device, characterized in that: The device comprises: A first acquisition module is configured to acquire facial key points in a facial image and a saliency detection result corresponding to the facial image; a selection module is configured to select target facial key points that meet a first saliency screening condition from among the facial key points based on the saliency detection result; A first determination module is configured to determine the three-dimensional coordinate points corresponding to the target facial key points in a preset three-dimensional facial model; A second determination module is configured to determine a target rotation matrix of the camera based on the three-dimensional coordinate points corresponding to the target facial key points and the intrinsic parameters of the camera that captures the facial image; An estimation module is configured to estimate the facial posture in the facial image based on the target rotation matrix of the camera.

11. A face recording device, characterized in that: The device comprises: A second acquisition module is configured to acquire a face image of a target object; An acquisition module, configured to obtain the face posture of the target object by using a preset face posture estimation method; wherein the preset face posture estimation method is any one of claims 1-7; A saving module, configured to save the face image of the target object in response to the face posture satisfying a preset posture condition; The output module is configured to output prompt information in response to the face posture not satisfying the preset posture condition.

12. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method according to any one of claims 1 to 7 or any one of claims 8 to 9.

13. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 7, or any one of claims 8 to 9.

Citation Information

Patent Citations

  • Facial pose estimation method and device

    CN105447462A

  • Method and apparatus for determining face attitude

    CN106503671A

  • Video abstraction generation method based on visual saliency detection and hierarchical clustering method

    CN110347870A

  • Face image processing method and device for model training

    CN110956065A

  • Face pose estimation method based on 3D face key points and geometric projection

    CN111259739A

Cited By

  • Data acquisition method, device and system, electronic equipment and readable storage medium

    CN120071420A

  • Face video processing method and device, electronic equipment and storage medium

    CN121170662A

  • Face image screening method and device, equipment, storage medium and program product

    CN121545205A