Face orientation detection method and apparatus, and device, vehicle, medium and program product

Abstract

The present disclosure relates to a face orientation detection method and apparatus, and a device, a vehicle, a medium and a program product. The method comprises: performing key point detection on a face in an image to be subjected to detection, so as to obtain a first facial key point; performing pose estimation on the face on the basis of the first facial key point and a second facial key point, so as to obtain the turning angle of the face, wherein the second facial key point is a preset facial key point with a known turning angle; performing calculation on the basis of the turning angle of the face and a turning angle threshold value corresponding to each preset orientation, so as to obtain the probability of the face directly in each preset orientation, wherein the probability decreases along with the increase of the difference between the turning angle of the face and the turning angle threshold value; and on the basis of the probability of the face directly in each preset orientation, determining a facial orientation detection result for the face. On the basis of the technical solution of the present disclosure, the consumption of computing resources can be reduced, and the real-time performance of face orientation detection is improved.

Description

Face orientation detection method, device, equipment, vehicle, medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application No. 2024102114890 filed on February 26, 2024, entitled “Face orientation detection method, device, equipment and medium”, filed by Beijing Rockwell Technology Co., Ltd. The full text of which is incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of artificial intelligence technology, and in particular to a face orientation detection method, apparatus, device, vehicle, medium, and program product. Background Art

[0004] Machine learning and deep learning are hot research areas in artificial intelligence. Face orientation detection typically uses deep learning classification and machine learning methods to process images and output face orientation detection results. For example, a single-channel image captured by a monocular camera is fed into a pre-trained classification detection model to output face orientation detection results.

[0005] In related technologies, deep learning classification methods train models such as convolutional neural networks using large amounts of labeled data, and use the trained models to identify facial orientation. Machine learning methods, on the other hand, rely on feature engineering and classification algorithms to extract features from facial images and build models to predict facial orientation. Both approaches require large amounts of labeled data to train the models, resulting in high data acquisition and labeling costs, high computational complexity, and significant computing resources. This can lead to delayed output of facial orientation detection results in practical applications. Summary of the Invention

[0006] In order to solve the above technical problems, the present disclosure provides a face orientation detection method, apparatus, device, vehicle, medium and program product.

[0007] The present disclosure first provides a face orientation detection method, including:

[0008] Perform key point detection on the face in the image to be tested to obtain the first face key point;

[0009] Estimating the face pose based on the first and second facial key points to obtain a rotation angle of the face; wherein the second facial key point is a pre-set facial key point with a known rotation angle;

[0010] Calculating, based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation, a probability that the face is facing each preset orientation; wherein the probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases;

[0011] A face orientation detection result of the face is determined based on the probability that the face is facing each preset orientation.

[0012] The present disclosure also provides a face orientation detection device, comprising:

[0013] A detection module is configured to perform key point detection on a face in a test image to obtain a first face key point;

[0014] a generating module configured to estimate the pose of the face based on the first facial key points and the second facial key points to obtain a rotation angle of the face; wherein the second facial key points are pre-set facial key points with a known rotation angle;

[0015] a prediction module configured to calculate, based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation, a probability that the face is facing each preset orientation; wherein the probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases;

[0016] The determination module is configured to determine a face orientation detection result of the face according to the probability that the face is facing each preset orientation.

[0017] An embodiment of the present disclosure also provides an electronic device, comprising: a processor, a memory configured to store instructions executable by the processor; the processor, configured to read the executable instructions from the memory and execute the instructions to implement the face orientation detection method as described above.

[0018] An embodiment of the present disclosure further provides a vehicle, comprising the face orientation detection device as described above, or the electronic device as described above.

[0019] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the face orientation detection method as described above is implemented.

[0020] An embodiment of the present disclosure further provides a computer program product, which includes a computer program. When the computer program is executed by a processor of an electronic device, the face orientation detection method as described above can be implemented.

[0021] Compared with the prior art, the technical solution provided by the embodiment of the present disclosure has the following advantages: by performing key point detection on the face in the image to be tested, a first face key point is obtained, and based on the first face key point and a second face key point with a preset known rotation angle, the rotation angle of the face is obtained, and then, according to the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation, the probability of the face facing each preset orientation is calculated, and the face orientation detection result of the face is determined according to the probability of the face facing each preset orientation. Therefore, there is no need to classify the detection model when detecting the face orientation, but instead the second face key point and the rotation angle threshold of each preset orientation are set, and the first face key point and the rotation angle threshold of each preset orientation are used to determine the face orientation. The face key point and the second face key point are used to obtain the rotation angle of the face, and the probability calculation is performed in combination with the rotation angle threshold to obtain the face orientation detection result. Mathematical calculation is used to replace the classification detection model based on deep learning / machine learning, which reduces the data collection and annotation costs of the classification detection model. In addition, only the face key point detection model needs to be introduced to perform key point detection to obtain the face key points. The face key point detection model consumes less computing resources than the classification detection model, which solves the problem of high computing resource consumption and easy delay in actual application in the face orientation detection method in related technologies, reduces the computational complexity, reduces computing resource consumption, and improves the real-time performance of face orientation detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] FIG1 is a schematic diagram of a flow chart of a face orientation detection method provided by an embodiment of the present disclosure;

[0025] FIG2 is a flow chart of another face orientation detection method provided by an embodiment of the present disclosure;

[0026] FIG3 is a schematic diagram of a flow chart of calculating the probability of a face facing a target according to an embodiment of the present disclosure;

[0027] FIG4 is a schematic structural diagram of a face orientation detection device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] Figure 1 is a flow chart of a face orientation detection method provided in an embodiment of the present disclosure. The method provided in an embodiment of the present disclosure can be executed by a face orientation detection device, which can be implemented using software and / or hardware and can be integrated into any electronic device with computing capabilities, such as a vehicle-mounted terminal.

[0031] As shown in FIG1 , the face orientation detection method provided by the embodiment of the present disclosure may include:

[0032] Step 101: perform key point detection on a face in a test image to obtain a first face key point.

[0033] In this embodiment, an image to be tested is obtained and facial key point detection is performed on the image to obtain a first facial key point of the face in the image to be tested. The image to be tested may be an image captured during face orientation detection. For example, a camera is provided in a vehicle to capture images within the vehicle, and the image captured by the camera may be used as the image to be tested.

[0034] As an example, the image to be tested is input into a pre-trained facial key point detection model for key point detection to obtain the first facial key point of the face in the image to be tested. The facial key point detection model can be obtained based on machine learning or deep learning technology. The input of the facial key point detection model is an image, and the output of the facial key point detection model can be the facial key point in the image.

[0035] Step 102: Estimate the face pose based on the first facial key point and the second facial key point to obtain a rotation angle of the face.

[0036] In this embodiment, the second facial key point is a preset facial key point with a known rotation angle.

[0037] Optionally, the face represented by the second facial key point can be a face facing the camera or a face at other rotation angles, that is, the position of the second facial key point relative to the camera is known, and no specific limitation is given here.

[0038] There are many ways to determine the rotation angle of the face in the image to be tested.

[0039] As a possible implementation method, the camera pose estimation function can be used to estimate the pose of the face; wherein, the camera pose estimation function is used to estimate the pose of the camera, and the pose of the camera includes the rotation and translation matrices of the camera. The camera pose estimation function can be, for example, the solvePnP function, which estimates the rotation and translation matrices of the camera through the position of the known three-dimensional point in the world coordinate system and the projection point on the image.

[0040] In this implementation, a fitting method of the camera pose estimation function is adopted. The calibrated camera parameters are obtained in advance through camera calibration. The camera parameters include intrinsic parameters, extrinsic parameters and distortion parameters. The calibrated camera parameters, the first face key point and the second face key point are used as inputs of the camera pose estimation function. The rotation vector and translation vector output by the camera pose estimation function are used to represent the rotation vector and translation vector of the face relative to the camera, so as to express the relative pose of the face in space.

[0041] Optionally, the camera parameters are iterated using the calibrated camera parameters as initial values. During each iteration, the first facial key point and the second facial key point are converted to the same coordinate system using the current camera parameters, and the first facial key point and the second facial key point in the same coordinate system are compared until the error between the first facial key point and the second facial key point in the same coordinate system is minimized. Then, the iterated camera parameters corresponding to the first facial key point and the second facial key point with the smallest error in the same coordinate system are converted into the rotation angle of the face.

[0042] As another possible implementation method, the first facial key point is compared with the second facial key point of the standard face to obtain the angle between the face represented by the first facial key point and the face represented by the second facial key point. Based on the angle and the known rotation angle of the second facial key point, the rotation angle of the face in the test image is generated.

[0043] Step 103 , calculating based on the rotation angle of the face and the rotation angle thresholds corresponding to the respective preset orientations to obtain the probability that the face is facing the respective preset orientations.

[0044] The above probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases.

[0045] In this embodiment, at least one orientation is preset, and a corresponding rotation angle threshold is set for each orientation. This threshold can be predetermined based on experiments and is used to represent the orientation. After the rotation angle of the face is obtained, a calculation is performed based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation to obtain the probability that the face is facing each preset orientation.

[0046] In the technical solution provided by the present disclosure, the rotation angle determined by the method provided by any embodiment includes a yaw angle, a pitch angle, and a roll angle, and the rotation angle threshold corresponding to each preset orientation includes a yaw threshold, a pitch threshold, and a roll threshold;

[0047] Accordingly, in the technical solution provided by the present disclosure, the probability of the face facing each preset orientation is calculated based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation, which can be achieved by the following methods:

[0048] For each preset orientation, the yaw angle and yaw threshold, pitch angle and pitch threshold, and roll angle and roll threshold are respectively input into the preset probability scoring function for calculation, and the first sub-probability corresponding to the yaw angle, the second sub-probability corresponding to the pitch angle, and the third sub-probability corresponding to the roll angle are output; based on at least one of the first sub-probability, the second sub-probability, and the third sub-probability, the probability that the face is facing the preset orientation is determined by using the square root average method.

[0049] For example, taking one of the preset orientations as an example, the first sub-probability is obtained by calculating based on the yaw angle of the face and the yaw threshold of the orientation, the second sub-probability is obtained by calculating based on the pitch angle of the face and the pitch threshold of the orientation, and the third sub-probability is obtained by calculating based on the roll angle of the face and the roll threshold of the orientation; based on the first sub-probability, the second sub-probability and the third sub-probability, the probability that the face is facing the preset orientation is determined.

[0050] Among them, the implementation method of comprehensively determining the final probability based on multiple sub-probabilities includes but is not limited to square root averaging, weighted averaging, etc.

[0051] Step 104 : determining a face orientation detection result of the face according to the probability that the face is facing each preset orientation.

[0052] The probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases.

[0053] In any embodiment of the present disclosure, a greater probability indicates a higher possibility that the face is facing the preset direction.

[0054] In the technical solutions provided by the embodiments of the present disclosure, after determining the probability of a face facing each preset orientation through any embodiment, the face orientation detection result of the face can be determined by the following methods:

[0055] If the probability of the face facing each preset direction is less than or equal to the threshold, it is determined that the face is not facing any preset direction. If there is a target probability greater than the threshold among the probabilities of the face facing each preset direction, the preset direction corresponding to the maximum value of the target probability is taken as the direction of the face. The threshold can be determined based on experiments or set according to the needs of the actual application scenario, and is not limited here.

[0056] As an example, for the voice wake-up scenario in a vehicle, the pre-set orientation may be the orientation of the device to be awakened, and a corresponding rotation angle threshold is set for the orientation of the device to be awakened. When the user says the specified wake-up word through voice collection and voice recognition, the aforementioned steps 101 to 103 are performed on the test image captured by the camera in the vehicle to obtain the probability that the user's face is facing the direction of the device to be awakened. If the probability is greater than or equal to the threshold, it is determined that the face is facing the device to be awakened, and the wake-up operation is performed at this time. If the probability is less than the threshold, it is determined that the face is not facing the device to be awakened. In some voice wake-up scenarios, face orientation detection is used to assist in voice wake-up to reduce the number of false wake-ups and improve the accuracy of voice wake-up.

[0057] According to the technical solution of the embodiment of the present disclosure, key point detection is performed on the face in the image to be tested to obtain a first face key point, and the rotation angle of the face is obtained based on the first face key point and a preset second face key point with a known rotation angle. Then, calculation is performed based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation to obtain the probability that the face is facing each preset orientation. Based on the probability that the face is facing each preset orientation, the face orientation detection result of the face is determined. Therefore, there is no need to classify the detection model when detecting the face orientation. Instead, the second face key point and the rotation angle threshold of each preset orientation are set, and the rotation angle of the face is obtained through the first face key point and the second face key point. Angle, combined with the rotation angle threshold to perform probability calculation to obtain the face orientation detection result, and mathematical calculation is used to replace the classification detection model based on deep learning / machine learning, and only the face key point detection model needs to be introduced. The face key point detection model consumes less computing resources than the classification detection model, and solves the problem that the face orientation detection method in the related technology consumes a lot of computing resources and is prone to delays in practical applications. It reduces the computational complexity, reduces the consumption of computing resources, improves the real-time performance of face orientation detection, and reduces the data collection and annotation costs of the classification detection model. When adding / reducing the orientation that needs to be paid attention to, it is only necessary to increase / decrease the rotation angle threshold without retraining the model.

[0058] Based on the above embodiment, FIG2 is a flow chart of another face orientation detection method provided by an embodiment of the present disclosure. As shown in FIG2 , the method includes:

[0059] Step 201: perform key point detection on the face in the image to be tested to obtain the first face key point.

[0060] Step 202: Using the calibrated camera parameters, the first facial key points, and the second facial key points as inputs to a camera pose estimation function to obtain a rotation vector and a translation vector.

[0061] The second facial key point is a pre-set facial key point with a known rotation angle. The first facial key point is a two-dimensional facial key point in the image coordinate system, and the second facial key point is a three-dimensional facial key point in the world coordinate system.

[0062] In this embodiment, a camera pose estimation function is used to determine the rotation angle of the face. The camera pose estimation function uses a camera projection model to establish the relationship between the world coordinate system and the image coordinate system. Using the input three-dimensional points and the corresponding projection points on the two-dimensional image, the camera pose estimation function estimates the initial pose of the camera. It then iteratively adjusts the camera's rotation and translation to minimize the error between the projection points of the three-dimensional points on the image and the true two-dimensional points. When the optimization process converges, the camera pose estimation function outputs the camera's rotation vector and translation vector to describe the camera's pose in space. In this step, the fitting method in the camera pose estimation function is used. The camera's intrinsic parameters, extrinsic parameters, and distortion parameters are pre-calibrated. The calibrated camera parameters, the first facial key point, and the second facial key point are used as input to the camera pose estimation function. The output rotation vector and translation vector are used to represent the rotation vector and translation vector of the face relative to the camera, thereby expressing the face's relative pose in space.

[0063] As an example, the camera parameters are iterated using the calibrated camera parameters as the initial values. In each iteration, the current camera parameters are used to convert the first facial key point from the image coordinate system to the world coordinate system, and the first facial key point and the second facial key point in the world coordinate system are compared until the error between the first facial key point and the second facial key point in the world coordinate system is minimized, and the target camera parameters are converted into the rotation angle of the face.

[0064] The target camera parameters include the iterated camera parameters corresponding to the first minimum error; the first minimum error includes the minimum value of the error between the first facial key point and the second facial key point in the world coordinate system.

[0065] In this example, during each iteration, the first facial key point is converted from the image coordinate system to the world coordinate system according to the calibrated camera intrinsic parameters, distortion parameters, and the currently adjusted camera rotation and translation, and compared with the second facial key point to obtain the error in this iteration. The camera parameters are adjusted through multiple iterations, and the error in the iteration process is obtained by the above method to minimize the error between the first facial key point and the second facial key point in the world coordinate system, thereby obtaining the target camera parameters, and the rotation vector and translation vector output by the camera pose estimation function corresponding to the target camera parameters are converted into the rotation angle of the face.

[0066] As another example, the camera parameters are iterated using the calibrated camera parameters as the initial values. During each iteration, the second facial key point is converted from the world coordinate system to the image coordinate system using the current camera parameters, and the first facial key point and the second facial key point in the image coordinate system are compared until the error between the first facial key point and the second facial key point in the image coordinate system is minimized, and the target camera parameters are converted into the rotation angle of the face.

[0067] The target camera parameters include the iterated camera parameters corresponding to the second minimum error; the second minimum error includes the minimum value of the error between the first facial key point and the second facial key point in the image coordinate system.

[0068] In this example, in each iteration, the second facial key point is converted from the world coordinate system to the image coordinate system according to the calibrated camera intrinsic parameters, distortion parameters and the current adjusted camera rotation and translation, and compared with the first facial key point to obtain the error in this iteration. After multiple iterations, the iterative camera parameters are obtained, and the error in the iterative process is obtained by the same method mentioned above to minimize the error between the first facial key point and the second facial key point in the image coordinate system, thereby obtaining the target camera parameters, and converting the rotation vector and translation vector output by the camera pose estimation function corresponding to the target camera parameters into the rotation angle of the face.

[0069] It should be noted that in the embodiments of the present disclosure, the first method for determining the rotation angle of the face provided in the aforementioned embodiments and / or the second method for determining the rotation angle of the face can be used to determine the rotation angle of the face.

[0070] The first method for determining the rotation angle of the face may be:

[0071] The camera parameters are iterated using the calibrated camera parameters as initial values. In each iteration, the current camera parameters are used to transform the first facial key point from the image coordinate system to the world coordinate system, and the first facial key point and the second facial key point in the world coordinate system are compared until the error between the first facial key point and the second facial key point in the world coordinate system is minimized, and the target camera parameters are converted into the rotation angle of the face; wherein the target camera parameters include the iterated camera parameters corresponding to the first minimum error; the first minimum error includes the minimum value of the error between the first facial key point and the second facial key point in the world coordinate system.

[0072] The second method to determine the rotation angle of the face can be:

[0073] The camera parameters are iterated using the calibrated camera parameters as initial values. In each iteration, the second facial key point is converted from the world coordinate system to the image coordinate system using the current camera parameters, and the first facial key point and the second facial key point in the image coordinate system are compared until the error between the first facial key point and the second facial key point in the image coordinate system is minimized, and the target camera parameters are converted into the rotation angle of the face; wherein the target camera parameters include the iterated camera parameters corresponding to the second minimum error; the second minimum error includes the minimum value of the error between the first facial key point and the second facial key point in the image coordinate system.

[0074] Step 203: Convert the rotation vector and translation vector output by the camera pose estimation function into the rotation angle of the face.

[0075] In this embodiment, the rotation angle of the second facial key point is set to zero as an example, that is, the rotation angles yaw, pitch and roll of the second facial key point are all 0, and the target camera parameters represent the rotation vector and translation vector of the face relative to the camera.

[0076] Optionally, in the disclosed technical solution, the operation of converting the target camera parameters into the rotation angle of the face included in any embodiment can be implemented by the following methods:

[0077] The rotation vector is converted into a rotation matrix, and the rotation matrix and translation vector are concatenated to obtain the posture matrix. The posture matrix is ​​decomposed into Euler angles to obtain the rotation angle of the face.

[0078] Among them, the implementation method of converting the rotation vector into a rotation matrix includes but is not limited to the Rodriguez function, and the implementation method of performing Euler angle decomposition on the posture matrix includes but is not limited to the projection matrix decomposition algorithm, and the obtained rotation angles of the face include yaw angle, pitch angle and roll angle.

[0079] Step 204 : Using a preset probability scoring function, calculate the rotation angle of the face and the rotation angle thresholds of each orientation to obtain the probability that the face is facing each preset orientation.

[0080] In this embodiment, a probability scoring function is pre-constructed. When calculating the probability that a face is facing a preset orientation, the rotation angle of the face and the rotation angle threshold of the orientation are substituted into the probability scoring function for calculation to obtain the probability that the face is facing the orientation.

[0081] In the technical solution provided in the present disclosure, the rotation angle of the face determined by any of the aforementioned embodiments includes a yaw angle, a pitch angle, and a roll angle, which respectively represent the heading angle, the pitch angle, and the roll angle. The rotation angle threshold corresponding to each orientation includes a yaw threshold, a pitch threshold, and a roll threshold. When calculating the probability that the face is facing each preset orientation, for each preset orientation, the yaw angle and the yaw threshold, the pitch angle and the pitch threshold, and the roll angle and the roll threshold are respectively input into a preset probability scoring function for calculation, and a first sub-probability corresponding to the yaw angle, a second sub-probability corresponding to the pitch angle, and a third sub-probability corresponding to the roll angle are output. Then, based on at least one of the first sub-probability, the second sub-probability, and the third sub-probability, the probability that the face is facing the preset orientation is determined by using a square root average method.

[0082] In this example, the square root mean is calculated as shown in formula (1):

[0083] Where p1 is the first sub-probability, p2 is the second sub-probability, and p3 is the third sub-probability, and p is the probability that the face is facing the specified orientation. The above is a calculation formula for determining the probability that the face is facing the specified orientation based on the first, second, and third sub-probabilities. In actual applications, if the specified orientation has no special requirements for one or two of yaw, pitch, and roll, you can also calculate only the remaining one or two sub-probabilities. When calculating the final probability, the square root of the digit can be changed accordingly. For example, etc., no specific limitation is given here.

[0084] The probability scoring function is explained below.

[0085] In the technical solutions provided by the present disclosure, the rotation angle threshold involved or determined by any embodiment includes a face-to-face value and an offset range; and the probability included in any embodiment decreases as the difference between the rotation angle of the face and the rotation angle threshold increases, and can be set by a probability scoring function according to the following rules:

[0086] When the rotation angle is equal to the positive value, the probability is determined to be the maximum value. When the rotation angle exceeds the boundary value of the offset range, the probability is determined to be zero. When the rotation angle changes from the positive value to the boundary value of the offset range, the probability decreases from the maximum value to zero.

[0087] As an example, taking the yaw threshold as an example, for the yaw threshold of a face facing a screen in the upper right position, the yaw threshold is {facing value: -20, offset range: [-10, 10]}, which means that when the yaw angle of the face is -20, the face is considered to be facing the right direction in the yaw dimension, and when the yaw angle of the face is in the range of [-30, -10], there is also a certain probability that the face is facing the right direction. The probability scoring function is shown in formula (2): p(x) = max(100-(100÷T 2 ×(xZ) 2 ),0) (2)

[0088] Wherein, x represents the rotation angle of the face, Z represents the facing value, and T represents the boundary value of the offset range. When x>Z, T takes the upper boundary, otherwise, T takes the lower boundary. For this formula, when x is equal to Z, the probability score value p(x)=100 is the highest score. When x is on the boundary of the offset range or outside the offset range, p(x)=0. When x is within the offset range, the closer to the facing value Z, the higher the probability score value p(x). It should be noted that the above formula is only an example and can be adjusted as needed. As long as the setting rules of the probability scoring function are met, no specific restrictions are made here.

[0089] Step 205 : determining a face orientation detection result of the face according to the probability that the face is facing each preset orientation.

[0090] In this embodiment, referring to FIG3 , after obtaining the yaw angle, pitch angle, and roll angle of a face, when determining whether the face is facing a preset orientation, the yaw angle and yaw threshold, the pitch angle and pitch threshold, and the roll angle and roll threshold are respectively input into a probability scoring function to obtain a probability score value for the yaw angle, a probability score value for the pitch angle, and a probability score value for the roll angle, thereby comprehensively determining the probability that the face is facing that orientation.

[0091] In the embodiment of the present disclosure, image acquisition based on a monocular camera is used in combination with facial key point detection, and only a small model for facial key point detection is additionally introduced. Compared with the method of deploying a classification detection model, computing resources are saved and computing time is reduced. Moreover, when adding / reducing orientations that need to be paid attention to, only the rotation angle threshold needs to be increased / reduced. It can be quickly deployed and debugged without the need to retrain the model. This solves the problem of high computing resource consumption and easy delay in actual application in the face orientation detection method in the related art, reduces the computational complexity, reduces computing resource consumption, and improves the real-time performance of face orientation detection.

[0092] The disclosed embodiment also provides a face orientation detection device.

[0093] FIG4 is a schematic structural diagram of a face orientation detection device provided by an embodiment of the present disclosure. As shown in FIG4 , the face orientation detection device includes: a detection module 41 , a generation module 42 , a prediction module 43 , and a determination module 44 .

[0094] The detection module 41 is configured to perform key point detection on the face in the image to be tested to obtain the first face key point;

[0095] A generating module 42 is configured to estimate the face pose based on the first face key point and the second face key point to obtain a rotation angle of the face; wherein the second face key point is a preset face key point with a known rotation angle;

[0096] The prediction module 43 is configured to calculate the probability of the face facing each preset orientation based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation; wherein the probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases;

[0097] The determination module 44 is configured to determine a face orientation detection result of the face according to the probability that the face is facing each preset orientation.

[0098] In one embodiment of the present disclosure, the generating module 42 is configured to:

[0099] The camera parameters are iterated using the calibrated camera parameters as the initial values. In each iteration, the first facial key point is converted from the image coordinate system to the world coordinate system using the current camera parameters, and the first facial key point in the world coordinate system is compared with the second facial key point until the error between the first facial key point and the second facial key point in the world coordinate system is minimized.

[0100] The target first camera parameter is converted into a rotation angle of the face; wherein the target camera parameter includes the iterated camera parameter corresponding to the first minimum error; the first minimum error includes the minimum value of the error between the first face key point and the second face key point in the world coordinate system.

[0101] In one embodiment of the present disclosure, the generating module 42 is configured to:

[0102] Iterate the camera parameters using the calibrated camera parameters as the initial values. In each iteration, use the current camera parameters to transform the second facial key point from the world coordinate system to the image coordinate system, and compare the first facial key point and the second facial key point in the image coordinate system until the error between the first facial key point and the second facial key point in the image coordinate system is minimized.

[0103] Convert the target camera parameters into a rotation angle of the face; wherein the target camera parameters include the iterated camera parameters corresponding to the second minimum error; the second minimum error includes the minimum value of the error between the first face key point and the second face key point in the image coordinate system.

[0104] In one embodiment of the present disclosure, the camera parameters corresponding to the minimum error represent the rotation vector and translation vector of the face relative to the camera, and the generation module 42 is configured to:

[0105] Convert the rotation vector into a rotation matrix, and concatenate the rotation matrix and the translation vector to obtain the attitude matrix;

[0106] The projection matrix decomposition algorithm is used to decompose the posture matrix into Euler angles to obtain the rotation angle of the face.

[0107] In one embodiment of the present disclosure, the rotation angle of a face includes a yaw angle, a pitch angle, and a roll angle. The rotation angle threshold corresponding to each orientation includes a yaw threshold, a pitch threshold, and a roll threshold. The prediction module 43 is configured to:

[0108] For each preset orientation, the yaw angle and yaw threshold, the pitch angle and pitch threshold, and the roll angle and roll threshold are input into the preset probability scoring function for calculation, and the first sub-probability corresponding to the yaw angle, the second sub-probability corresponding to the pitch angle, and the third sub-probability corresponding to the roll angle are output;

[0109] According to at least one of the first sub-probability, the second sub-probability, and the third sub-probability, a probability that the human face is facing the preset orientation is determined by adopting a square root average method.

[0110] In one embodiment of the present disclosure, the rotation angle threshold includes a face value and an offset range, and the probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases, including:

[0111] When the face rotation angle is equal to the face-to-face value, the probability is determined to be the maximum;

[0112] When the rotation angle of the face exceeds the boundary value of the offset range, the probability is determined to be zero;

[0113] When the rotation angle of the face changes from the positive value to the boundary value of the offset range, the probability decreases from the maximum value to zero.

[0114] In one embodiment of the present disclosure, the determination module 44 is configured to:

[0115] If the probability that the face is facing each preset direction is less than or equal to the threshold, it is determined that the face is not facing any preset direction;

[0116] If there is a target probability greater than a threshold among the probabilities of the face facing each preset direction, the preset direction corresponding to the maximum value among the target probabilities is taken as the direction of the face.

[0117] The face orientation detection device provided in the embodiments of the present disclosure can execute any face orientation detection method provided in the embodiments of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. For any content not fully described in the embodiments of the present disclosure, please refer to the description of any method embodiment of the present disclosure.

[0118] An embodiment of the present disclosure further provides an electronic device, which includes one or more processors and a memory.

[0119] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0120] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the methods of the embodiments of the present disclosure above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0121] In one example, the electronic device may further include an input device and an output device, these components being interconnected via a bus system and / or other forms of connection mechanisms. Furthermore, the input device may include, for example, a keyboard, a mouse, etc. The output device may output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0122] Of course, for simplicity, only some of the components in the electronic device related to the present disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0123] The embodiments of the present disclosure also provide a vehicle, which may include the face orientation detection device as described above, or the electronic device as described above.

[0124] In addition to the above methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor of an electronic device, the processor executes any method provided by the embodiments of the present disclosure.

[0125] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0126] In addition, the embodiments of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor of an electronic device, the processor is enabled to perform any method provided by the embodiments of the present disclosure.

[0127] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0128] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0129] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein. Industrial Applicability

[0130] The present disclosure relates to a face orientation detection method, apparatus, equipment, vehicle, medium and program product, wherein the method comprises: performing key point detection on a face in a test image to obtain a first face key point; performing posture estimation on the face based on the first face key point and a second face key point to obtain a rotation angle of the face; wherein the second face key point is a preset face key point with a known rotation angle; performing calculation based on the rotation angle of the face and a rotation angle threshold corresponding to each preset orientation to obtain a probability that the face is facing each preset orientation; the probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases; and determining a face orientation detection result of the face based on the probability that the face is facing each preset orientation.

Claims

1. A method for detecting face orientation, the method comprising: Perform key point detection on the face in the image to be tested to obtain the first face key point; Estimating the face pose based on the first and second facial key points to obtain a rotation angle of the face; wherein the second facial key point is a pre-set facial key point with a known rotation angle; Calculating, based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation, a probability that the face is facing each preset orientation; wherein the probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases; A face orientation detection result of the face is determined based on the probability that the face is facing each preset orientation.

2. The method according to claim 1, wherein The step of estimating the pose of the face based on the first facial key points and the second facial key points to obtain a rotation angle of the face includes: Iterating the camera parameters using the calibrated camera parameters as initial values; in each iteration, transforming the first facial key point from the image coordinate system to the world coordinate system using the current camera parameters, and comparing the first facial key point with the second facial key point in the world coordinate system until the error between the first facial key point and the second facial key point in the world coordinate system is minimized; Convert target camera parameters into a rotation angle of the face; wherein the target camera parameters include iterated camera parameters corresponding to a first minimum error; the first minimum error includes a minimum value of an error between a first facial key point and a second facial key point in the world coordinate system.

3. The method according to claim 1 or 2, wherein The step of estimating the pose of the face based on the first facial key points and the second facial key points to obtain a rotation angle of the face includes: Iterating the camera parameters using the calibrated camera parameters as initial values; in each iteration, transforming the second facial key point from a world coordinate system to an image coordinate system using the current camera parameters, and comparing the first facial key point and the second facial key point in the image coordinate system until the error between the first facial key point and the second facial key point in the image coordinate system is minimized; Convert target camera parameters into a rotation angle of the face; wherein the target camera parameters include iterated camera parameters corresponding to a second minimum error; and the second minimum error includes a minimum value of an error between a first facial key point and a second facial key point in the image coordinate system.

4. The method according to claim 2 or 3, wherein: The target camera parameters represent a rotation vector and a translation vector of the face relative to the camera, and converting the target camera parameters into a rotation angle of the face includes: Convert the rotation vector into a rotation matrix, and concatenate the rotation matrix and the translation vector to obtain a posture matrix; The projection matrix decomposition algorithm is used to perform Euler angle decomposition on the posture matrix to obtain the rotation angle of the face.

5. The method according to any one of claims 1 to 4, wherein: The rotation angle of the face includes a yaw angle, a pitch angle, and a roll angle, and the rotation angle threshold corresponding to each orientation includes a yaw threshold, a pitch threshold, and a roll threshold. The calculation based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation to obtain the probability that the face is facing each preset orientation includes: For each preset orientation, the yaw angle and the yaw threshold, the pitch angle and the pitch threshold, and the roll angle and the roll threshold are respectively input into a preset probability scoring function for calculation, and a first sub-probability corresponding to the yaw angle, a second sub-probability corresponding to the pitch angle, and a third sub-probability corresponding to the roll angle are output; The probability that the face is facing the preset orientation is determined by adopting a square root average method according to at least one of the first sub-probability, the second sub-probability, and the third sub-probability.

6. The method according to any one of claims 1 to 5, wherein: The rotation angle threshold includes a face value and an offset range, and the probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases, including: When the rotation angle of the face is equal to the facing value, determining that the probability is maximum; When the rotation angle of the face exceeds the boundary value of the offset range, determining the probability to be zero; When the rotation angle of the face changes from the facing value to the boundary value of the offset range, the probability decreases from the maximum value to zero.

7. The method according to any one of claims 1 to 6, wherein: The determining of the face orientation detection result of the face according to the probability that the face is facing each preset orientation includes: If the probability that the face is facing each preset direction is less than or equal to a threshold, it is determined that the face is not facing any preset direction; If there is a target probability greater than the threshold among the probabilities that the face is facing each preset direction, the preset direction corresponding to the maximum value among the target probabilities is used as the direction of the face.

8. A face orientation detection device, comprising: A detection module is configured to perform key point detection on a face in a test image to obtain a first face key point; a generating module configured to estimate the pose of the face based on the first facial key points and the second facial key points to obtain a rotation angle of the face; wherein the second facial key points are pre-set facial key points with a known rotation angle; a prediction module configured to calculate, based on the rotation angle of the face and the rotation angle threshold corresponding to each preset orientation, a probability that the face is facing each preset orientation; wherein the probability decreases as the difference between the rotation angle of the face and the rotation angle threshold increases; The determination module is configured to determine a face orientation detection result of the face according to the probability that the face is facing each preset orientation.

9. The face orientation detection device according to claim 8, wherein: The generation module is configured to iterate the camera parameters using the calibrated camera parameters as initial values. During each iteration, the first facial key point is converted from the image coordinate system to the world coordinate system using the current camera parameters, and the first facial key point and the second facial key point in the world coordinate system are compared until the error between the first facial key point and the second facial key point in the world coordinate system is minimized; the target camera parameters are converted into the rotation angle of the face; wherein the target camera parameters include the iterated camera parameters corresponding to the first minimum error; and the first minimum error includes the minimum value of the error between the first facial key point and the second facial key point in the world coordinate system.

10. The face direction detection device according to claim 8 or 9, wherein: The generation module is configured to iterate the camera parameters using the calibrated camera parameters as initial values. During each iteration, the second facial key point is converted from the world coordinate system to the image coordinate system using the current camera parameters, and the first facial key point and the second facial key point in the image coordinate system are compared until the error between the first facial key point and the second facial key point in the image coordinate system is minimized; the target camera parameters are converted into the rotation angle of the face; wherein the target camera parameters include the iterated camera parameters corresponding to the second minimum error; and the second minimum error includes the minimum value of the error between the first facial key point and the second facial key point in the image coordinate system.

11. The face direction detection device according to claim 9 or 10, wherein: The target camera parameters represent a rotation vector and a translation vector of the face relative to the camera; the generation module is configured to convert the rotation vector into a rotation matrix, and concatenate the rotation matrix and the translation vector to obtain a posture matrix; The projection matrix decomposition algorithm is used to perform Euler angle decomposition on the posture matrix to obtain the rotation angle of the face.

12. The face orientation detection device according to any one of claims 8 to 11, wherein: The rotation angles of the face include a yaw angle, a pitch angle, and a roll angle. The rotation angle thresholds corresponding to each orientation include a yaw threshold, a pitch threshold, and a roll threshold. The prediction module is configured to, for each preset orientation, input the yaw angle and the yaw threshold, the pitch angle and the pitch threshold, and the roll angle and the roll threshold into a preset probability scoring function for calculation, and output a first sub-probability corresponding to the yaw angle, a second sub-probability corresponding to the pitch angle, and a third sub-probability corresponding to the roll angle; and determine the probability that the face is facing the preset orientation by using a square root average method based on at least one of the first sub-probability, the second sub-probability, and the third sub-probability.

13. The face orientation detection device according to any one of claims 8 to 12, wherein: The rotation angle threshold includes a facing value and an offset range. When the rotation angle of the face is equal to the facing value, the probability is determined to be maximum; when the rotation angle of the face exceeds the boundary value of the offset range, the probability is determined to be zero. When the rotation angle of the face changes from the facing value to the boundary value of the offset range, the probability decreases from the maximum value to zero.

14. The face orientation detection device according to any one of claims 8 to 13, wherein: The determining module is configured to determine that the human face is not facing any preset direction if the probability that the human face is facing each preset direction is less than or equal to a threshold; If there is a target probability greater than the threshold among the probabilities that the face is facing each preset direction, the preset direction corresponding to the maximum value among the target probabilities is used as the direction of the face.

15. An electronic device comprising: processor; a memory configured to store instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the face orientation detection method according to any one of claims 1 to 7.

16. A vehicle comprising: The face orientation detection device according to any one of claims 8 to 14, or the electronic device according to claim 15.

17. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of an electronic device, implements the face orientation detection method according to any one of claims 1 to 7.

18. A computer program product, comprising a computer program; when the computer program is executed by a processor of an electronic device, the computer program can implement the face orientation detection method according to any one of claims 1 to 7.

Images