Vehicle face orientation recognition method and device, electronic equipment, storage medium and vehicle

Abstract

The invention discloses a vehicle face orientation recognition method and device, electronic equipment, a storage medium and a vehicle. The method comprises the following steps: acquiring a face image shot by a camera, identifying a head image from the face image, performing head orientation classification operation on the head image, judging the head orientation category of the head image through the head orientation classification operation, and determining the head orientation of the head image according to the head orientation category of the head image. The head orientation category comprises a forward orientation category and a non-forward orientation category; and when the head orientation category is a non-forward orientation category, taking the recognized head orientation category as a face orientation. According to the method, the head orientation is identified through image classification, and the head orientation is identified as a non-forward orientation category when the rotation angle of the head is very large, so that the head orientation category is taken as the face orientation, and therefore, the face orientation can be identified through the camera even if the rotation angle of the head is very large.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicles, and particularly to a vehicle face orientation recognition method, device, electronic device, storage medium, and vehicle. Background Art

[0002] In the distraction detection function of the vehicle monitoring system, such as the distraction detection function in the Occupancy Monitoring System (OMS), it is necessary to obtain the face orientation angle to determine whether the driver's head deviates from the front, and then determine whether there is distraction.

[0003] The existing face orientation recognition method captures a face image through a camera, and then estimates the approximate orientation angle by judging the positional relationship between face key points (eyes, nose, mouth, etc.), and then determines the face orientation based on the orientation angle.

[0004] However, when the rotation angle of the human head is very large, it is easy to cause the face to be blocked. Since the face key points are predicted based on the entire face, when the face is completely blocked or partially blocked, the prediction of the face key points is inaccurate, resulting in an inaccurate orientation angle predicted based on the face key points, and thus the face orientation cannot be determined. Summary of the Invention

[0005] Based on this, in view of the technical problem that the existing technology cannot determine the face orientation when the rotation angle of the human head is very large, it is necessary to provide a vehicle face orientation recognition method, device, electronic device, storage medium, and vehicle.

[0006] The present invention provides a vehicle face orientation recognition method, including:

[0007] Obtain a face image captured by a camera, identify a human head image from the face image, perform a human head orientation classification operation on the human head image, and determine the human head orientation category of the human head image through the human head orientation classification operation. The human head orientation category includes a forward orientation category and a non - forward orientation category;

[0008] When the human head orientation category is the non - forward orientation category, use the identified human head orientation category as the face orientation.

[0009] Further, the identifying a human head image from the face image includes:

[0010] Identify a human head detection frame and a face detection frame from the face image, and use the image within the human head detection frame as the human head image. The range of the human head detection frame is larger than the range of the face detection frame.

[0011] Further, the operation of classifying the head orientation of the head image and determining the head orientation category of the head image through the head orientation classification operation includes:

[0012] Input the head image into a head orientation image classification model to obtain the head orientation category of the head image output by the head orientation image classification model. The head orientation image classification model is pre-trained with multiple images and their corresponding head orientation categories.

[0013] Further, the method further includes:

[0014] When the head orientation category is the forward orientation category, identify the image two-dimensional coordinates of the face key points in the image coordinate system from the face image;

[0015] Determine the face orientation angle according to the image two-dimensional coordinates of the face key points in the face image in the image coordinate system. The image coordinate system is a coordinate system established on the face image.

[0016] Furthermore, the identifying the image two-dimensional coordinates of the face key points in the image coordinate system from the face image includes: identifying the image two-dimensional coordinates of the face key points in the image coordinate system within the face detection frame identified from the face image.

[0017] Furthermore, the determining the face orientation angle according to the image two-dimensional coordinates of the face key points in the face image in the image coordinate system includes:

[0018] Based on the camera parameters, calculate the conversion relationship for converting the standard three-dimensional coordinates of the face key points in the world coordinate system of three-dimensional space into the image two-dimensional coordinates, and calculate the face orientation angle according to the conversion relationship.

[0019] Still further, the calculating the conversion relationship for converting the standard three-dimensional coordinates of the face key points in the world coordinate system of three-dimensional space into the image two-dimensional coordinates based on the camera parameters and calculating the face orientation angle according to the conversion relationship includes:

[0020] Substitute the standard three-dimensional coordinates of the face key points in the world coordinate system of three-dimensional space and the image two-dimensional coordinates into the formula: s*p = F*[R|t]P, where s is the scale coefficient, F is the camera intrinsic matrix, p is the image two-dimensional coordinate, P is the standard three-dimensional coordinate, R is the rotation matrix of the camera extrinsic parameters, and t is the translation matrix of the camera extrinsic parameters;

[0021] Calculate the rotation matrix and the translation matrix;

[0022] Calculate the face orientation angle according to the rotation matrix.

[0023] Further, calculating the face orientation angle according to the rotation matrix includes:

[0024] Calculating the rotation angle of the face orientation relative to the camera coordinate system according to the rotation matrix;

[0025] Calculating the face orientation angle as the difference between the rotation angle and the installation angle of the camera.

[0026] The present invention provides a vehicle face orientation recognition device, including:

[0027] A head orientation classification module, configured to obtain a face image captured by a camera, identify a head image from the face image, perform a head orientation classification operation on the head image, and determine the head orientation category of the head image through the head orientation classification operation, where the head orientation category includes a forward orientation category and a non-forward orientation category;

[0028] A face orientation determination module, configured to use the identified head orientation category as the face orientation when the head orientation category is the non-forward orientation category.

[0029] The present invention provides an electronic device, including:

[0030] At least one processor; and,

[0031] A memory communicatively connected to at least one of the processors; wherein,

[0032] The memory stores instructions executable by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the vehicle face orientation recognition method as described above.

[0033] The present invention provides a storage medium that stores computer instructions, and when a computer executes the computer instructions, it is used to execute all steps of the vehicle face orientation recognition method as described above.

[0034] The present invention provides a vehicle, including the vehicle face orientation recognition device as described above, or the electronic device as described above.

[0035] The present invention captures a face image through a camera, classifies the head orientation of the face image, and when the head orientation category is a non-forward orientation category, uses the recognized head orientation category as the face orientation. The present invention identifies the head orientation through image classification. Since when the head rotation angle is very large, it will be recognized as a non-forward orientation category, and at this time, the head orientation category is used as the face orientation. Therefore, even when the head rotation angle is very large, the face orientation can be recognized through the camera, thereby providing algorithm support for the distraction detection function of the DMS to determine whether the driver is facing forward and whether they are distracted. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flowchart of the working process of a vehicle face orientation recognition method according to an embodiment of the present invention;

[0037] Figure 2 is a flowchart of the working process of a vehicle face orientation recognition method according to another embodiment of the present invention;

[0038] Figure 3 is a flowchart of the working process of head orientation image classification and recognition according to the best embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of the relationship between a world coordinate system, an image, and a camera coordinate system according to an embodiment of the present invention;

[0040] Figure 5 is a schematic diagram of image recognition according to an example of the present invention;

[0041] Figure 6 is a schematic diagram of face orientation angles;

[0042] Figure 7 is a flowchart of the working process of a vehicle face orientation recognition method according to the best embodiment of the present invention;

[0043] Figure 8 is a schematic diagram of a vehicle face orientation recognition device according to an embodiment of the present invention;

[0044] Figure 9 is a schematic diagram of the hardware structure of an electronic device according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The following further describes the specific embodiments of the present invention with reference to the accompanying drawings. The same components are denoted by the same reference numerals. It should be noted that the terms "front", "rear", "left", "right", "up", and "down" used in the following description refer to the directions in the drawings, and the terms "inner" and "outer" respectively refer to the directions towards or away from the geometric center of a specific component.

[0046] Such as Figure 1The following is a flowchart of the operation of a vehicle face orientation recognition method according to an embodiment of the present invention, including:

[0047] Step S101, obtain a face image captured by a camera, identify a head image from the face image, perform a head orientation classification operation on the head image, and determine the head orientation category of the head image through the head orientation classification operation. The head orientation category includes a forward orientation category and a non-forward orientation category, and the non-forward orientation category includes multiple head orientation categories;

[0048] Step S102, when the head orientation category is a non-forward orientation category, use the identified head orientation category as the face orientation.

[0049] Specifically, the present invention can be applied to an electronic device with processing capabilities, such as an electronic control unit (ECU) of a vehicle or an extended domain control unit (XCU).

[0050] The electronic device first executes step S101 to obtain a face image. Preferably, the face image is captured by a monocular camera. Then, perform a head orientation classification operation on the head image, and determine the head orientation category of the head image through the head orientation classification operation.

[0051] When the face is not at the center position of the camera and the head rotation angle is large, it is easy for the face to be blocked, resulting in inaccurate prediction of key points and inaccurate prediction of the orientation angle. Therefore, step S101 identifies the head image from the input face image, performs a head orientation classification on the head image, predicts whether the head is turning at a large angle, and where it is facing (up, down, left, right).

[0052] Among them, the head orientation category of the face image includes a forward orientation category identified when the head turns at a small angle, and a non-forward orientation category identified when the head turns at a large angle.

[0053] The definition of the large angle category is usually related to the actual scenario and project, and is also related to the face key point failure condition. In some embodiments, the definition is as follows:

[0054] left, turn the head to the left until more than half of the face is not visible, which is considered a large angle left turn;

[0055] right, turn the head to the right until more than half of the face is not visible, which is considered a large angle right turn;

[0056] up, raise the head until the eyes are not visible, which is considered a large angle head up;

[0057] down, lower the head until the mouth is not visible, which is considered a large-angle head-down.

[0058] In some embodiments, the non-front-facing orientation categories include large-angle left turn, large-angle right turn, large-angle head-up, and large-angle head-down.

[0059] After performing the head orientation classification, step S102 is executed. When the classification result is a non-front-facing orientation category, it means that the face is not at the center position of the camera and the head rotation angle is relatively large. Therefore, directly use the identified head orientation category as the face orientation.

[0060] In some embodiments, the head orientation categories include large-angle left turn, large-angle right turn, large-angle head-up, and large-angle head-down. When the classification result is a non-front-facing orientation category:

[0061] If the head orientation category indicated by the non-front-facing orientation category is a large-angle left turn, then the face orientation is a large-angle left turn;

[0062] If the head orientation category indicated by the non-front-facing orientation category is a large-angle right turn, then the face orientation is a large-angle right turn;

[0063] If the head orientation category indicated by the non-front-facing orientation category is a large-angle head-up, then the face orientation is a large-angle head-up;

[0064] If the head orientation category indicated by the non-front-facing orientation category is a large-angle head-down, then the face orientation is a large-angle head-down.

[0065] The present invention captures a face image through a camera, performs a head orientation classification operation on the face image, and when the head orientation category is a non-front-facing orientation category, uses the identified head orientation category as the face orientation. The present invention identifies the head orientation through image classification. Since when the head rotation angle is very large, it will be recognized as a non-front-facing orientation category, and at this time, the head orientation category is used as the face orientation. Therefore, even when the head rotation angle is very large, the face orientation can be recognized through the camera, thereby providing algorithm support for the distraction detection function of DMS to determine whether the driver is facing forward and whether distracted.

[0066] As Figure 2 shown is a flowchart of a method for recognizing a vehicle face orientation in another embodiment of the present invention, including:

[0067] Step S201, obtain a face image captured by a camera.

[0068] Step S202, identify a head detection frame and a face detection frame from the face image, and use the image within the head detection frame as the head image. The range of the head detection frame is larger than the range of the face detection frame.

[0069] Step S203: Input the head image into the head orientation image classification model to obtain the head orientation category of the head image output by the head orientation image classification model. The head orientation classification identifies the head orientation category of the face image. The head orientation category includes a forward orientation category and a non-forward orientation category. The non-forward orientation category includes multiple head orientation categories. The head orientation image classification model is pre-trained with multiple images and their corresponding head orientation categories.

[0070] Step S204: When the head orientation category is a non-forward orientation category, use the identified head orientation category as the face orientation.

[0071] Step S205: When the head orientation category is a forward orientation category, identify the image two-dimensional coordinates of the face key points in the image coordinate system from the face image.

[0072] In one embodiment, the identifying the image two-dimensional coordinates of the face key points in the image coordinate system from the face image includes: identifying the image two-dimensional coordinates of the face key points in the image coordinate system within the face detection frame identified from the face image.

[0073] Step S206: Determine the face orientation angle according to the image two-dimensional coordinates of the face key points in the face image. The image coordinate system is a coordinate system established on the face image.

[0074] In one embodiment, the determining the face orientation angle according to the image two-dimensional coordinates of the face key points in the face image includes:

[0075] Based on the camera parameters, calculate the conversion relationship for converting the standard three-dimensional coordinates of the face key points in the world coordinate system of three-dimensional space into the image two-dimensional coordinates. According to the conversion relationship, calculate the face orientation angle.

[0076] In one embodiment, the based on the camera parameters, calculating the conversion relationship for converting the standard three-dimensional coordinates of the face key points in the world coordinate system of three-dimensional space into the image two-dimensional coordinates, and calculating the face orientation angle according to the conversion relationship includes:

[0077] Substitute the standard three-dimensional coordinates of the face key points in the world coordinate system of three-dimensional space and the image two-dimensional coordinates into the formula: s*p = F*[R|t]P, where s is the scale factor, F is the camera intrinsic matrix, p is the image two-dimensional coordinate, P is the standard three-dimensional coordinate, R is the rotation matrix of the camera extrinsic parameters, and t is the translation matrix of the camera extrinsic parameters;

[0078] Calculate the rotation matrix and the translation matrix;

[0079] Calculate the face orientation angle according to the rotation matrix.

[0080] In one embodiment, the calculating the face orientation angle according to the rotation matrix includes:

[0081] Calculate the rotation angle of the face orientation relative to the camera coordinate system according to the rotation matrix;

[0082] Calculate the face orientation angle as the difference between the rotation angle and the installation angle of the camera.

[0083] Specifically, first perform step S201 to obtain a face image. Preferably, obtain a face image captured by a monocular camera.

[0084] Then, perform step S202 to identify a head detection frame and a face detection frame from the face image, and use the image within the head detection frame as the head image. The range of the head detection frame is larger than the range of the face detection frame.

[0085] Specifically, existing image detection algorithms, such as a CNN model, can be used for head / face detection to detect a head detection frame 51 (red frame) and a face detection frame 52 (green frame) as shown in Figure 5 Then, use the image within the head detection frame as the head image.

[0086] By identifying the head detection frame and the face detection frame, the turning of the head can be identified using the larger - range head detection frame, resulting in a more accurate classification and recognition.

[0087] Then, perform step S203 to classify the head orientation of the head image. Input the head image into a head orientation image classification model to obtain the head orientation category of the head image output by the head orientation image classification model.

[0088] Among them, the head orientation image classification model is pre - trained with multiple images and their corresponding pre - calibrated head orientation categories.

[0089] In some embodiments, the head orientation image classification model includes a CNN layer and a classification layer.

[0090] As Figure 3 shown is a schematic diagram of the head orientation image classification and recognition of the best embodiment of the present invention, including:

[0091] Input the face image 301 into the Convolutional Neural Networks (CNN) layer 302. Extract the image features 303 through the CNN layer 302, and then enter the classification layer 304. The classification layer 304 outputs the scores 305 for each classification category. Select the classification category with the highest score as the head orientation classification, and the classification categories include the forward orientation category and one or more non-forward orientation categories.

[0092] Among them, the CNN network is applicable to all classification models. The CNN layer can be implemented using ResNet-18. The classification layer uses the clshead layer, usually a two-layer fully connected layer. The classification categories include the forward orientation category, the large-angle left turn category, the large-angle right turn category, the large-angle head-up category, and the large-angle head-down category. Among them, the frontal face angle category is the normal category, and the large-angle left turn category (left), the large-angle right turn category (right), the large-angle head-up category (up), and the large-angle head-down category (down) are the large-angle orientation categories, that is, the non-forward orientation categories. The model finally outputs the scores for 5 categories. The final category is i = argmax(scores).

[0093] Specific categories can be:

[0094] 0: normal;

[0095] 1: left;

[0096] 2: right;

[0097] 3: up

[0098] 4: down

[0099] When the head orientation category is a non-forward orientation category, execute step S204, and use the identified head orientation category as the face orientation. When the head orientation category is a forward orientation category, the face is at the center position of the camera and the head does not turn at a large angle. Therefore, execute steps S205 to S206 to determine the specific face orientation angle.

[0100] First, execute step S205 to identify the image two-dimensional coordinates of the face key points in the image coordinate system from the face image.

[0101] Specifically, first identify the face key points from the face image, and then determine the image two-dimensional coordinates of the face key points in the image coordinate system. The image coordinate system is a coordinate system established on the face image. Specifically, the image two-dimensional coordinates of the face key points can be determined according to the pixels of the face key points in the face image.

[0102] There are multiple facial key points, and for each facial key point, its two-dimensional image coordinates are determined.

[0103] Existing general key point detection algorithms can be used to identify facial key points and determine the two-dimensional image coordinates of the facial key points in the image coordinate system.

[0104] In some embodiments, a CNN model is used to detect facial key points, and the pixel coordinates of 68 facial key points of a face are predicted. The pixel coordinates are the two-dimensional image coordinates of the facial key points in the image coordinate system.

[0105] In one embodiment, the identifying the two-dimensional image coordinates of the facial key points in the image coordinate system from the facial image includes: identifying the two-dimensional image coordinates of the facial key points in the image coordinate system within the face detection box identified from the facial image.

[0106] As Figure 5 shown, the facial key points 53 ( Figure 5 the red dots in

[0107] are identified within the face detection box 52. In this embodiment, a smaller face detection box is used to accurately determine the facial key points in order to obtain a more accurate three-dimensional facial pose.

[0108] After obtaining the two-dimensional image coordinates of the facial key points in the camera coordinate system, step S206 is executed. According to the two-dimensional image coordinates of the facial key points in the image coordinate system in the facial image, the facial orientation angle is determined, and the image coordinate system is the coordinate system established on the facial image.

[0109] In some embodiments, after obtaining the facial orientation angle, it further includes: using the orientation indicated by the facial orientation angle as the facial orientation.

[0110] As Figure 7 shown is the flowchart of a method for recognizing the facial orientation of a vehicle according to the best embodiment of the present invention, including:

[0111] Step S701, obtaining a picture;

[0112] Step S702, performing head / face detection;

[0113] Step S703, classifying the large angles of the head for the image within the head detection box. If it is a large angle, the orientation category is output; otherwise, step S704 is executed;

[0114] Step S704, identifying facial key points for the image within the face detection box;

[0115] Step S705, performing facial orientation estimation to obtain the orientation angle.

[0116] This embodiment is based on the face orientation estimation of a monocular camera, and further provides algorithm support for the distraction detection function of the DMS to determine whether the driver is facing forward and whether they are distracted. In this embodiment, image classification is used to identify the face orientation. When the rotation angle of the human head is very large and it is identified as a non-forward orientation category, the face orientation is determined according to the image classification, so that even when the rotation angle of the human head is very large, the face orientation can be identified by the monocular camera. When the rotation angle of the human head is not large and it is identified as a forward orientation category, the image two-dimensional coordinates of the face key points in the face image are used to judge the face orientation angle, thereby improving the accuracy of the face orientation angle.

[0117] In one embodiment, determining the face orientation angle according to the image two-dimensional coordinates of the face key points in the image coordinate system includes:

[0118] Based on the camera parameters, calculate the conversion relationship for converting the standard three-dimensional coordinates of the face key points in the world coordinate system of the three-dimensional space into the image two-dimensional coordinates, and calculate the face orientation angle according to the conversion relationship.

[0119] In one embodiment, the calculating the conversion relationship for converting the standard three-dimensional coordinates of the face key points in the world coordinate system of the three-dimensional space into the image two-dimensional coordinates based on the camera parameters, and calculating the face orientation angle according to the conversion relationship includes:

[0120] Substitute the standard three-dimensional coordinates of the face key points in the world coordinate system of the three-dimensional space and the image two-dimensional coordinates into the formula: s*p = F*[R|t]P, where s is the scale coefficient, F is the camera internal parameter matrix, p is the image two-dimensional coordinate, P is the standard three-dimensional coordinate, R is the rotation matrix of the camera external parameters, and t is the translation matrix of the camera external parameters;

[0121] Calculate the rotation matrix and the translation matrix;

[0122] Calculate the face orientation angle according to the rotation matrix.

[0123] Such as Figure 4As shown, the principle of pinhole imaging and 3D perspective transformation are used to estimate the orientation angle of a human face. A 3D coordinate point P in the world coordinate system 41 is projected onto the image coordinate system, which is the 2D pixel coordinate system on the image 42. The coordinate point P is converted into the coordinate point p through the camera intrinsic matrix F, the rotation matrix R, and the translation matrix t of the camera extrinsic parameters. Among them, the camera parameters of the camera 44 include the camera intrinsics and the camera extrinsics. The camera intrinsics is the camera intrinsic matrix F, and the camera intrinsic matrix F describes the conversion relationship between the image coordinate system and the camera coordinate system 43. The camera extrinsics includes the rotation matrix R and the translation matrix t. The rotation matrix R and the translation matrix t represent the conversion relationship between the world coordinate system 41 and the camera coordinate system 43. Therefore, through the camera intrinsic matrix, the rotation matrix, and the translation matrix, the conversion formula between the three-dimensional coordinates in the world coordinate system 41 and the two-dimensional coordinates in the image coordinate system is established. Since the standard three-dimensional coordinates of the facial key points in the world coordinate system of the three-dimensional space and the image two-dimensional coordinates of the facial key points in the image coordinate system are known, and the camera intrinsic matrix can be obtained through camera calibration, the rotation matrix R and the translation matrix t representing the conversion relationship between the world coordinate system 41 and the camera coordinate system 43 can be solved.

[0124] Substitute the standard three-dimensional coordinates of the facial key points in the world coordinate system of the three-dimensional space and the image two-dimensional coordinates into the formula: s*p = F*[R|t]P, where s is the scale factor, F is the camera intrinsic matrix, p is the image two-dimensional coordinate, P is the standard three-dimensional coordinate, R is the rotation matrix of the camera extrinsic parameters, and t is the translation matrix of the camera extrinsic parameters;

[0125] Calculate the rotation matrix and the translation matrix.

[0126] Specifically, calculate the rotation matrix and the translation matrix according to formula (1):

[0127] s*p = F*[R|t]P (1)

[0128] Among them, s is the scale factor, F is the camera intrinsic matrix, p is the image two-dimensional coordinate, P is the standard three-dimensional coordinate, R is the rotation matrix of the camera extrinsic parameters, and t is the translation matrix of the camera extrinsic parameters.

[0129] The specific formula expansion is as follows:

[0130]

[0131] Among them, (u, v) is the image two-dimensional coordinate p, is the camera intrinsic matrix F, f x represents the pixel focal length of the camera in the x direction, f yRepresents the pixel focal length of the camera in the y direction, c x Represents the pixel offset of the camera optical axis in the x direction in the image coordinate system, c y Represents the pixel offset of the camera optical axis in the y direction in the image coordinate system Is the rotation matrix R Is the translation matrix t, and (X, Y, Z) are the three-dimensional coordinates P

[0132] Such as Figure 6 Shown is a schematic diagram of the face orientation angle. Such as Figure 6 Shown, a 3D coordinate system is established with the center of the human head 60 as the origin. The face orientation angle is the angle of rotation around the three coordinate axes, and is divided into the first angle component (pitch) 61 of nodding up and down, the second angle component (yaw) 62 of shaking the head left and right, and the third angle component (roll) 63 of tilting the head

[0133] Such as Figure 6 In the example shown, the first angle component 61 is the rotation angle around the y axis, the second angle component 62 is the rotation angle around the z axis, and the third angle component 63 is the rotation angle around the x axis

[0134] When the world coordinate system is set to the coordinate system in the above figure, then the rotation matrix R from the world coordinate system to the camera coordinate system is the face rotation matrix that needs to be obtained. The three angle components of pitch, yaw, and roll can be decomposed from R. Based on the three angle components of pitch, yaw, and roll, the face orientation angle can be calculated

[0135] Such as Figure 5 Shown, the world coordinate system 41 is at the center position of the human head. Therefore, the standard three-dimensional coordinates of the face key points can be directly used (ignoring the differences in the face shapes of different people). The internal parameter matrix of the camera can be obtained through camera calibration. Therefore, according to the standard three-dimensional coordinates of several face key points and the corresponding image two-dimensional coordinates, the rotation matrix R and the translation matrix t can be solved, representing the conversion relationship between the camera coordinate system and the world coordinate system. The solution method is not unique, and a feasible method is the pnp algorithm, and the solvePnP function in the OpenCV library can be used to solve it

[0136] After solving the rotation matrix R, it is necessary to decompose each rotation angle. Actually, it is necessary to solve three Euler angles. The solution method can be obtained using the arctan2 function in the NumPy package of Python. The arctan2 function inputs the specific values of the opposite side and the adjacent side to obtain the arctangent value. Specifically, it is as follows

[0137] θ z = arctan 2(-r 31 , r 11 )

[0138]

[0139] θ x = arctan2(-r32, r33)

[0140] Wherein, θ z is the yaw angle component, θ y is the pitch angle component, θ x is the roll angle component.

[0141] In this embodiment, the rotation matrix and the translation matrix are solved by using the standard three-dimensional coordinates of the face key points in the world coordinate system of the three-dimensional space and the corresponding two-dimensional image coordinates, and the face orientation angle is obtained based on the rotation matrix.

[0142] In one of the embodiments, calculating the face orientation angle according to the rotation matrix includes:

[0143] Calculating the rotation angle of the face orientation relative to the camera coordinate system according to the rotation matrix;

[0144] Calculating the face orientation angle as the difference between the rotation angle and the installation angle of the camera.

[0145] Specifically, the three angles (pitch, yaw, roll) calculated and solved based on the rotation matrix are the rotation angles relative to the camera coordinate system, and the face orientation angle is the orientation angle of the face relative to the vehicle, which needs to be converted.

[0146] Assuming that the three installation angles of the camera relative to the vehicle are t0, t1, and t2, then the final orientation angle is (pitch - t0, yaw - t1, roll - t2).

[0147] In this embodiment, the rotation angle calculated based on the rotation matrix is converted by the vehicle installation angle to obtain an accurate face orientation angle.

[0148] Based on the same inventive concept, as Figure 8 shown in the schematic diagram of a vehicle face orientation recognition device according to an embodiment of the present invention, including:

[0149] A head orientation classification module 801, configured to obtain a face image captured by a camera, identify a head image from the face image, perform a head orientation classification operation on the head image, and determine the head orientation category of the head image through the head orientation classification operation, where the head orientation category includes a forward orientation category and a non-forward orientation category;

[0150] A face orientation determination module 802, configured to use the recognized head orientation category as the face orientation when the head orientation category is a non-forward orientation category.

[0151] The present invention captures a face image through a camera, performs a head orientation classification operation on the face image, and when the head orientation category is a non-forward orientation category, uses the recognized head orientation category as the face orientation. The present invention identifies the head orientation through image classification. Since when the head rotation angle is very large, it will be recognized as a non-forward orientation category, and at this time, the head orientation category is used as the face orientation. Therefore, even when the head rotation angle is very large, the face orientation can be recognized through the camera, thereby providing algorithm support for the distraction detection function of the DMS to determine whether the driver is facing forward and whether they are distracted.

[0152] In one embodiment, the recognizing a head image from the face image includes:

[0153] Recognize a head detection frame and a face detection frame from the face image, and use the image within the head detection frame as the head image, where the range of the head detection frame is larger than the range of the face detection frame.

[0154] In one embodiment, the performing a head orientation classification operation on the head image and determining the head orientation category of the head image through the head orientation classification operation includes:

[0155] Input the head image into a head orientation image classification model to obtain the head orientation category of the head image output by the head orientation image classification model, and the head orientation image classification model is pre-trained with multiple images and corresponding head orientation categories.

[0156] In one embodiment, the device further includes a second face orientation determination module, configured to:

[0157] When the head orientation category is a forward orientation category, recognize the image two-dimensional coordinates of the face key points in the image coordinate system from the face image;

[0158] Determine the face orientation angle according to the image two-dimensional coordinates of the face key points in the image coordinate system in the face image, where the image coordinate system is a coordinate system established on the face image.

[0159] In one embodiment, the recognizing the image two-dimensional coordinates of the face key points in the image coordinate system from the face image includes: recognizing the image two-dimensional coordinates of the face key points in the image coordinate system within the face detection frame recognized from the face image.

[0160] In one embodiment, determining the face orientation angle according to the two-dimensional image coordinates of the face key points in the face image in the image coordinate system includes:

[0161] Based on the camera parameters, calculate the conversion relationship for converting the standard three-dimensional coordinates of the face key points in the world coordinate system of the three-dimensional space into the two-dimensional image coordinates, and calculate the face orientation angle according to the conversion relationship.

[0162] In one embodiment, the calculating the conversion relationship for converting the standard three-dimensional coordinates of the face key points in the world coordinate system of the three-dimensional space into the two-dimensional image coordinates based on the camera parameters, and calculating the face orientation angle according to the conversion relationship includes:

[0163] Substitute the standard three-dimensional coordinates of the face key points in the world coordinate system of the three-dimensional space and the two-dimensional image coordinates into the formula: s*p = F*[R|t]P, where s is the scale factor, F is the camera internal parameter matrix, p is the two-dimensional image coordinates, P is the standard three-dimensional coordinates, R is the rotation matrix of the camera external parameters, and t is the translation matrix of the camera external parameters;

[0164] Calculate the rotation matrix and the translation matrix;

[0165] Calculate the face orientation angle according to the rotation matrix.

[0166] In one embodiment, the calculating the face orientation angle according to the rotation matrix includes:

[0167] Calculate the rotation angle of the face orientation relative to the camera coordinate system according to the rotation matrix;

[0168] Calculate the face orientation angle as the difference between the rotation angle and the installation angle of the camera.

[0169] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0170] As Figure 9 shown is a schematic hardware structure diagram of an electronic device according to the present invention, including:

[0171] At least one processor 901; and,

[0172] A memory 902 communicatively connected to at least one of the processors 901; wherein,

[0173] The memory 902 stores instructions that can be executed by at least one of the processors. The instructions are executed by at least one of the processors, enabling at least one of the processors to execute the vehicle face orientation recognition method described above.

[0174] Figure 9 Take one processor 901 as an example.

[0175] The electronic device may further include: an input device 903 and a display device 904.

[0176] The processor 901, the memory 902, the input device 903, and the display device 904 may be connected through a bus or other means. In the figure, connection through a bus is taken as an example.

[0177] The memory 902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the vehicle face orientation recognition method in the embodiments of the present application. For example, Figure 1 、 Figure 2 The method flow shown. The processor 901 executes various functional applications and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 902, that is, implements the vehicle face orientation recognition method in the above embodiments.

[0178] The memory 902 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the vehicle face orientation recognition method, etc. In addition, the memory 902 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 902 may optionally include a memory remotely set relative to the processor 901, and these remote memories can be connected to the device executing the vehicle face orientation recognition method through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0179] The input device 903 can receive input user clicks and generate signal inputs related to user settings and function controls of the vehicle face orientation recognition method. The display device 904 may include a display screen and other display devices.

[0180] When the one or more modules are stored in the memory 902 and run by the one or more processors 901, they execute the vehicle face orientation recognition method in any of the above method embodiments.

[0181] The present invention captures a face image through a camera, classifies the head orientation of the face image, and when the head orientation category is a non-forward orientation category, uses the recognized head orientation category as the face orientation. The present invention identifies the head orientation through image classification. Since when the head rotation angle is very large, it will be recognized as a non-forward orientation category, and at this time, the head orientation category is used as the face orientation. Therefore, even when the head rotation angle is very large, the face orientation can be recognized through the camera, thereby providing algorithm support for the distraction detection function of the DMS to determine whether the driver is facing forward and whether they are distracted.

[0182] An embodiment of the present invention provides a storage medium that stores computer instructions, which, when executed by a computer, are used to perform all the steps of the vehicle face orientation recognition method described above.

[0183] In the context of the present disclosure, the storage medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The storage medium can be a machine-readable signal medium or a machine-readable storage medium. Optionally, the storage medium can be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0184] An embodiment of the present invention provides a vehicle that includes the vehicle face orientation recognition device described above, or the electronic device described above. It can be understood that the vehicle can also include: a processor, a memory, and a computer program. Among them, the computer program is stored in the memory and is configured to be executed by the processor to implement the vehicle face orientation recognition method provided by the embodiments of the present disclosure. Among them, the processor and the memory have been described in the Figure 9 illustrated embodiments and will not be elaborated here.

[0185] The above-described embodiments merely represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims

1. A vehicle face orientation recognition method, characterized in that: include: Acquire a face image captured by a camera, identify a head image from the face image, perform a head orientation classification operation on the head image, and determine a head orientation category of the head image through the head orientation classification operation, wherein the head orientation category includes a positive orientation category and a non-positive orientation category; When the head orientation category is a non-positive orientation category, the identified head orientation category is used as the face orientation.

2. The vehicle face orientation recognition method according to claim 1, characterized in that: The step of identifying a head image from the face image comprises: A head detection frame and a face detection frame are identified from the face image, and the image within the head detection frame is used as the head image. The range of the head detection frame is larger than the range of the face detection frame.

3. The vehicle face orientation recognition method according to claim 1, characterized in that: The performing a head orientation classification operation on the head image and determining the head orientation category of the head image by the head orientation classification operation includes: The head image is input into a head orientation image classification model to obtain a head orientation category of the head image output by the head orientation image classification model, wherein the head orientation image classification model is pre-trained using multiple images and corresponding head orientation categories.

4. The vehicle face orientation recognition method according to claim 1, characterized in that: The method further comprises: When the head orientation category is a forward orientation category, identifying two-dimensional image coordinates of key facial points in an image coordinate system from the face image; The face orientation angle is determined according to the two-dimensional image coordinates of the face key points in the face image in the image coordinate system, where the image coordinate system is a coordinate system established on the face image.

5. The vehicle face orientation recognition method according to claim 4, characterized in that: The step of identifying the two-dimensional image coordinates of facial key points in the image coordinate system from the facial image includes: identifying the two-dimensional image coordinates of facial key points in the image coordinate system within a face detection frame identified from the facial image.

6. The vehicle face orientation recognition method according to claim 4, characterized in that: The step of determining the face orientation angle according to the two-dimensional image coordinates of the face key points in the face image in the image coordinate system includes: Based on the camera parameters, a conversion relationship is calculated to convert the standard three-dimensional coordinates of the key points of the face in the world coordinate system of the three-dimensional space into the two-dimensional coordinates of the image, and the face orientation angle is calculated according to the conversion relationship.

7. The vehicle face orientation recognition method according to claim 6, characterized in that: The step of calculating, based on the camera parameters, a conversion relationship of converting the standard three-dimensional coordinates of the key points of the face in the world coordinate system of the three-dimensional space into the two-dimensional coordinates of the image, and calculating the face orientation angle according to the conversion relationship includes: Substitute the standard three-dimensional coordinates of the facial key points in the world coordinate system of the three-dimensional space and the two-dimensional coordinates of the image into the formula: s*p=F*[R|t]P, where s is the scale factor, F is the camera intrinsic parameter matrix, p is the two-dimensional coordinates of the image, P is the standard three-dimensional coordinates, R is the rotation matrix of the camera extrinsic parameters, and t is the translation matrix of the camera extrinsic parameters; Calculating the rotation matrix and the translation matrix; The face orientation angle is calculated according to the rotation matrix.

8. The vehicle face orientation recognition method according to claim 7, characterized in that: Calculating the face orientation angle according to the rotation matrix includes: Calculating the rotation angle of the face relative to the camera coordinate system based on the rotation matrix; The face orientation angle is calculated as the difference between the rotation angle and the camera installation angle.

9. A vehicle face orientation recognition device, characterized in that: include: A head orientation classification module is used to obtain a face image taken by a camera, identify a head image from the face image, perform a head orientation classification operation on the head image, and determine the head orientation category of the head image through the head orientation classification operation, wherein the head orientation category includes a positive orientation category and a non-positive orientation category; The face orientation determination module is used to use the identified head orientation category as the face orientation when the head orientation category is a non-positive orientation category.

10. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the vehicle face orientation recognition method as described in any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium stores computer instructions, which, when executed by a computer, are used to execute all steps of the vehicle face orientation recognition method as described in any one of claims 1 to 8.

12. A vehicle, characterized in that: It includes the vehicle face orientation recognition device as described in claim 9, or the electronic device as described in claim 10.

Images