Method, device and equipment for determining depth information of driver, vehicle and medium

Abstract

The invention relates to a method, device and equipment for determining depth information of a driver, a vehicle and a medium. The method includes acquiring a facial image of a driver taken by a camera. The method further includes determining, by a deep learning model, rotation information based on the facial image, wherein the rotation information indicates rotation-related information in a transformational relationship from a world coordinate system to a camera coordinate system. The method further includes determining translation information based on the facial image by utilizing a statistical model, where the translation information indicates translation-related information in a transformational relationship from a world coordinate system to a camera coordinate system. Further, the method includes determining depth information of the driver relative to the camera based on the rotation information and the translation information. In this way, the accuracy of the determined depth information of the driver can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and more particularly, to a method, apparatus, device, vehicle, and medium for determining depth information of a driver. Background Art

[0002] Image-based depth information estimation refers to the use of computer vision technology to infer the depth information of pixels in an object or scene relative to the camera from an image. By estimating the driver's depth information, more driver behavior prediction and feedback information can be provided to the driving assistance system and autonomous driving technology, helping the system to work better with the driver and improve driving safety and comfort.

[0003] A depth camera is a camera that can simultaneously obtain the depth information of the pixels in the image and the scene. A binocular camera is a camera system composed of two camera modules, which can simulate the visual effect of human binocular observation. The depth information of the driver can be easily obtained by analyzing the images taken by the depth camera or binocular camera. However, due to the high cost of these cameras, monocular cameras are more often used in actual scenes to capture the driver in the vehicle. Therefore, it is necessary to estimate the depth information of the driver by analyzing the images taken by the monocular camera. Summary of the invention

[0004] Embodiments of the present disclosure propose a method, apparatus, device, vehicle, and medium for determining the depth information of a driver. In embodiments of the present disclosure, an image of the driver in a vehicle can be captured by a camera placed in the vehicle. In order to determine the transformation relationship from a world coordinate system to a camera coordinate system, embodiments of the present disclosure can use a deep learning model to determine rotation information related to rotation in the transformation relationship, and use a statistical model to determine translation information related to translation in the transformation relationship. Then, embodiments of the present disclosure can determine the depth information of the driver based on the facial image of the driver and the determined transformation relationship. In this way, a deep learning model can be used to generate rotation information with high accuracy, thereby significantly improving the accuracy of the determined driver depth information.

[0005] In a first aspect of the present disclosure, a method for determining depth information of a driver is provided. The method includes acquiring a facial image of the driver captured by a camera. The method also includes determining rotation information based on the facial image by a deep learning model, the rotation information indicating information related to rotation in a conversion relationship from a world coordinate system to a camera coordinate system. The method also includes determining translation information based on the facial image by utilizing a statistical model, the translation information indicating information related to translation in a conversion relationship from a world coordinate system to a camera coordinate system. In addition, the method also includes determining the depth information of the driver relative to the camera based on the rotation information and the translation information.

[0006] In a second aspect of the present disclosure, a device is provided. The device includes a facial image acquisition unit configured to acquire a facial image of a driver captured by a camera. The device also includes a rotation information determination unit configured to determine rotation information based on the facial image by a deep learning model, wherein the rotation information indicates information related to rotation in a conversion relationship from a world coordinate system to a camera coordinate system. The device also includes a translation information determination unit configured to determine translation information based on the facial image by utilizing a statistical model, wherein the translation information indicates information related to translation in a conversion relationship from a world coordinate system to a camera coordinate system. In addition, the device also includes a depth information determination unit configured to determine the depth information of the driver relative to the camera based on the rotation information and the translation information.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a method for determining the depth information of the driver, the method comprising obtaining a facial image of the driver taken by a camera. The method also includes determining rotation information based on the facial image by a deep learning model, the rotation information indicating information related to rotation in the conversion relationship from the world coordinate system to the camera coordinate system. The method also includes determining translation information based on the facial image by utilizing a statistical model, the translation information indicating information related to translation in the conversion relationship from the world coordinate system to the camera coordinate system. In addition, the method also includes determining the depth information of the driver relative to the camera based on the rotation information and the translation information.

[0008] In a fourth aspect of the present disclosure, a vehicle is provided, comprising a camera and the electronic device provided according to the third aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method provided according to the first aspect of the present disclosure.

[0010] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure may be implemented;

[0013] Figure 2 A flow chart showing a method for determining depth information of a driver according to some embodiments of the present disclosure is shown;

[0014] Figure 3 A schematic diagram showing the architecture of a rotation information generation model according to some embodiments of the present disclosure;

[0015] Figure 4 A schematic diagram showing the architecture of a translation information generation model according to some embodiments of the present disclosure;

[0016] Figure 5 A schematic diagram showing an example process of projecting a driver's reference point from a world coordinate system to a camera coordinate system using a rotation matrix and a translation vector according to some embodiments of the present disclosure, thereby determining the driver's depth information;

[0017] Figure 6 A schematic diagram showing an example process of determining a projection scale according to some embodiments of the present disclosure;

[0018] Figure 7 A flow chart showing a process for determining depth information of a driver according to some embodiments of the present disclosure;

[0019] Figure 8 A block diagram showing an apparatus for determining depth information of a driver according to some embodiments of the present disclosure; and

[0020] Fig. 9 A block diagram of a device in which various embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] A monocular camera refers to a camera device with only one camera module. In the task of 3D scene reconstruction or depth estimation using a monocular camera, some traditional solutions use deep learning methods to perform these tasks. Although these solutions have good accuracy, they require a lot of computing resources, data resources and time costs. Some traditional solutions use statistical methods of 3D deformable facial models to perform monocular 3D tasks. Although these solutions have higher execution efficiency than solutions using deep learning, they are not satisfactory in terms of accuracy because they cannot obtain personalized and accurate 3D facial models during the calculation process and can usually only obtain approximate solutions.

[0024] To this end, an embodiment of the present disclosure proposes a scheme for determining the depth information of the driver. In an embodiment of the present disclosure, a camera is placed in the vehicle, and an image of the driver in the vehicle can be captured by the camera. In order to determine the depth information of the driver based on the image, it is necessary to determine the conversion relationship from the world coordinate system to the camera coordinate system. The embodiment of the present disclosure adopts a deep learning model to determine the rotation information related to the rotation in the conversion relationship, and adopts a statistical model to determine the translation information related to the translation in the conversion relationship. The rotation information and the translation information are combined to obtain the conversion relationship from the world coordinate system to the camera coordinate system. Then, the embodiment of the present disclosure can determine the depth information of the driver based on the facial image of the driver and the conversion relationship.

[0025] In this way, a deep learning model can be used to generate rotation information with high accuracy, thereby avoiding the use of a statistical model to calculate the rotation information. Since the defects of the traditional statistical model scheme have a greater impact on the accuracy of the rotation information than on the accuracy of the translation information, the embodiments of the present disclosure can significantly improve the accuracy of the determined driver depth information.

[0026] Figure 1 1 is a schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. Figure 1As shown, in the environment 100, a driver 102 and a camera 104 are included in the vehicle. The camera 104 may be a color (RGB) camera that can capture images of three color channels of red (R), green (G) and blue (B). This camera uses the RGB color model to represent each pixel in the image as a numerical combination of the three color channels, thereby generating a color image. The camera 104 may also be an infrared (IR) camera that can sense infrared radiation and convert it into a visible image. Unlike ordinary color cameras, infrared cameras can receive light signals within the infrared band and convert them into images or thermal imaging images visible to the human eye. Since infrared cameras can achieve night vision functions in night vision and low light environments without relying on external light sources, and have the characteristics of less interference from strong light, infrared cameras can be used instead of color cameras in some scenes of autonomous driving or assisted driving. By using an infrared camera to capture the driver's image, the vehicle can determine the driver's depth information in a wider range of usage scenarios.

[0027] In environment 100, camera 104 may capture image 106 of a driver in a vehicle, which may generally include the face, torso, arms, etc. of driver 102. For example, Figure 1 As shown, the image 106 may include a facial image 108 of the driver 102. The environment 100 also includes a rotation information generation model 110, which uses a deep learning method to generate a rotation matrix (also referred to as rotation information) 114 related to rotation in the conversion relationship from the world coordinate system to the camera coordinate system of the camera 104. The environment 100 also includes a translation information generation model 112, which uses a statistical method to generate a translation vector (also referred to as translation information) 116 related to translation in the conversion relationship from the world coordinate system to the camera coordinate system of the camera 104. The environment 100 also includes a three-dimensional facial model 118, which can be a three-dimensional facial model created by a three-dimensional facial modeling technology (e.g., a three-dimensional deformable facial modeling (3DMM) technology). In some embodiments, the three-dimensional facial model 118 can adopt the average facial shape of the group to which the driver 102 belongs (e.g., the average facial shape of Asians).

[0028] Since the camera 104 can be placed at any position in the environment, a reference coordinate system is selected in the environment to describe the position of the camera 104, and is used to describe the position of any point or object in the environment. The reference coordinate system is called the world coordinate system. The camera coordinate system is a coordinate system associated with the camera 104. The origin of the camera coordinate system can be located at the optical center of the camera. The horizontal axis (X axis) and the vertical axis (Y axis) of the camera coordinate system can be aligned with the imaging plane of the camera 104, and the vertical axis (Z axis) is the optical axis of the camera 104. A point in the camera coordinate system represents the position and direction of the point relative to the camera, and can be represented by (x, y, z). The conversion between the camera coordinate system and the world coordinate system may involve the intrinsic and extrinsic parameters of the camera 104. The intrinsic parameters represent the internal parameters of the camera 104, such as focal length, principal point position, pixel size, etc., and the intrinsic parameters are usually fixed when the camera leaves the factory. The extrinsic parameters represent the position and direction of the camera 104 in the world coordinate system, and are used to convert points in the world coordinate system into the camera coordinate system. In the environment 100, the rotation matrix 114 and the translation vector 116 are external parameters of the camera 104. In the embodiment of the present disclosure, the rotation matrix 114 and the translation vector 116 can be used to convert the coordinates of a designated reference point of the driver's face (the position of the point represents the position of the driver, for example, the tip of the nose) in the world coordinate system into coordinates in the camera coordinate system of the camera 104, so as to determine the depth from the reference point to the camera based on the coordinates, and the depth can be represented by the depth information 124. The depth information 124 represents both the depth information of the reference point and the depth information of the driver 102.

[0029] Figure 2 1 is a flow chart of a method 200 for determining depth information of a driver according to some embodiments of the present disclosure. The method 200 may be performed by a device or electronic device coupled to or in communication with a camera. Figure 2 As shown, in box 202, method 200 obtains a facial image of the driver captured by a camera. Figure 1 In the illustrated environment 100 , an image 106 including a driver 102 may be captured by a camera 104 , and the method 200 may then obtain a facial image 108 of the driver 102 in the image 106 , wherein the driver's face occupies a major portion of the facial image 108 .

[0030] At block 204, rotation information is determined based on the facial image by a deep learning model. Figure 1In the environment 100 shown, the method 200 can generate a rotation matrix 114 based on the driver's facial image 108 using a deep learning model in the rotation information generation model 110, and the rotation matrix 114 indicates the rotation-related information in the conversion relationship from the world coordinate system to the camera coordinate system of the camera 104. In the embodiment of the present disclosure, the rotation information generation model 110 mainly uses a deep learning method to generate the rotation matrix 114. The deep learning model is usually composed of a multi-layer neural network, and each layer can learn different levels of feature representation. In addition, the deep learning model can automatically learn features from the data without manually involving a feature extractor, so that the features of the data can be better captured. Therefore, the rotation matrix 114 generated using the deep learning method has a higher accuracy than the rotation matrix calculated using a statistical method.

[0031] At block 206, method 200 may determine translation information based on the facial image by utilizing a statistical model. Figure 1 In the illustrated environment 100, the method 200 can generate a translation vector 116 based on the driver's facial image 108 using the translation information generation model 112, where the translation vector 116 indicates information related to translation in the conversion relationship from the world coordinate system to the camera coordinate system of the camera 104. The translation information generation model 112 mainly uses a statistical method to calculate the translation vector 116, and therefore does not involve a large number of parameters and a complex calculation process.

[0032] At block 208, method 200 may determine depth information of the driver relative to the camera based on the rotation information and the translation information. Figure 1 In the illustrated environment 100, the method 200 may determine the depth information 124 based on the driver's facial image 108, the rotation matrix 114, and the translation vector 116. The depth information 124 may be the depth of a specified reference point of the driver's face relative to the camera 104. By utilizing the depth information 124, a computer on the vehicle may, for example, analyze the driver's posture, thereby issuing an alarm in time when dangerous driving behavior occurs, so as to improve the safety of the driving process.

[0033] It should be noted that although Figure 2 204 is shown before box 206, but it is not intended to limit the order of the operations performed at box 204 and box 206. On the contrary, the operations performed at box 204 and box 206 can be performed in a swapped order or simultaneously. In this way, method 200 can use a deep learning method to generate a rotation matrix with higher accuracy, thereby further improving the accuracy of the driver's depth information.

[0034] In order to obtain the driver's facial image, in some embodiments, a driver image of the driver in the vehicle may be obtained, and then the driver's facial image may be determined based on the driver image by using a facial detection model. Figure 1 In the environment 100 shown, the camera 104 may capture a driver image 106 that includes more body parts of the driver, and then a facial detection model may be used to determine a driver facial image 108 from the driver image 106, in which the face of the driver 102 occupies most of the image area.

[0035] In order to generate a rotation matrix, in some embodiments, a deep learning model can be used to determine the driver's head posture Euler angles based on the driver's facial image, and then determine the rotation matrix based on the head posture Euler angles. Figure 3 FIG. 3 is a schematic diagram showing the architecture of a rotation information generation model 300 according to some embodiments of the present disclosure. Figure 3 As shown, the rotation information generation model 300 includes a head posture model 304, which is a deep learning model. The head posture model 304 receives the driver's facial image 302 (e.g., Figure 1 ) as input, and output Euler angles 306. Euler angles are used to describe the orientation of a rigid body in a three-dimensional Euclidean space. For a reference system in a three-dimensional space, the orientation of any coordinate system can be expressed by three Euler angles, i.e., yaw, pitch, and roll, and these three angles represent the clockwise or counterclockwise rotation angle of the object around the three coordinate axes. For example, if the rotation angles of the driver's head around the X-axis, Y-axis, and Z-axis in the three-dimensional space are 30 degrees, 60 degrees, and 45 degrees, respectively, then its Euler angles can be expressed as (30 degrees, 60 degrees, 45 degrees), i.e., 30 degrees of rotation around the X-axis, then 60 degrees of rotation around the Y-axis, and finally 45 degrees of rotation around the Z-axis. Such Euler angles can be used to describe the posture of an object or its rotation state in a three-dimensional space.

[0036] In some embodiments, the head posture model 304 may include an input layer, a feature extraction layer, a pooling layer, a fully connected layer, and an output layer. In the input layer, the driver's facial image 302 is input to the head posture model 304. In some embodiments, the driver's facial image 302 may be a grayscale image, and it is obtained by an infrared camera (e.g., Figure 1The facial image 302 is obtained by taking a picture with the camera 104 in the image. In the feature extraction layer, a convolutional neural network (CNN) can be used to extract features in the facial image 302. The convolutional neural network includes a plurality of convolutional layers, each of which is composed of a group of convolutional kernels, and each convolutional kernel extracts a specific feature. By introducing multi-core convolution, different features in the image can be extracted using different convolutional kernels. However, as the number of convolutional kernels increases, the dimension of the convolutional layer becomes larger and larger, so a pooling layer can be used to reduce the dimension of the convolutional layer. For example, the size of the pooling window can be limited, and then the maximum value pooling is used to take the maximum value within the pooling window range as the sampled output value. A fully connected layer can be added after the pooling layer to map the extracted image features to the output of the head posture. The head posture model 304 can flatten the features of the image into a vector, and map the features through a plurality of fully connected layers to generate a prediction result. Then, the prediction result is input to the output layer to generate the pitch angle, yaw angle, and roll angle. After generating the predicted Euler angle, a loss function such as mean square error can be used to measure the angular difference between the predicted Euler angle and the pre-labeled true Euler angle, and then an algorithm such as stochastic gradient descent can be used to minimize the loss function and update the model parameters of the head posture model 304 through back propagation.

[0037] By training the head posture model 304 on a large number of labeled images, the head posture model 304 can accurately capture the image features in the driver's facial image 302, and can automatically learn the mapping relationship from image features to Euler angles by comparing the predicted Euler angles with the actual Euler angles, thereby being able to predict Euler angles with higher accuracy.

[0038] like Figure 3 As shown, after the head posture model 304 outputs the Euler angle 306, the rotation information generation model 300 can generate a rotation matrix 314 based on the Euler angle 306. In some embodiments, the rotation matrix 314 can be calculated based on the Euler angle 306 using the right-hand rule internal rotation, that is, the three-dimensional key point is first rotated around the yaw axis (Y axis) by the yaw angle, then rotated around the pitch axis (X axis) by the pitch angle, and finally rotated around the roll axis (Z axis) by the roll angle. α represents the yaw angle, β represents the pitch angle, γ represents the roll angle, and R y (α) represents the rotation matrix around the yaw axis, R x (β) represents the rotation matrix around the pitch axis, R z (γ) represents the rotation matrix of the roll axis, and the rotation matrix 314R (α, β, γ) can be expressed by the following equation (1):

[0039]

[0040] It should be noted that the above embodiment only shows an Euler angle rotation sequence as an example, and in other embodiments, the rotation matrix can also be calculated based on other rotation sequences. For example, in some embodiments, the rotation can be performed first around the roll axis, then around the pitch axis, and finally around the yaw axis.

[0041] In this way, the rotation information generation model 300 can use a deep learning method to generate a rotation matrix 314 based on the driver's facial image 302. Since the head posture model 304 can improve the accuracy of the predicted Euler angles, the accuracy of the rotation matrix 314 is also improved, so that the coordinates in the world coordinate system can be more accurately converted to the camera coordinate system.

[0042] In order to convert the coordinates in the world coordinate system into the camera coordinate system, it is also necessary to determine the translation vector after generating the rotation matrix. In some embodiments, a two-dimensional key point set can be determined based on the driver's facial image by utilizing a facial key point detection model. Then, a three-dimensional key point set (also referred to as a three-dimensional key point set) in the world coordinate system corresponding to the two-dimensional key point set can be determined based on the two-dimensional key point set. Then, a translation vector can be determined based on the two-dimensional key point set and the three-dimensional key point set. In some embodiments, in order to determine the three-dimensional key point set, a three-dimensional facial model corresponding to the driver's face can be obtained, and then the three-dimensional key points corresponding to the two-dimensional key points in the two-dimensional key point set in the three-dimensional facial model are determined. In some embodiments, a three-dimensional coordinate set in the camera coordinate system can be determined based on the two-dimensional key point set and the intrinsic parameters of the camera, and then a translation matrix is ​​determined based on the three-dimensional coordinate set in the world coordinate system corresponding to the three-dimensional key point set and the three-dimensional key point set in the camera coordinate system.

[0043] Figure 4 FIG. 4 is a schematic diagram showing the architecture of a translation information generation model 400 according to some embodiments of the present disclosure. Figure 4 As shown, the translation information generation model 400 uses the facial key point detection model 404 to detect the driver's facial image 402 (e.g., Figure 3402 ) in the driver's facial image 302, thereby generating a two-dimensional key point set 406. Facial key points refer to a plurality of important points with specific meanings and positions in the facial image, which can be used to identify various characteristic parts of the face, such as eyes, nose, mouth, eyebrows, and other important structures. In some embodiments, facial key points may include eye key points (inner corners of the left and right eyes, outer corners of the eyes, center points of the eyes, etc.), nose key points (nose tip, nose wings, etc.), mouth key points (corners of the mouth, center points of the mouth, etc.), eyebrow key points (starting points and ending points of the left and right eyebrows, etc.), and facial edge key points, etc. Each point in the two-dimensional key point set 406 is a two-dimensional point, and its coordinates in the facial image 402 can be represented by (x, y).

[0044] After determining the two-dimensional key point set 406, the translation information generation model 400 can obtain a pre-generated three-dimensional facial model as the three-dimensional facial model 408 of the driver 102, and determine a three-dimensional key point set 410 corresponding to the two-dimensional key point set 406 in the three-dimensional facial model 408 (including the coordinates of each three-dimensional key point in the three-dimensional key point set 410 in the world coordinate system). The three-dimensional facial model 408 can be a three-dimensional facial model created by a three-dimensional facial modeling technology (e.g., a three-dimensional deformable facial modeling (3DMM) technology). In some embodiments, the average facial shape of the group to which the driver 102 belongs (e.g., the average facial shape of Asians) can be used as the three-dimensional facial model 408. Then, the translation information generation model 400 can generate a translation vector 416 based on the two-dimensional key point set 406 and the three-dimensional key point set 410.

[0045] In some embodiments, the coordinates of point P in the world coordinate system can be represented by (U, V, W), the coordinates of point P in the camera coordinate system of camera 104 can be represented by (X, Y, Z), the rotation matrix can be represented by R, and the translation vector can be represented by t. Then the conversion relationship between the coordinates of point P in the world coordinate system and the coordinates of point P in the camera coordinate system can be represented by the following equation (2):

[0046]

[0047] In order to determine the transformation relationship from the coordinates of point P in the world coordinate system to the coordinates in the camera coordinate system (i.e., the rotation matrix R and the translation vector t), it is necessary to determine the coordinates of a certain number of points in the world coordinate system and the coordinates of these points in the camera coordinate system. Since the translation information generation model 400 has determined the coordinates of the three-dimensional key point set 410 and the three-dimensional key points in the three-dimensional key point set 410 in the world coordinate system, it is also necessary to determine the coordinates of these points in the camera coordinate system. In some embodiments, a three-dimensional coordinate set corresponding to the two-dimensional key point set 406 can be generated by filling the coordinates of the Z axis (e.g., 1) into the coordinates of each two-dimensional key point in the two-dimensional key point set 406. The coordinates in the three-dimensional coordinate set can be, for example, (x, y, 1). Then, based on the filled three-dimensional coordinate set and the intrinsic parameters of the camera, the three-dimensional coordinates in the camera coordinate system corresponding to the filled three-dimensional coordinates are calculated. In this way, through the correspondence between the two-dimensional key point set 406 and the three-dimensional key point set 410, the coordinates of multiple points in the world coordinate system and the coordinates in the camera coordinate system can be obtained, so that multiple equations can be generated and the analytical solutions of the rotation matrix R and the translation vector t can be calculated. If the multiple equations have multiple solutions, the least squares solution (i.e., approximate solution) is calculated, so that the translation information generation model 400 can obtain the translation vector 416 and the rotation matrix 418 by calculation. However, in the embodiment of this article, only the translation vector 416 will be used, and the rotation matrix 418 will be discarded.

[0048] In this way, the translation vector 416 can be calculated by a statistical method, and compared with the deep learning method, the statistical method saves a lot of computing resources and computing time. However, in the process of calculating the rotation matrix R and the translation vector t, there are multiple links that will lead to a reduction in the accuracy of the final calculation result. For example, using the average facial model of the group as the three-dimensional facial model 408 of the driver 102, and only being able to calculate an approximate solution when solving multiple sets of equations. Therefore, in the embodiment of the present disclosure, the rotation matrix 418 calculated by the translation information generation model 400 is abandoned, and the translation vector 416 and the rotation matrix 314 generated by the rotation information generation model 300 are used to determine the depth information of the driver, thereby achieving the determination of the driver's depth information with higher accuracy within an acceptable resource consumption and time.

[0049] After the rotation matrix and the translation vector are determined, in order to determine the depth information of the driver, in some embodiments, the reference point of the driver's face can be determined, and the coordinates of the reference point in the world coordinate system can be determined. Then, based on the rotation matrix and the translation vector, the coordinates of the reference point in the camera coordinate system can be determined, and based on the vertical axis value (Z axis value) of the coordinates of the reference point in the camera coordinate system, the depth information of the reference point can be determined as the depth information of the driver.

[0050] Figure 5 FIG. 5 is a schematic diagram of an example process 500 for projecting a driver's reference point from a world coordinate system to a camera coordinate system using a rotation matrix and a translation vector according to some embodiments of the present disclosure, thereby determining the driver's depth information. Figure 5 As shown, process 500 may determine a driver (e.g., Figure 1 For example, the nose tip of the driver 102 may be determined as the reference point, which means that the nose tip of the driver 102 is relative to the camera (eg, Figure 1 The depth information of the camera 104 in FIG. 1 may represent the depth information of the driver 102 relative to the camera 104. Then, the three-dimensional face model of the driver 102 (e.g., Figure 4 The three-dimensional facial model 408 in the image is used to determine the coordinates 502 (expressed as (U1, V1, W1)) of the nose tip in the world coordinate system 508. In order to convert the coordinates 502 into the coordinates 504 (expressed as (X1, Y1, Z1)) in the camera coordinate system 510, the values ​​of the coordinates 504 (X1, Y1, Z1) can be obtained based on the coordinates 502, the rotation matrix 514 and the translation vector 516 using the equation (2) described above. The rotation matrix 514 is calculated using the following formula: Figure 3 The rotation matrix generated by the translation information generation model 300 shown in FIG. 5 is obtained by using the following formula: Figure 4 After obtaining the coordinate 504 of the nose tip of the driver 102 in the camera coordinate system 510 of the camera 104, the value of the Z axis of the coordinate 504 (i.e., Z1) indicates the depth of the point relative to the camera 104. However, Z1 indicates the distance from the nose tip to the origin in the coordinate system. To obtain the distance between the two points in the real world, the projection scale 506 of the camera coordinate system 510 to the real world can be determined, and then the depth information 524 from the reference point to the camera can be determined based on the value of Z1 and the projection scale 506.

[0051] In this way, the rotation matrix 514 generated by the deep learning method and the translation vector 516 generated by the statistical method can be used to determine the depth information 524 of the driver 102 relative to the camera 104, thereby significantly improving the accuracy of the determined driver depth information.

[0052] In order to determine the projection scale 506, in some embodiments, two reference points (i.e., the first reference point and the second reference point) in the driver's facial image can be determined, and the real distance between the two reference points in the real world can be obtained, and the pixel distance between the two reference points in the driver's facial image can be determined. Then, the projection scale can be determined based on the pixel distance and the real distance between the two reference points. In some embodiments, the ratio of the real distance and the pixel distance between the two reference points can be determined as the projection scale. Then, the depth information of the driver relative to the camera in the real world can be determined based on the Z-axis value of the driver's reference point in the camera coordinate system and the projection scale. In some embodiments, when selecting two reference points in the driver's facial image, a point whose depth between the two points is greater than a predetermined threshold can be selected as a reference point.

[0053] Figure 6 FIG. 6 is a schematic diagram showing an example process 600 for determining a projection scale according to some embodiments of the present disclosure. Figure 6 As shown, process 600 can select two points in driver's facial image 608 as reference points. Since the greater the depth between the two reference points, the better, a threshold of the depth between the two points can be preset, and points whose depth between the two points exceeds the threshold can be selected as reference points. Figure 6 In the example shown, the driver's nose tip and chin point are selected as two reference points. After determining the two reference points, process 600 can determine the pixel distance between the nose tip and the chin point in the driver's facial image 608 (e.g., 210 pixels). In addition, process 600 can also determine the empirical value of the real distance between the nose tip and the chin point in the real world (e.g., the distance from the nose tip to the chin in the real world is approximately 90 mm). Then, process 600 can determine the projection scale 606 based on the ratio between the real distance between the nose tip and the chin point and the pixel distance. In some embodiments, the Z-axis value of the coordinates of the driver's reference point in the camera coordinate system (e.g., Figure 5 The Z1 value of the coordinate 504 in (Z1 value) is multiplied by the projection scale 606 to obtain the depth information of the driver.

[0054] In this way, the projection scale of the camera coordinate system to the real world can be estimated, so that the depth information of the reference point relative to the camera in the real world can be determined based on the coordinates of the reference point in the camera coordinate system.

[0055] In some embodiments, a world coordinate system can be established with reference to the real world without scaling. In these embodiments, after the reference point of the driver's face is converted from the world coordinate system to the camera coordinate system, the driver's depth information can be determined based on the Z-axis value of the coordinates in the obtained camera coordinate system. In this case, the depth information of the driver relative to the camera can be determined without determining the projection scale.

[0056] Figure 7 FIG. 7 is a flow chart showing a process 700 for determining depth information of a driver according to some embodiments of the present disclosure. Figure 7 As shown, at block 702, process 700 may acquire a driver image. Figure 1 In the illustrated environment 100, the process 700 may obtain an image 106 of a driver 102 in a vehicle captured by a camera 104. In some embodiments, the camera 104 may be an infrared camera, so the driver image 106 captured by the camera 104 is a grayscale image.

[0057] At block 704, process 700 may utilize a facial detection model to determine a facial image of the driver. Figure 1 In the environment shown, process 700 can utilize face detection model to detect the driver's face part in driver image 106, thereby generate driver's face image 108.Because traditional face detection model is trained for color image, the performance of these models on grayscale image is poor, which can cause the accuracy of subsequent calculation process to reduce.Therefore, in some embodiments, training data and corresponding label can be generated for grayscale image, and grayscale image is utilized to train face detection model then.Can improve the accuracy of driver's face image 108 detected under infrared camera scene in this way.

[0058] At block 706, process 700 may determine Euler angles using a head pose model. Figure 3 In the rotation information generation model 300 shown, a deep learning head posture model 304 can be trained, which receives the driver's facial image 302 as input and outputs Euler angles 306 including yaw angle, pitch angle and roll angle. In some embodiments, the head posture model 304 can be constructed based on a convolutional neural network to better learn the features in the image and improve the accuracy of the generated Euler angles 306. In some embodiments, a transformer-based model can be used to extract features in the image, and the self-attention mechanism and parallel computing characteristics of the transformer model can be fully utilized to improve the overall performance of the head posture model 304. Similarly, in some embodiments, since the camera 104 is an infrared camera and the driver's facial image 302 is a grayscale image, the head posture model 304 can be trained based on the grayscale image, which can improve the accuracy of the Euler angles 306 generated in the infrared camera scene. In box 708, the process 700 uses Euler angles to calculate the rotation matrix. For example, in Figure 3In the rotation information generation model 300 shown, the process 700 can generate a rotation matrix 314 based on the Euler angle 306 and through the equation (1) described above, where the rotation matrix 314 indicates the information related to the rotation in the conversion relationship from the world coordinate system to the camera coordinate system.

[0059] At block 710, process 700 may utilize a key point detection model to determine facial key points in the driver's facial image. Figure 4 In the translation information generation model 400 shown, the key point model 404 receives the driver's facial image 402 as input and generates a two-dimensional key point set 406. The key point detection model 404 is also a deep learning model, and in some embodiments, since the camera 104 is an infrared camera and the driver's facial image 302 is a grayscale image, the key point detection model 304 can be trained based on the grayscale image, which can improve the accuracy of the two-dimensional key point set 406 determined in the infrared camera scene.

[0060] At block 712, process 700 may calculate a translation vector using a set of 2D key points on the driver's face and a 3D facial model. Figure 4 In the translation information generation model 400 shown, the group average facial model can be used as the three-dimensional facial model 408 of the driver 102, and then a three-dimensional key point set 410 corresponding to the two-dimensional key point set 406 is determined in the three-dimensional facial model 408. Based on the two-dimensional key point set 406 and the three-dimensional key point set 410, the translation vector 416 can be determined using equation (2) described above, and the translation vector 416 indicates information related to the translation in the conversion relationship from the world coordinate system to the camera coordinate system.

[0061] At block 714, process 700 may utilize the determined rotation matrix and translation vector to transform the coordinates of the driver's facial reference points in the world coordinate system into the camera coordinate system. Figure 5 In the process 500 shown, the nose tip of the driver may be determined as a reference point, and then based on the rotation matrix 314 ( Figure 5 The rotation matrix 514 in ), the translation vector 416 calculated at block 712 ( Figure 5 The coordinates 504 of the nose tip point in the camera coordinate system are calculated using equation (2) described above, based on the translation vector 516 in the world coordinate system and the coordinates 502 of the nose tip point in the world coordinate system.

[0062] At block 716, process 700 may determine a projection scale based on the true distance and the pixel distance between the two reference points. Figure 6In the illustrated process 600, the nose tip and chin of the driver may be determined as two reference points, and then the pixel distance 602 between the nose tip and the chin in the driver's facial image 608 and the empirical value of the real distance 604 between the nose tip and the chin in the real world may be determined. Then, the projection scale 606 may be determined by determining the ratio between the real distance 604 and the pixel distance 602.

[0063] At block 718, process 700 may determine the driver's depth information based on the determined projection scale and the Z-axis value of the coordinates of the driver's reference point in the camera coordinate system. Figure 5 In the illustrated process 500 , the Z-axis value (ie, Z1 ) of the coordinate 504 of the driver's facial reference point in the camera coordinate system may be determined, and then the depth information 524 of the driver relative to the camera may be determined by multiplying the value of Z1 by the projection scale 506 .

[0064] In this way, process 700 can utilize deep learning methods to generate a rotation matrix with higher accuracy, thereby further improving the accuracy of the driver's depth information.

[0065] Figure 8 FIG. 8 is a block diagram of an apparatus 800 for determining depth information of a driver according to some embodiments of the present disclosure. Figure 8 As shown, the device 800 includes a facial image acquisition unit 802, which is configured to acquire a facial image of the driver taken by a camera. The device 800 also includes a rotation information determination unit 804, which is configured to determine rotation information based on the facial image by a deep learning model, and the rotation information indicates information related to rotation in the conversion relationship from the world coordinate system to the camera coordinate system. The device 800 also includes a translation information determination unit 806, which is configured to determine translation information based on the facial image by utilizing a statistical model, and the translation information indicates information related to translation in the conversion relationship from the world coordinate system to the camera coordinate system. In addition, the device 800 also includes a depth information determination unit 808, which is configured to determine the depth information of the driver relative to the camera based on the rotation information and the translation information.

[0066] It is understood that the device 800 of the present disclosure can achieve at least one of the advantages that can be achieved by the method or process described above. For example, the device 800 can use a deep learning method to generate a rotation matrix with high accuracy, thereby further improving the accuracy of the driver's depth information.

[0067] Fig. 9A schematic block diagram of an example device 900 that can be used to implement an embodiment of the present disclosure is shown. As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0068] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0069] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as method 200. For example, in some embodiments, method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform method 200 in any other appropriate manner (e.g., by means of firmware).

[0070] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0071] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0072] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, although each operation is depicted in a specific order, this should be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0073] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for determining depth information of a driver, comprising: Acquire a facial image of the driver captured by a camera; Determining rotation information based on the facial image by a deep learning model, the rotation information indicating information related to rotation in a conversion relationship from a world coordinate system to a camera coordinate system; determining translation information based on the facial image by using a statistical model, the translation information indicating information related to translation in the transformation relationship from the world coordinate system to the camera coordinate system; as well as Based on the rotation information and the translation information, the depth information of the driver relative to the camera is determined.

2. The method according to claim 1, wherein the deep learning model is a head posture model, and acquiring the facial image of the driver comprises: acquiring a driver image of the driver in the vehicle; as well as The facial image is determined based on the driver image by utilizing a facial detection model.

3. The method of claim 2, wherein determining the rotation information comprises: Determining the driver's head posture Euler angles by the head posture model based on the facial image; as well as Based on the head posture Euler angles, a rotation matrix is ​​determined as the rotation information.

4. The method of claim 1, wherein the deep learning model is a head pose model, and determining the translation information comprises: determining a set of two-dimensional key points based on the facial image by utilizing a facial key point detection model; Based on the two-dimensional key point set, determining a three-dimensional key point set in the world coordinate system corresponding to the two-dimensional key point set; as well as A translation vector is determined as the translation information based on the two-dimensional key point set and the three-dimensional key point set in the world coordinate system.

5. The method according to claim 4, wherein determining the set of three-dimensional key points comprises: Acquiring a three-dimensional facial model corresponding to the driver's face; and Determine three-dimensional key points in the three-dimensional facial model that correspond to two-dimensional key points in the two-dimensional key point set.

6. The method of claim 4, wherein determining the translation vector comprises: Determine a three-dimensional coordinate set in the camera coordinate system based on the two-dimensional key point set and the intrinsic parameters of the camera; as well as The translation vector is determined based on a set of three-dimensional coordinates in the world coordinate system and a set of three-dimensional coordinates in the camera coordinate system corresponding to the set of three-dimensional key points.

7. The method of claim 1, wherein determining the depth information of the driver in the facial image further comprises: determining reference points of the driver's face; Determine a first coordinate of the reference point in the world coordinate system; Determine a second coordinate of the reference point in the camera coordinate system based on the first coordinate in the world coordinate system, the rotation information, and the translation information; as well as Based on the vertical axis value of the second coordinate, the depth information of the reference point is determined as the depth information of the driver.

8. The method of claim 7, wherein determining the depth information of the driver in the facial image further comprises: determining a first reference point and a second reference point in the facial image; Acquire a real distance between the first reference point and the second reference point in the real world; determining a pixel distance between the first reference point and the second reference point in the facial image; as well as The depth information of the reference point is determined based on the vertical axis value of the second coordinate, the pixel distance, and the real distance.

9. The method of claim 8, wherein determining the depth information of the driver in the facial image further comprises: determining a ratio of the true distance to the pixel distance; as well as The depth information of the reference point is determined based on the ratio and the vertical axis value of the second coordinate.

10. The method of claim 8, wherein a depth between the first reference point and the second reference point in the facial image is greater than a predetermined threshold.

11. The method according to claim 1, wherein: The camera is an infrared camera and the facial image is a grayscale image; and The face detection model is trained based on grayscale images.

12. A device for determining depth information of a driver, comprising: A facial image acquisition unit, configured to acquire a facial image of the driver captured by a camera; a rotation information determining unit configured to determine rotation information based on the facial image by a deep learning model, wherein the rotation information indicates information related to rotation in a conversion relationship from a world coordinate system to a camera coordinate system; a translation information determining unit configured to determine translation information based on the facial image by using a statistical model, the translation information indicating information related to translation in the conversion relationship from the world coordinate system to the camera coordinate system; as well as A depth information determination unit is configured to determine the depth information of the driver relative to the camera based on the facial image, the rotation information and the translation information.

13. An electronic device comprising: at least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the apparatus to perform the method according to any one of claims 1-11.

14. A vehicle comprising a camera and the electronic device according to claim 13.

15. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 11.

Images