A head pose correction method and system based on camera distortion compensation

Abstract

The present application relates to the field of head pose estimation, and more particularly to a head pose correction method and system based on camera distortion compensation. The present application takes the center point of the face to be recognized in the captured image as the target point, calculates the head pose rotation angle error according to the relative position of the target point and the center point of the captured image, realizes the individual error calculation for each face to be recognized, and corrects the error according to the position of each face to be recognized in the captured image, so that the deflection angle of the face to be recognized in the camera monitoring picture can be accurately calculated, and more accurate head pose recognition results can be obtained when the head pose is analyzed based on the deflection angle.

Description

Technical Field

[0001] This invention relates to the field of head pose estimation, and more particularly to a head pose correction method and system based on camera distortion compensation. Background Technology

[0002] Head posture refers to the position and angle of the human head, which can be represented in three-dimensional space by pitch, yaw, and roll angles. Depending on the angle, postures such as head tilting, head lowering, head turning left, head turning right, head tilting left, and head tilting right can be identified. Head posture recognition typically employs deep learning methods, analyzing and processing video from cameras to obtain head posture information.

[0003] Most existing head posture detection devices use fixed-position wide-field-of-view cameras. When there is a certain angle between the target and the camera, the camera captures the side of the face, which can easily misidentify the head posture as turning left or right. In addition, the wide-field-of-view camera lens itself has a certain degree of distortion, which causes the target position to be stretched or compressed, which will also increase the head posture recognition error. Summary of the Invention

[0004] To address the shortcomings of low head pose recognition accuracy in the existing technologies, this invention proposes a head pose correction method based on camera distortion compensation. This method can correct head pose angle errors caused by camera distortion and when the target appears at the side of the camera, thereby improving the accuracy of head pose detection.

[0005] This invention proposes a head pose correction method based on camera distortion compensation, combined with the horizontal error δ of the head pose rotation angle. x and vertical error δ y The head pose angle obtained through the head pose algorithm is corrected; the horizontal error of the head pose angle δ x and vertical error δ y The calculation formula is:

[0006] δ x =γ x -(θ h -180°)

[0007] δ y =θ v +γ y

[0008] θ h θ represents the horizontal angle of the center point of the camera's shooting direction. v γ represents the vertical angle of the center point of the camera's shooting direction. x γ represents the actual horizontal angle difference between target point m and center point o in the captured image. yIndicates the actual angular difference in the vertical direction between the target point m and the center point o; the target point m is the center point of the face to be recognized on the captured image.

[0009] Preferably:

[0010] When x < w / 2, then: γ x = a x - β x

[0011] When x ≥ w / 2, then: γ x = a x + β x

[0012] γ y = a y + β y

[0013]

[0014]

[0015] w and h respectively represent the length and height of the circumscribed rectangle of the captured image, (x, y) and (w / 2, h / 2) respectively represent the coordinates of the target point m and the center point o on the captured image, a x represents the horizontal angular error from the target point m to the center point o on the captured image, a y represents the vertical angular error from the target point m to the center point o on the captured image;

[0016] β x represents the error compensation of the target point m(x, y) in the horizontal direction, β y represents the error compensation of the target point m(x, y) in the vertical direction;

[0017] k h represents the error value between the camera field angle and the captured image field angle in the horizontal direction; k v represents the error value between the camera field angle and the captured image field angle in the vertical direction; k dh represents the error component of the captured image diagonal field angle in the horizontal direction; k dv represents the error component of the captured image diagonal field angle in the vertical direction.

[0018] Preferably:

[0019]

[0020]

[0021] where, fov h represents the horizontal field angle of the captured image, fovv This indicates the vertical field of view of the captured image.

[0022] Preferred:

[0023] k h =(FOV) h -fov h ) / 2

[0024] k v =(FOV) v -fov v ) / 2

[0025] k dh =(fov) dh -fov h ) / 2-k h

[0026] k dv =k v -(FOV v -fov dv ) / 2

[0027] Among them, fov h fov represents the horizontal field of view of the captured image. v fov represents the vertical field of view of the captured image. dh fov represents the horizontal component of the diagonal field of view of the captured image. dv FOV represents the vertical component of the diagonal field of view of a captured image. h FOV represents the horizontal field of view of the camera capturing images. v This indicates the vertical field of view of the camera that captured the image.

[0028] The preferred method for correcting head posture angle is as follows:

[0029]

[0030]

[0031] This indicates the corrected head attitude yaw angle, i.e., the horizontal rotation angle; This indicates the corrected head posture pitch angle, i.e., the vertical rotation angle; f h f represents the head pose yaw angle obtained by analyzing the face in the captured image using a head pose algorithm. v This represents the head pose pitch angle obtained by analyzing the face in the captured image using a head pose algorithm.

[0032] Preferably, it includes the following steps:

[0033] S1. Obtain the horizontal field of view (fov) of the captured image. h Vertical field of view fov v The horizontal component of the diagonal field of view, fov dh The vertical component of the field of view angle in the diagonal direction, fov dv ; Obtain the horizontal field of view (FOV) of the camera corresponding to the captured image. h and vertical field of view (FOV) v ;

[0034] S2. Analyze the captured image using a head pose algorithm to obtain the head pose yaw angle f of the face to be identified. h and head posture pitch angle f v Obtain the coordinates of the target point m and the center point o on the captured image, and calculate the horizontal error δ of the head posture rotation angle. x and vertical error δ y ;

[0035] S3, combined with f h f v δ x and δ y Calculate the corrected head posture angle.

[0036] Preferably, the target point is the center point of the bounded rectangle or regular polygon of the face to be identified in the captured image.

[0037] Preferably, the method for obtaining the head pose angle through the head pose algorithm includes the following steps:

[0038] SA1. Construct the neural network and label the samples; the labeled samples are captured images, and the labels of the labeled samples are the head pose angles of each face in the captured images. The head pose angles include the head pose yaw angle f. h and head posture pitch angle f v The input to the neural network is the captured image, and the output of the neural network is the head pose angle of each face in the captured image.

[0039] SA2, train the neural network by learning from labeled samples, and use the trained neural network as a head pose recognition model.

[0040] SA3. Input the target image into the head pose recognition model to obtain the head pose angle of each face in the target image output by the head pose recognition model.

[0041] This invention also proposes a head posture correction system based on camera distortion compensation, providing a carrier for the above method; the system includes a memory storing a computer program, which, when executed, is used to implement the head posture correction method based on camera distortion compensation.

[0042] Preferably, the system further includes a processor connected to a memory, the processor being used to execute the computer program to implement the head pose correction method based on camera distortion compensation.

[0043] The advantages of this invention are:

[0044] (1) The head posture correction method based on camera distortion compensation proposed in this invention takes the center point of the face to be identified in the captured image as the target point, calculates the head posture rotation angle error according to the relative position of the target point and the center point of the captured image, realizes the individual calculation of error for each face to be identified, and corrects the error of each face to be identified according to its position in the captured image. The deflection angle of the face to be identified in the camera monitoring screen can be accurately calculated. Based on this, a more accurate head posture recognition result can be obtained when performing head posture analysis.

[0045] (2) The present invention fully considers the inconsistency between the camera field of view and the field of view of the captured image. First, the position of the target point is compensated for the error. Then, the angle difference between the target point and the center point is calculated based on the compensated target point. Then, the head posture angle is corrected, which ensures the accuracy of the error calculation and further ensures the accuracy of head posture recognition.

[0046] (3) The present invention also proposes a head posture correction system based on camera distortion compensation, which provides a carrier for the head posture correction method based on camera distortion compensation, and facilitates the promotion and application of the method. Attached Figure Description

[0047] Figure 1 This is a flowchart of a head pose correction method based on camera distortion compensation;

[0048] Figure 2 Here is a flowchart of the head pose algorithm;

[0049] Figure 3 Images taken for an example;

[0050] Figure 4 To show the frontal image of target 1 in the embodiment. Detailed Implementation

[0051] Existing head pose algorithms are used to analyze captured images to obtain the head pose angle of the face to be identified in the image. The head pose angle includes the head pose yaw angle f.h and head posture pitch angle f v Existing head pose algorithms only consider the captured image when analyzing head pose angles, without taking into account camera distortion, thus resulting in errors in the final head pose angle obtained.

[0052] This embodiment proposes a head pose correction method based on camera distortion compensation. It considers the impact of camera distortion on the captured image and calculates the horizontal error δ of the head pose rotation angle caused by camera distortion. x and vertical error δ y Combined with the horizontal error δ of the head posture rotation angle x and vertical error δ y The head pose angle obtained through the head pose algorithm is corrected to ensure the accuracy of head pose angle recognition.

[0053] Reference Figure 1 The head pose correction method based on camera distortion compensation in this embodiment specifically includes the following steps S1-S3.

[0054] S1. Obtain the horizontal field of view (fov) of the captured image. h Vertical field of view fov v The horizontal component of the diagonal field of view, fov dh The vertical component of the field of view angle in the diagonal direction, fov dv ; Obtain the horizontal field of view (FOV) of the camera corresponding to the captured image. h and vertical field of view (FOV) v .

[0055] S2. Analyze the captured image using a head pose algorithm to obtain the head pose yaw angle f of the face to be identified. h and head posture pitch angle f v Obtain the coordinates of the target point m and the center point o on the captured image, and calculate the horizontal error δ of the head posture rotation angle. x and vertical error δ y The target point is the center point of the face to be identified in the captured image. Specifically, the center point of the bounding rectangle or regular polygon of the face to be identified in the captured image can be selected as the target point.

[0056] S3, combined with f h f v δ x and δ y Calculate the corrected head attitude angle; the corrected head attitude angle includes the corrected head attitude yaw angle. and the corrected head posture pitch angle

[0057]

[0058]

[0059] Indicates the yaw angle of the corrected head pose, i.e., the rotation angle in the horizontal direction; Indicates the pitch angle of the corrected head pose, i.e., the rotation angle in the vertical direction; f h Indicates the yaw angle of the head pose obtained by parsing the face in the captured image through the head pose algorithm, f v Indicates the pitch angle of the head pose obtained by parsing the face in the captured image through the head pose algorithm.

[0060] Horizontal error δ of the head pose rotation angle x and vertical error δ y The calculation formula is:

[0061] δ x = γ x -(θ h -180°) (2-1)

[0062] δ y = θ v + γ y (2-2)

[0063] θ h Indicates the horizontal angle of the center point of the camera shooting direction, θ v Indicates the vertical angle of the center point of the camera shooting direction; γ x Indicates the actual horizontal angle difference between the target point m and the center point o on the captured image, γ y Indicates the actual vertical angle difference between the target point m and the center point o; the target point m is the center point of the face to be recognized on the captured image.

[0064] When x < w / 2, then: γ x = a x - β x

[0065] When x ≥ w / 2, then: γ x = a x + β x

[0066] γ y = a y + β y

[0067] a x Indicates the horizontal angle error between the target point m and the center point o on the captured image, a y Indicates the vertical angle error between the target point m and the center point o on the captured image; βx β represents the error compensation for target point m in the horizontal direction. y This represents the error compensation for target point m in the vertical direction.

[0068]

[0069]

[0070] w and h represent the length and height of the bounding rectangle of the captured image, respectively; (x, y) and (w / 2, h / 2) represent the coordinates of the target point m and the center point o on the captured image, respectively; fov h fov represents the horizontal field of view of the captured image. v This indicates the vertical field of view of the captured image.

[0071]

[0072]

[0073] k h This represents the horizontal error between the camera's field of view and the field of view of the captured image; k v This represents the error value in the vertical direction between the camera's field of view and the field of view of the captured image; k dh k represents the horizontal error component of the field of view in the diagonal direction of the captured image. dv This represents the error component of the field of view angle in the vertical direction of the captured image.

[0074] k h =(FOV) h -fov h ) / 2

[0075] k v =(FOV) v -fov v ) / 2

[0076] k dh =(fov) dh -fov h ) / 2-k h

[0077] k dv =k v -(FOV v -fov dv ) / 2

[0078] fov h fov represents the horizontal field of view of the captured image. v fov represents the vertical field of view of the captured image. dhfov represents the horizontal component of the diagonal field of view of the captured image. dv FOV represents the vertical component of the diagonal field of view of a captured image. h FOV represents the horizontal field of view of the camera capturing images. v FOV represents the vertical field of view of the camera capturing images. h and FOV v These are inherent parameters of the camera and can be read from the camera's instruction manual.

[0079] Head pose algorithm

[0080] There are many existing head pose algorithms, among which neural networks are commonly used to implement them. In this embodiment, a head pose recognition model is obtained through machine learning, and the captured images are analyzed using this model to obtain the head pose angle of each face.

[0081] Reference Figure 2 In this embodiment, the steps for implementing the head pose algorithm are as follows.

[0082] SA1. Construct a neural network and label samples; the labeled samples are captured images, and the labels of the labeled samples are the head pose angles of each face in the captured images. The input of the neural network is the captured images, and the output of the neural network is the head pose angles of each face in the captured images.

[0083] SA2, train the neural network by learning from labeled samples, and use the trained neural network as a head pose recognition model.

[0084] SA3. Input the target image into the head pose recognition model to obtain the head pose rotation angle of each face in the target image output by the head pose recognition model, that is, the head pose yaw angle f of the face. h and head posture pitch angle f v .

[0085] In this embodiment, the labels on the labeled samples are manually labeled.

[0086] Specifically, in SA1, multiple labeled samples are constructed, which are differentiated into training samples and test samples. In SA2, the neural network is iterated multiple times. Each time, the neural network learns a specified number of training samples to update the network parameters. Then, the updated neural network calculates the head pose angle of each face in a set number of test samples as the model label of the test samples. The model label accuracy of the neural network is calculated based on the difference between the model label and the label of the test samples. If the model label accuracy is lower than the set value, the neural network learns new training samples. In this way, through parameter iteration, the neural network is used as the head pose recognition model until the model label accuracy of the neural network reaches the set value.

[0087] Example

[0088] In this embodiment, the target point is the center point of the bounding rectangle of the face to be identified in the captured image.

[0089] In this embodiment, head pose analysis and correction are performed on the two faces in the captured image. The correction results are shown in Tables 1-3 below:

[0090] Table 1: Camera Parameter Statistics Table

[0091] <![CDATA[FOV h ]]> <![CDATA[FOV v ]]> <![CDATA[θ h ]]> <![CDATA[θ v ]]> Camera 98 55 180 0

[0092] Table 2: Statistics of Captured Images

[0093]

[0094]

[0095] Table 3: Head Posture Data Statistics Table

[0096]

[0097] In the above table, This indicates the actual value of the head attitude yaw angle; This indicates the actual value of the head tilt angle; a x 'Indicates the horizontal angle of the face to be identified in the captured image showing the face to be identified from the front; a y 'Indicates the vertical angle of the face to be identified in the captured image showing the face to be identified from the front.

[0098] In Tables 1-3, the angle unit is °. The image length 1638 and width 987 means that there are 1638 pixels in the length direction and 987 pixels in the width direction of the image.

[0099] In this embodiment, Figure 3The horizontal and vertical angles of the center point of the camera's shooting direction corresponding to the captured image shown are 180° and 0°, respectively; according to a x and a y By correcting the horizontal and vertical angles of target 1 and target 2 in the captured image, we can obtain:

[0100] The corrected horizontal angle corresponding to target 1 is 180 - 25.15 - 4.1 = 150.75, which is close to a. x =151.1; The corrected vertical angle corresponding to target 1 is 0 + 13.27 + 0.56 = 13.83, which is close to a. y =13.8;

[0101] The corrected horizontal angle corresponding to target 2 is 180 + 9.9 + 0.49 = 190.39, which is close to a. x =190.2; the corrected vertical angle corresponding to target 2 is 0 + 5.64 + 0.16 = 5.8, which equals a y =5.8;

[0102] As can be seen, the present invention achieves the correction of the angle of a face at different positions in the captured image.

[0103] At the same time, combined and The comparison and and The comparison shows that the errors of the horizontal head posture rotation angles of target 1 and target 2 obtained by the present invention are -0.43 and 0.3 respectively, and the absolute values ​​are both less than 0.5; the errors of the vertical head posture rotation of target 1 and target 2 are -1.03 and -0.5 respectively; which shows that the head posture rotation obtained by the present invention is accurate and reliable.

[0104] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A head pose correction method based on camera distortion compensation, characterized in that, Combined with head posture angle horizontal error δ x and vertical error δ y Correct the head pose angle obtained through the head pose algorithm; horizontal error of head pose angle. δ x and vertical error δ y The calculation formula is: δ x =γ x -(θ h -180°) δ y =θ v +γ y θ h This indicates the horizontal angle of the center point of the camera's shooting direction. θ v Indicates the vertical angle of the center point of the camera's shooting direction; γ x Indicates the target point on the captured image m With the center point o The actual angle difference in the horizontal direction, γ y Indicates the target point m With the center point o The actual angle difference in the vertical direction; Target point m The center point of the face to be identified in the captured image; when x <w / 2, then: γ x =a x -β x when x≧w / 2, then: γ x =a x +β x γ y =a y +β y w and h These represent the length and height of the bounding rectangle of the captured image, respectively. (x,y) and (w / 2,h / 2) Representing the target points respectively m and center point o Coordinates on the captured image, a x Indicates the target point on the captured image m To the center point o Horizontal angle error, a y Indicates the target point on the captured image m To the center point o The vertical angle error; β x Indicates the target point m(x,y) Error compensation in the horizontal direction, β y Indicates the target point m(x,y) Error compensation in the vertical direction; k h This indicates the error value in the horizontal direction between the camera's field of view and the field of view of the captured image; k v This indicates the error value in the vertical direction between the camera's field of view and the field of view of the captured image; k dh This represents the error component of the field of view in the horizontal direction of the captured image. k dv This represents the error component of the field of view in the vertical direction of the captured image. in, fov h Indicates the horizontal field of view of the captured image. fov v Indicates the vertical field of view of the captured image; in, fov h Indicates the horizontal field of view of the captured image. fov v Indicates the vertical field of view of the captured image. fov dh This represents the horizontal component of the diagonal field of view of the captured image. fov dv This represents the vertical component of the diagonal field of view of the captured image. FOV h This indicates the horizontal field of view of the camera that captures the images. FOV v This indicates the vertical field of view of the camera that captured the image.

2. The head pose correction method based on camera distortion compensation as described in claim 1, characterized in that, The method for correcting head posture angle is as follows: x =f h -δ x y =f v +δ y x This indicates the corrected head attitude yaw angle, i.e., the horizontal rotation angle; y This indicates the corrected head posture pitch angle, which is the vertical rotation angle. f h This represents the head pose yaw angle obtained by analyzing the face in the captured image using a head pose algorithm. f v This represents the head pose pitch angle obtained by analyzing the face in the captured image using a head pose algorithm.

3. The head pose correction method based on camera distortion compensation as described in claim 1, characterized in that, Includes the following steps: S1. Obtain the horizontal field of view of the captured image. fov h Vertical field of view fov v The horizontal component of the diagonal field of view. fov dh The vertical component of the field of view in the diagonal direction fov dv ; Obtain the horizontal field of view of the camera corresponding to the captured image. FOV h and vertical field of view FOV v ; S2. Analyze the captured image using a head pose algorithm to obtain the head pose yaw angle of the face to be identified. f h and head posture pitch angle f v ; Obtain target points on the captured image m and center point o The coordinates are determined, and the horizontal error of the head posture rotation angle is calculated. δ x and vertical error δ y ; S3, Combination f h , f v , δ x and δ y Calculate the corrected head posture angle.

4. The head pose correction method based on camera distortion compensation as described in claim 1, characterized in that, The target point is the center point of the bounded rectangle or regular polygon of the face to be identified in the captured image.

5. The head pose correction method based on camera distortion compensation as described in claim 1, characterized in that, The head pose angle obtained through the head pose algorithm includes the following steps: SA1. Construct the neural network and label the samples; the labeled samples are captured images, and the labels of the labeled samples are the head pose angles of each face in the captured images, including the head pose yaw angle. f h and head posture pitch angle f v The input to the neural network is the captured image, and the output of the neural network is the head pose angle of each face in the captured image. SA2, train the neural network by learning from labeled samples, and use the trained neural network as a head pose recognition model. SA3. Input the target image into the head pose recognition model to obtain the head pose angle of each face in the target image output by the head pose recognition model.

6. A head pose correction system based on camera distortion compensation, characterized in that, It includes a memory storing a computer program, which, when executed, is used to implement the head pose correction method based on camera distortion compensation as described in any one of claims 1-5.

7. The head pose correction system based on camera distortion compensation as described in claim 6, characterized in that, It also includes a processor connected to a memory, the processor being used to execute the computer program to implement the head pose correction method based on camera distortion compensation as described in any one of claims 1-5.

Images