Eyeball positioning method and device, anti-peeping control method and device and storage medium

Abstract

The invention discloses an eyeball positioning method, an anti-peeping control method and device and a storage medium. The eyeball positioning method comprises the following steps: acquiring image coordinates of a plurality of feature points, internal parameters of a camera and a human head model; determining a face posture angle of the target user in a camera coordinate system according to the image coordinates of the plurality of feature points, the human head model and internal parameters of the camera; and determining left eye camera coordinates and right eye camera coordinates of the target user according to the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera and the face posture angle of the target user in the camera coordinate system. Compared with a method for positioning the eyeball position of the target user according to an image shot by a camera. According to the eyeball positioning method provided by the invention, the influence of the face posture angle of the target user on the positioned eyeball position is considered, and the accuracy of the positioned eyeball position is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of eye positioning, and more specifically, to an eye positioning method, an anti-peeping control method, a device and a storage medium. Background Art

[0002] When a camera captures a face, the face pose angle may cause the face image information captured by the camera to be incomplete. For example, when the face turns left and right, the yaw angle of the face pose angle is relatively large, and the camera may only capture a side face image of the face. When the face nods up and down, the pitch angle of the face pose angle is relatively large, and the camera may only capture an image of the chin or the top of the head.

[0003] In the related art, the eye image information can be determined based on the face image information captured by the camera, and then the position of the eye can be located according to the eye image information. When the face pose angle causes the face image information captured by the camera to be incomplete, the eye image information determined according to the face image information is also incomplete, resulting in inaccurate positioning of the eye position. Summary of the Invention

[0004] Embodiments of the present invention provide an eye positioning method, an anti-peeping control method, a device and a storage medium.

[0005] The eye positioning method provided by the embodiments of the present invention includes: obtaining the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model, where the image coordinates of the multiple feature points are determined based on the face image of the target user captured by the camera, and the image coordinates of the multiple feature points include the left eye image coordinates and the right eye image coordinates of the target user; determining the face pose angle of the target user in the head coordinate system according to the image coordinates of the multiple feature points and the human head model; determining the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of the multiple feature points, the internal parameters of the camera, and the human head model, where the camera coordinate system is determined based on the camera; determining the face pose angle of the target user in the camera coordinate system according to the transformation matrix and the face pose angle of the target user in the head coordinate system; and determining the left eye camera coordinates and the right eye camera coordinates of the target user according to the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera, and the face pose angle of the target user in the camera coordinate system.

[0006] When positioning the eyes of the target user, first determine the face pose angle of the target user, and then locate the position of the eyes of the target user according to the face pose angle of the target user, the left eye image coordinates, the right eye image coordinates captured by the camera, and the internal parameters of the camera.

[0007] Compared with the method of directly locating the eye position of the target user based on the image captured by the camera, the eye positioning method of the embodiment of the present invention takes into account the influence of the face pose angle of the target user on the located eye position, and improves the accuracy of the located eye position.

[0008] In some embodiments, the image coordinates of the multiple feature points further include the image coordinates of the tip of the nose of the target user, the image coordinates of the left corner of the mouth of the target user, and the image coordinates of the right corner of the mouth.

[0009] When the camera can identify a human face through a face recognition algorithm, the image coordinates of multiple feature points can be obtained without performing additional steps to obtain the image coordinates of multiple feature points.

[0010] In some embodiments, the face pose angle includes the yaw angle, pitch angle, and roll angle of the face.

[0011] Yaw (yaw angle) corresponds to the left and right rotation of the face, pitch (pitch angle) corresponds to the up and down rotation of the face, roll (roll angle) corresponds to the side rotation of the face, and the yaw angle, pitch angle, and roll angle of the face of the target user can comprehensively represent the facial orientation of the target user in three-dimensional space.

[0012] In some embodiments, the determining the face pose angle of the target user in the head coordinate system according to the image coordinates of the multiple feature points and the human head model includes: determining the coordinates of the multiple feature points in the head coordinate system according to the image coordinates of the multiple feature points and the human head model; determining the face pose angle of the target user in the head coordinate system according to the coordinates of the multiple feature points in the head coordinate system, the image coordinates of the multiple feature points, and the internal parameters of the camera.

[0013] The Perspective-n-Point (PnP) algorithm can be used to solve for the face pose angle of the target user in the head coordinate system according to the three-dimensional coordinates of the given multiple feature points in the head coordinate system, the two-dimensional image coordinates corresponding to the multiple feature points, and the internal parameters of the camera.

[0014] In some embodiments, determining the transformation matrix between the head coordinate system and the camera coordinate system based on the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model includes: determining the coordinates of the multiple feature points in the camera coordinate system based on the image coordinates of the multiple feature points and the internal parameters of the camera; determining the coordinates of the multiple feature points in the head coordinate system based on the image coordinates of the multiple feature points and the human head model; and determining the transformation matrix between the head coordinate system and the camera coordinate system based on the coordinates of the multiple feature points in the camera coordinate system and the coordinates of the multiple feature points in the head coordinate system.

[0015] The transformation matrix between the coordinates of multiple feature points in the camera coordinate system and the coordinates of multiple feature points in the head coordinate system is the transformation matrix between the head coordinate system and the camera coordinate system.

[0016] In some embodiments, determining the transformation matrix between the head coordinate system and the camera coordinate system based on the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model includes: determining the gaze point image coordinates of the target user based on the weighted average of the left eye image coordinates and the right eye image coordinates of the target user; and determining the transformation matrix between the head coordinate system and the camera coordinate system based on the gaze point image coordinates, the internal parameters of the camera, and the human head model.

[0017] When the camera captures the face of the target user, eye movement tracking can be performed on the target user. What eye movement tracking research focuses on most is the change of the gaze point of the target user. The transformation matrix between the head coordinate system and the camera coordinate system can be determined based on the position of the gaze point of the target user.

[0018] In some embodiments, determining the left eye camera coordinates and the right eye camera coordinates based on the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera, and the face pose angle of the target user in the camera coordinate system includes: determining the pupil distance of the target user based on the difference between the left eye image coordinates and the right eye image coordinates; and determining the left eye camera coordinates and the right eye camera coordinates based on the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera, the face pose angle of the target user in the camera coordinate system, and the pupil distance of the target user.

[0019] Determining the left eye camera coordinates and the right eye camera coordinates based on the head pose angle and the pupil distance in the camera coordinate system makes the finally calculated left eye camera coordinates and right eye camera coordinates more accurate, that is, the located eye positions are more accurate.

[0020] In some embodiments, after determining the left-eye camera coordinates and the right-eye camera coordinates, the eye positioning method further includes: obtaining the external parameters of the camera; determining the left-eye world coordinates of the target user according to the external parameters of the camera and the left-eye camera coordinates; and determining the right-eye world coordinates of the target user according to the external parameters of the camera and the right-eye camera coordinates.

[0021] The world coordinates of the eye can be obtained based on the external parameters of the camera and the camera coordinates of the eye, which is beneficial to positioning the eye position of the target user.

[0022] An embodiment of the present invention provides an anti-peeping control method for a display device. The anti-peeping control method includes: determining the left-eye camera coordinates and the right-eye camera coordinates of the target user according to the eye positioning method of the above embodiment; determining whether the target user is within the visible range of the display device according to the left-eye camera coordinates and the right-eye camera coordinates; and controlling the display device to trigger an anti-peeping protection function when the target user is within the visible range of the display device and the target user is not an authorized user of the display device.

[0023] In some embodiments, the determining whether the target user is within the visible range of the display device according to the left-eye camera coordinates and the right-eye camera coordinates includes:

[0024] determining the distances and relative angles between the left eye and the right eye of the target user and the display device according to the left-eye camera coordinates and the right-eye camera coordinates;

[0025] Determining that the target user is within the visible range of the display device when the distances between the left eye and the right eye of the target user and the display device are less than the visible distance of the display device and the relative angles between the left eye and the right eye of the target user and the display device are less than the visible angle of the display device.

[0026] The eye positioning device provided by the embodiment of the present invention includes an acquisition module and a calculation module. The acquisition module is used to acquire the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model. The image coordinates of the multiple feature points are determined based on the face image of the target user captured by the camera. The image coordinates of the multiple feature points include the left eye image coordinates and the right eye image coordinates of the target user. The calculation module is used to determine the face pose angle of the target user in the head coordinate system according to the image coordinates of the multiple feature points and the human head model. The calculation module is further used to determine the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of the multiple feature points, the internal parameters of the camera, and the human head model. The camera coordinate system is determined based on the camera. The calculation module is further used to determine the face pose angle of the target user in the camera coordinate system according to the transformation matrix and the face pose angle of the target user in the head coordinate system. The calculation module is further used to determine the left eye camera coordinates and the right eye camera coordinates of the target user according to the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera, and the face pose angle of the target user in the camera coordinate system.

[0027] The display device of the eye positioning device provided by the embodiment of the present invention includes a camera, an eye positioning device, a determination module, and an anti-peeping protection module. The camera is used to capture the face image of the target user. The eye positioning device is used to determine the left eye camera coordinates and the right eye camera coordinates of the target user. The determination module is used to determine whether the target user is within the visible range of the display device according to the left eye camera coordinates and the right eye camera coordinates of the target user. The determination module is further used to determine whether the target user is an authorized user of the display device. The anti-peeping protection module is used to trigger the anti-peeping protection function of the display device when the target user is within the visible range of the display device and the target user is not an authorized user of the display device.

[0028] The embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium is used to store a computer program, and when the computer-readable storage medium is executed, it implements the eye positioning method or the anti-peeping control method of any of the above embodiments.

[0029] Embodiments of the present invention provide an eyeball positioning method, an anti-peeping control method, an eyeball positioning device, a display device, and a storage medium. The eyeball positioning method includes: obtaining the image coordinates of multiple feature points, the internal parameters of a camera, and a human head model, where the feature points are the feature points on the human head model, and the image coordinates of the feature points are determined based on the face image of the target user captured by the camera. The image coordinates of the multiple feature points include the left eye image coordinates and the right eye image coordinates of the target user; determining the face pose angle of the target user in the head coordinate system according to the image coordinates of the multiple feature points and the human head model; determining the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of the multiple feature points, the internal parameters of the camera, and the human head model, where the camera coordinate system is determined based on the camera; determining the face pose angle of the target user in the camera coordinate system according to the transformation matrix and the face pose angle of the target user in the head coordinate system; and determining the left eye camera coordinates and the right eye camera coordinates of the target user according to the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera, and the face pose angle of the target user in the camera coordinate system.

[0030] When positioning the eyeballs of the target user, first determine the face pose angle of the target user, and then locate the eyeball position of the target user according to the face pose angle of the target user, the left eye image coordinates, the right eye image coordinates captured by the camera, and the internal parameters of the camera. Compared with the method of directly positioning the eyeball position of the target user according to the image captured by the camera, the eyeball positioning method of the embodiments of the present invention takes into account the influence of the face pose angle of the target user on the positioned eyeball position, and improves the accuracy of the positioned eyeball position.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0033] Figure 1 is a flowchart of the eyeball positioning method according to the embodiments of the present invention;

[0034] Figure 2 is a schematic diagram of multiple feature points and the head pose angle according to the embodiments of the present invention;

[0035] Figure 3 is a schematic diagram of the distribution of multiple feature points on the human head model according to the embodiments of the present invention;

[0036] Figure 4 is a schematic diagram of the camera coordinate system and the head coordinate system according to the embodiments of the present invention;

[0037] Figure 5 are schematic diagrams of the eye positioning device and the display device according to the embodiments of the present invention;

[0038] Figure 6 is a schematic diagram of the visible range of the display device according to the embodiments of the present invention;

[0039] Figure 7 is a schematic diagram of the process of enabling the anti-peeping mode according to the embodiments of the present invention;

[0040] Figure 8 is a schematic diagram of the process after enabling the anti-peeping mode according to the embodiments of the present invention;

[0041] Figure 9 is a schematic diagram of the process of the display device identifying a human face according to the embodiments of the present invention;

[0042] Figure 10 is a schematic diagram of the display device performing human face quality assessment according to the embodiments of the present invention. Specific embodiments

[0043] The following details the embodiments of the present invention. The embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0044] When the camera captures a human face, the face pose angle may cause the face image information collected by the camera to be incomplete. For example, when the face turns left and right, the yaw angle (heading angle) of the face pose angle is relatively large, and the camera may only capture a side face image of the human face. When the face nods or looks up and down, the pitch angle (pitch angle) of the face pose angle is relatively large, and the camera may only capture an image of the chin or the top of the head.

[0045] In the related art, the eye image information can be determined based on the face image information collected by the camera, and then the position of the eye can be located based on the eye image information. When the face pose angle causes the face image information collected by the camera to be incomplete, the eye image information determined based on the face image information is also incomplete, resulting in inaccurate positioning of the eye position.

[0046] Referring to Figure 1 , the embodiments of the present invention provide an eye positioning method. In some embodiments, the eye positioning method includes:

[0047] Step S10: Obtain the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model. The image coordinates of the multiple feature points are determined based on the facial image of the target user captured by the camera. The image coordinates of the multiple feature points include the left eye image coordinates and the right eye image coordinates of the target user.

[0048] Step S20: Determine the face pose angle of the target user in the head coordinate system according to the image coordinates of the multiple feature points and the human head model.

[0049] Step S30: Determine the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of the multiple feature points, the internal parameters of the camera, and the human head model. The camera coordinate system is determined based on the camera.

[0050] Step S40: Determine the face pose angle of the target user in the camera coordinate system according to the transformation matrix and the face pose angle of the target user in the head coordinate system.

[0051] Step S50: Determine the left eye camera coordinates and the right eye camera coordinates of the target user according to the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera, and the face pose angle of the target user in the camera coordinate system.

[0052] When positioning the eyeballs of the target user, first determine the face pose angle of the target user, and then locate the eyeball position of the target user according to the face pose angle of the target user, the left eye image coordinates, the right eye image coordinates captured by the camera, and the internal parameters of the camera.

[0053] Compared with the method of directly positioning the eyeball position of the target user according to the image captured by the camera, the eyeball positioning method of the embodiment of the present invention takes into account the influence of the face pose angle of the target user on the positioned eyeball position, and improves the accuracy of the positioned eyeball position.

[0054] Specifically, referring to Figure 2 and Figure 3 , in step S10 and step S20, the multiple feature points can be multiple points on the face of the target user. The camera can obtain the facial image of the target user by photographing the target user. The facial image of the target user is a two-dimensional planar image, and the image coordinates are also two-dimensional coordinates. The image coordinates of the multiple feature points can be determined according to the facial image of the target user captured by the camera.

[0055] The process of the camera photographing the face of the target user is a process of converting three-dimensional space coordinates into two-dimensional planar coordinates. According to the principle of lens imaging, the three-dimensional space coordinates of multiple feature points on the face of the target user are presented in the camera and then projected onto the corresponding screen to obtain the corresponding two-dimensional planar coordinates.

[0056] The internal parameters of the camera can be the parameters for transforming camera coordinates to image coordinates. The camera takes the optical center as the center to obtain the world coordinates of multiple feature points on the face of the target user as camera coordinates, and the camera coordinates are three-dimensional coordinates. The camera coordinates are presented inside the camera and then projected onto the corresponding screen to obtain the corresponding image coordinates. The corresponding image coordinates can be obtained based on the internal parameters of the camera and the camera coordinates, or the corresponding camera coordinates can be obtained based on the internal parameters of the camera and the image coordinates.

[0057] The face pose angle of the target user represents the facial orientation of the target user in three-dimensional space. The multiple feature points can be multiple points on the face of the target user, so the face pose angle of the target user can be determined based on the three-dimensional space of the multiple feature points.

[0058] For different people, the differences in the distribution of multiple feature points on the face are not significant. The human head model can be a predetermined human head model. The distribution of multiple feature points on the human head model can estimate the distribution of multiple feature points on the human head of the target user to determine the face pose angle of the target user in the head coordinate system.

[0059] Refer to Figure 4 In steps S30 and S40, when the head of the target user is not in the center of the camera, there is a coordinate rotation transformation between the head coordinate system and the camera coordinate system, and it is necessary to calculate the rotation matrix between the head coordinate system and the camera coordinate system. It is necessary to first calculate and determine the transformation matrix between the head coordinate system and the camera coordinate system, and then convert the head pose angle in the head coordinate system to the head pose angle in the camera coordinate system.

[0060] The left-eye image coordinates and the right-eye image coordinates are the two-dimensional coordinates of two feature points on the face of the target user. The internal parameters of the camera can be the parameters for transforming camera coordinates to image coordinates. The left-eye camera coordinates and the right-eye camera coordinates can be obtained based on the left-eye image coordinates, the right-eye image coordinates, the internal parameters of the camera, and the head pose angle in the camera coordinate system.

[0061] Compared with directly determining the left-eye camera coordinates and the right-eye camera coordinates based on the left-eye image coordinates, the right-eye image coordinates, and the internal parameters of the camera, the eye positioning method of the embodiment of the present invention takes into account the influence of the head pose angle in the camera coordinate system, making the left-eye camera coordinates and the right-eye camera coordinates more accurate, that is, the positioned eye positions are more accurate.

[0062] Refer to Figure 2 And Figure 3 In some embodiments, the image coordinates of the multiple feature points further include the nose tip image coordinates of the target user, the left mouth corner image coordinates of the target user, and the right mouth corner image coordinates of the target user.

[0063] When the camera can identify a human face through a face recognition algorithm, the image coordinates of multiple feature points can be obtained without performing additional steps to obtain the image coordinates of multiple feature points.

[0064] Specifically, when the camera captures a face image of a target user, the camera can identify the human face through a face recognition algorithm. In related technologies, face recognition algorithms such as the Multi-task convolutional neural network (MTCNN) identify a human face by identifying key points such as the eyes, nose, and mouth of the human face.

[0065] In some embodiments, the face pose angles include the yaw angle, pitch angle, and roll angle of the human face.

[0066] Yaw (yaw angle) corresponds to the left-right rotation of the human face, pitch (pitch angle) corresponds to the up-down rotation of the human face, roll (roll angle) corresponds to the side rotation of the human face, and the yaw angle, pitch angle, and roll angle of the target user's human face can comprehensively represent the facial orientation of the target user in three-dimensional space.

[0067] Specifically, there are three face pose angles, namely yaw (yaw angle), pitch (pitch angle), and roll (roll angle). The pitch angle Pitch rotated on the X-axis in the head coordinate system is denoted as θ x , the yaw angle Yaw rotated on the Y-axis in the head coordinate system is denoted as θ y , and the roll angle Roll rotated on the Z-axis in the head coordinate system is denoted as θ z .

[0068] In some embodiments, step S20, determining the face pose angle of the target user in the head coordinate system according to the image coordinates of multiple feature points and the human head model, includes:

[0069] Step S21, determining the coordinates of multiple feature points in the head coordinate system according to the image coordinates of multiple feature points and the human head model;

[0070] Step S22, determining the face pose angle of the target user in the head coordinate system according to the coordinates of multiple feature points in the head coordinate system, the image coordinates of multiple feature points, and the internal parameters of the camera.

[0071] The Perspective-n-Point (PnP) algorithm can be used to solve the face pose angle of the target user in the head coordinate system according to the three-dimensional coordinates of multiple given feature points in the head coordinate system, the corresponding two-dimensional image coordinates of multiple feature points, and the internal parameters of the camera.

[0072] Specifically, in step S21, on the human head model, the positions of the tip of the nose, the left eye, the right eye, the left corner of the mouth, and the right corner of the mouth are determined. Therefore, the three-dimensional coordinates of the tip of the nose, the left eye, the right eye, the left corner of the mouth, and the right corner of the mouth in the head coordinate system are also determined. On the human head model, the positions of the left eye and the right eye are symmetrical, and the positions of the left corner of the mouth and the right corner of the mouth are also symmetrical.

[0073] In the head coordinate system, the coordinates of the tip of the nose can be (0, 0, 0), the coordinates of the chin can be (0, -330, -65), the coordinates of the left eye can be (-225, 170, -135), the coordinates of the right eye can be (225, 170, -135), the coordinates of the left corner of the mouth can be (-150, -150, -125), and the coordinates of the right corner of the mouth can be (150, -150, -125).

[0074] In step S22, by using the Perspective-n-Point (PnP) algorithm, the corresponding face pose rotation matrix R head1 , and based on R head1 the corresponding yaw, pitch, and roll angles are obtained. The specific calculation formulas are as follows:

[0075]

[0076] R head1 is a 3*3 matrix, R head1 [0,0] is the number in the first row and first column of the matrix, R head1 [0,1] is the number in the first row and second column of the matrix, R head1 [1,0] is the number in the second row and first column of the matrix.

[0077] If sy ≥ 1e-6:

[0078]

[0079] Otherwise:

[0080]

[0081] R[2,1] is the number in the third row and second column of the R head1 matrix, R[2,2] is the number in the third row and third column of the R head1 matrix, R[2,0] is the number in the third row and first column of the R head1 matrix, R[1,0] is the number in the second row and first column of the R head1 matrix, R[0,0] is the number in the first row and first column of the R head1 matrix, R[1,2] is the R head1The number in the second row and third column of the matrix, and R[1,1] is R head1 The number in the second row and second column of the matrix.

[0082] Refer to Figure 4 , in some embodiments, step S30, determining the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model, includes:

[0083] Step S310, determining the coordinates of multiple feature points in the camera coordinate system according to the image coordinates of the multiple feature points and the internal parameters of the camera;

[0084] Step S311, determining the coordinates of multiple feature points in the head coordinate system according to the image coordinates of the multiple feature points and the human head model;

[0085] Step S312, determining the transformation matrix between the head coordinate system and the camera coordinate system according to the coordinates of the multiple feature points in the camera coordinate system and the coordinates of the multiple feature points in the head coordinate system.

[0086] The transformation matrix between the coordinates of multiple feature points in the camera coordinate system and the coordinates of multiple feature points in the head coordinate system is the transformation matrix between the head coordinate system and the camera coordinate system.

[0087] Specifically, in step S310, (u1, v1) may be the image coordinates of the feature point, may be the internal parameter matrix of the camera, and according to the image coordinates of the feature point and the internal parameter matrix of the camera, the coordinates of a specific point in the camera

[0088] ″′

[0089] coordinate system can be obtained. (x1, y1, z1) may be the coordinates of the feature point in the camera coordinate system, and the calculation formula is:

[0090]

[0091] In step S311, on the human head model, the positions of the feature points are determined. Therefore, the three-dimensional coordinates of the feature points in the head coordinate

[0092] ″′

[0093] system are also determined. (x2, y2, z2) may be the coordinates of the feature point in the head coordinate system.

[0094] In step S312, the rotation matrix between the head coordinate system and the camera coordinate system can be denoted as R local, the transformation matrix between the coordinates of multiple feature points in the camera coordinate system and the coordinates of multiple feature points in the head coordinate system is the transformation matrix between the head coordinate system and the camera coordinate system, that is, Based on (x′1, y′1, z′1) and (x′2, y′2, z′2), R can be calculated and determined. local .

[0095] Refer to Figure 4 , in some embodiments, step S30, determining the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model, includes:

[0096] Step S320, determining the gaze point image coordinates of the target user according to the weighted average of the left eye image coordinates and the right eye image coordinates of the target user;

[0097] Step S321, determining the transformation matrix between the head coordinate system and the camera coordinate system according to the gaze point image coordinates, the internal parameters of the camera, and the human head model.

[0098] When the camera captures the face of the target user, eye movement tracking can be performed on the target user. The most concerned aspect of eye movement tracking research is the change of the gaze point of the target user. The transformation matrix between the head coordinate system and the camera coordinate system can be determined according to the position of the gaze point of the target user.

[0099] Specifically, in step S320, the image coordinates of the left eye and the right eye can be denoted as (u l , v l ), (u r , v r ). Calculate the image coordinates of the center point of the left and right eyes, that is, the gaze point image coordinates of the target user:

[0100]

[0101] In step S321, the rotation matrix between the head coordinate system and the camera coordinate system can be denoted as R local , R local = [C1 C2 C3].

[0102] Calculate the homogeneous coordinates of the gaze point in the camera coordinate system. The calculation formula is:

[0103]

[0104]

[0105] Calculate the column vectors C1, C2, and C3:

[0106]

[0107]

[0108] C1 = cross(C2, C3), where C1 is the cross product of column vectors C2 and C3.

[0109] In step S40, the face pose angle R of the target user in the camera coordinate system head2 = R local R head1 .

[0110] In some embodiments, step S50, determining the left-eye camera coordinates and the right-eye camera coordinates according to the left-eye image coordinates, the right-eye image coordinates, the internal parameters of the camera, and the face pose angle of the target user in the camera coordinate system, includes:

[0111] Step S51, determining the pupil distance of the target user according to the difference between the left-eye image coordinates and the right-eye image coordinates;

[0112] Step S52, determining the left-eye camera coordinates and the right-eye camera coordinates according to the left-eye image coordinates, the right-eye image coordinates, the internal parameters of the camera, the face pose angle of the target user in the camera coordinate system, and the pupil distance of the target user.

[0113] Determining the left-eye camera coordinates and the right-eye camera coordinates according to the head pose angle and the pupil distance in the camera coordinate system makes the finally calculated left-eye camera coordinates and right-eye camera coordinates more accurate, that is, the located eyeball position is more accurate.

[0114] Specifically, in step S51, the image coordinates (u l , v l ) and (u r , v r ) of the left eye and the right eye can determine the pupil distance d of the target user, and d can be

[0115] In step S52, the left-eye camera coordinates can be denoted as (x l , y l , z l ), and the right-eye camera coordinates can be denoted as (x r , y r , z r ).

[0116] The face pose angle R of the target user in the camera coordinate system head2 = R local R head1 , R head2 can be denoted as:

[0117]

[0118] Furthermore, referring to R head1 obtain θ x , θ y and θ z . For the calculation process of R head1 the corresponding θ x2 , θ y2 and θ z2 can be obtained.

[0119] C x can be denoted as cos(θ x2 ), C y can be denoted as cos(θ y2 ), C z can be denoted as cos(θ z2 ), S x can be denoted as sin(θ x2 ), S y can be denoted as sin(θ y2 ), S z can be denoted as sin(θ z2 ).

[0120]

[0121]

[0122]

[0123]

[0124] After determining the left-eye camera coordinates and the right-eye camera coordinates, the eye positioning method further includes:

[0125] Obtain the external parameters of the camera;

[0126] Determine the left-eye world coordinates of the target user according to the external parameters of the camera and the left-eye camera coordinates;

[0127] Determine the right-eye world coordinates of the target user according to the external parameters of the camera and the right-eye camera coordinates.

[0128] According to the external parameters of the camera and the camera coordinates of the eyeball, the world coordinates of the eyeball can be obtained, which is beneficial to positioning the position of the target user's eyeball.

[0129] Specifically, the external parameters of the camera are parameters in the world coordinate system, such as the position and rotation direction of the camera. The world coordinate is a three-dimensional coordinate system in the real world. When the camera captures the face image of the target user, due to changes such as the rotation and movement of the camera, there is a coordinate rotation transformation between the camera coordinate system and the world coordinate system, and the external parameter matrix of the camera can be the transformation matrix between the camera coordinate system and the world coordinate system.

[0130] Based on the left-eye camera coordinates (x l , y l , z l ) and the extrinsic parameters of the camera, the left-eye world coordinates can be obtained. For the right-eye camera coordinates (x r , y r , z r ) and the extrinsic parameters of the camera, the right-eye world coordinates can be obtained.

[0131] Referring to Figure 5 , an embodiment of the present invention provides an eye positioning device 10 and a display device 100. The display device 100 includes the eye positioning device 10. The eye positioning device 10 includes an acquisition module 11 and a calculation module 12. The acquisition module 11 is configured to acquire the image coordinates of a plurality of feature points, the intrinsic parameters of the camera 20, and the human head model. The image coordinates of the plurality of feature points are determined based on the face image of the target user captured by the camera 20. The image coordinates of the plurality of feature points include the left-eye image coordinates and the right-eye image coordinates of the target user. The calculation module 12 is configured to determine the face pose angle of the target user in the head coordinate system according to the image coordinates of the plurality of feature points and the human head model. The calculation module 12 is further configured to determine the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of the plurality of feature points, the intrinsic parameters of the camera 20, and the human head model. The camera coordinate system is determined based on the camera 20. The calculation module 12 is further configured to determine the face pose angle of the target user in the camera coordinate system according to the transformation matrix and the face pose angle of the target user in the head coordinate system. The calculation module 12 is further configured to determine the left-eye camera coordinates and the right-eye camera coordinates of the target user according to the left-eye image coordinates, the right-eye image coordinates, the intrinsic parameters of the camera 20, and the face pose angle of the target user in the camera coordinate system.

[0132] Specifically, the eye positioning device 10 can implement the eye positioning method of the above embodiment. The beneficial effects of the eye positioning device 10 include all the beneficial effects of the eye positioning method, which will not be elaborated here one by one.

[0133] An embodiment of the present invention provides an anti-peeping control method, which is used for the display device 100. The anti-peeping control method includes:

[0134] Step S100: Determine the left-eye camera coordinates and the right-eye camera coordinates of the target user according to the eye positioning method of the above embodiment;

[0135] Step S200: Determine whether the target user is within the visible range of the display device 100 according to the left-eye camera coordinates and the right-eye camera coordinates;

[0136] Step S300: When the target user is within the visible range of the display device 100 and the target user is not an authorized user of the display device 100, control the display device 100 to trigger the anti-peeping protection function.

[0137] The anti-peeping control method according to the embodiment of the present invention can improve the sensitivity of the anti-peeping protection function, and at the same time prevent the anti-peeping protection function of the display device 100 from being erroneously triggered as much as possible.

[0138] Specifically, referring to Figure 6 , the display device 100 has a corresponding visible range. When the target user is not within this visible range, it can be considered that the target user cannot see the display device 100. Even if the target user is not an authorized user of the display device 100, the display device 100 will not trigger the anti-peeping protection function.

[0139] Since the embodiment of the present invention can more accurately locate the eyes of the target user, it is possible to more accurately determine whether the target user is within the visible range of the display device 100, avoiding erroneously triggering the anti-peeping protection function of the display device 100.

[0140] In some embodiments, step S200: determining whether the target user is within the visible range of the display device 100 according to the left-eye camera coordinates and the right-eye camera coordinates includes:

[0141] Step S210: Determine the distances and relative angles between the left eye and the right eye of the target user and the display device 100 according to the left-eye camera coordinates and the right-eye camera coordinates;

[0142] Step S220: When the distances between the left eye and the right eye of the target user and the display device 100 are less than the visible distance of the display device 100 and the relative angles between the left eye and the right eye of the target user and the display device 100 are less than the visible angle of the display device 100, determine that the target user is within the visible range of the display device 100.

[0143] The display device 100 has a corresponding visible distance and visible angle. When the target user is not within this visible distance or visible angle, it can be considered that the target user cannot see the display device 100, preventing the anti-peeping protection function of the display device 100 from being erroneously triggered.

[0144] Specifically, the visible angle refers to the angle at which the user can clearly observe all the content on the screen from different directions. Taking the vertical normal line of the display device 100 as the standard, the visible part of the display device 100 is within the visible angle range to the left or right perpendicular to the normal line, and the light-shielding part of the display device 100 is outside the visible angle range to the left or right perpendicular to the normal line.

[0145] The target user can view the screen displayed by the display device 100 at the visible part position of the display device 100. A light-shielding structure can be provided on the display device 100, and the light-shielding structure can be a grating presenting grayish black. After arranging the position of the grating, the light emitted by the pixels outside the viewing angle is blocked by the grating and cannot pass through. The user cannot view the screen displayed by the display device 100 at the light-shielding part position of the display device 100.

[0146] The screen size of the display device 100 can be 14 inches, the viewing distance of the display device 100 is 4m, and the viewing angle is 50 degrees. When the distances between the left and right eyes of the target user and the display device 100 are less than 4m and the left and right eyes of the target user are within the viewing angle of the display device 100, it is determined that the target user is within the visible range of the display device 100.

[0147] In some embodiments, the display device 100 includes a camera 20, an eyeball positioning device 10, a determination module 30, and an anti-peeping protection module 40. The camera 20 is used to capture the facial image of the target user. The eyeball positioning device 10 is used to determine the left-eye camera coordinates and right-eye camera coordinates of the target user. The determination module 30 is used to determine whether the target user is within the visible range of the display device 100 according to the left-eye camera coordinates and right-eye camera coordinates of the target user, and the determination module 30 is also used to determine whether the target user is an authorized user of the display device 100. The anti-peeping protection module 40 is used to trigger the anti-peeping protection function of the display device 100 when the target user is within the visible range of the display device 100 and the target user is not an authorized user of the display device 100.

[0148] Refer to Figure 7 , the anti-peeping mode of the display device 100 can be flexibly turned on or off.

[0149] For example, when using a computer for work, it is not always necessary to turn on the anti-peeping function. The anti-peeping function is only turned on when dealing with some work with privacy functions or work that you don't want others to view. Therefore, the present invention can be implemented by starting or closing with hotkeys without making any changes to the hardware. For example, setting the windows + s hotkey to start the anti-peeping function and the windows + q hotkey to turn off the anti-peeping.

[0150] The anti-peeping mode of the display device 100 can flexibly authorize users. After turning on the anti-peeping function, it is first necessary to confirm which people can view the computer screen, that is, it is necessary to authorize the real viewers. The authorization is divided into two situations. One is the real owner of the computer, who has the highest anti-peeping authority.

[0151] After enabling the anti-peeping function, he / she can choose a historical portrait for anti-peeping or choose the currently captured face image for anti-peeping. Therefore, the system will wait for the user to confirm at this time. The hot key F1 is to use the historical portrait, and F2 is to perform anti-peeping after confirming the current screen. If no person is detected in the screen at this time, the system will give a prompt message.

[0152] The second case is that the computer owner authorizes all people who want to view the computer, and the number of people ≥ 2 at this time. After the owner presses the F2 hot key, the system will pop up a screen for him / her to confirm whether it is a truly authorized person. If it is confirmed, press the hot key E to enter the anti-peeping function. When someone else views the screen, the system will give an anti-peeping prompt message. During this process, the anti-peeping function can be exited by pressing the hot key Windows + Q at any time.

[0153] Refer to Figure 8 , after enabling the anti-peeping function, the built-in camera 20 of the computer will determine whether there is someone within the effective range of the camera 20. If no person is detected and the detected authorized user has been away from the computer for more than 30 seconds, the system will automatically lock the display screen to protect privacy. When someone is detected, the system enters the anti-peeping real-time monitoring.

[0154] Refer to Figure 9 , the process of the display device 100 real-time recognizing a face is as follows: First, read an image from the camera 20 for face detection, and then use the sort tracking algorithm to perform real-time tracking on the detected face to obtain a tracking ID number. Determine whether the current tracking ID number is an existing ID before. If it is, determine again whether the currently detected face is an authorized user. If it is, continue with the real-time detection. If not, activate the screen anti-peeping processing mechanism.

[0155] If the current tracking ID number is not in the previous tracking ID number queue, it means that a "new" viewer has entered. Here, the "new" viewer has two meanings: one is a real stranger, and the second is that the authorized person re-enters the range of the camera 20 after leaving the screen. At this time, perform face five key point detection (the centers of two eyes, the tip of the nose, and the two corners of the mouth), perform face pose estimation based on the 5 key points, and measure the distance between the eyes and the screen based on the interpupillary distance.

[0156] When the face is within the visible range of the display device 100, evaluate the face quality. The purpose of face quality evaluation is to prevent phenomena such as blurring and occlusion caused by someone moving quickly in an environment that meets the anti-peeping conditions, mainly to prevent false triggering. When the face meets the conditions, perform live detection again to prevent the influence of a billboard or the like behind.

[0157] When both the face quality assessment and the liveness detection meet the conditions, calculate the similarity between the currently extracted face features and the authorized face features in sequence. If the similarity is greater than the set threshold, it is considered that the person watching the screen is not the authorized person. For accurate judgment, this patent adopts a multiple voting mechanism for multiple judgments, that is, judge five times. If the system considers a person to be a peeper four times consecutively out of five times, then this person is considered to be a real peeper.

[0158] Refer to Figure 10 , when performing face quality assessment, the face quality model can participate in multi-label training to assess face quality. The information affecting face quality can be divided into 9 dimensions, and the face quality model trains a weight coefficient for each dimension simultaneously. After the model is trained, input a face, and finally output the face quality score.

[0159] The embodiments of the present invention provide a computer-readable storage medium. The computer-readable storage medium is used to store a computer program, and when the computer-readable storage medium is executed, it implements the eye positioning method or the anti-peeping control method of any of the above embodiments.

[0160] The embodiments of the eye positioning method and the anti-peeping control method can refer to the above embodiments. The beneficial effects of the computer-readable storage medium include all the beneficial effects of the eye positioning method and the anti-peeping control method, which will not be elaborated here one by one.

[0161] The embodiments of the present invention provide an eye positioning method, an anti-peeping control method, an eye positioning device 10, a display device 100, and a storage medium. The eye positioning method includes: obtaining the image coordinates of multiple feature points, the internal parameters of the camera 20, and the human head model, where the feature points are the feature points on the human head model, and the image coordinates of the feature points are determined based on the face image of the target user captured by the camera 20. The image coordinates of multiple feature points include the left eye image coordinates and the right eye image coordinates of the target user; determining the face pose angle of the target user in the head coordinate system according to the image coordinates of multiple feature points and the human head model; determining the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of multiple feature points, the internal parameters of the camera 20, and the human head model, where the camera coordinate system is determined based on the camera 20; determining the face pose angle of the target user in the camera coordinate system according to the transformation matrix and the face pose angle of the target user in the head coordinate system; determining the left eye camera coordinates and the right eye camera coordinates of the target user according to the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera 20, and the face pose angle of the target user in the camera coordinate system.

[0162] When positioning the eyes of the target user, first determine the face pose angle of the target user, and then locate the eye position of the target user according to the face pose angle of the target user, the left-eye image coordinates, the right-eye image coordinates captured by the camera 20, and the internal parameters of the camera 20. Compared with the method of directly positioning the eye position of the target user according to the image captured by the camera 20, the eye positioning method of the embodiment of the present invention takes into account the influence of the face pose angle of the target user on the located eye position, and improves the accuracy of the located eye position.

[0163] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples" or "some examples", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0164] In addition, the term "connection" should be understood in a broad sense. For example, it may include fixed connection, may also include detachable connection, or integral connection; it may include direct connection, may also be indirectly connected through an intermediate medium, and may also include the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present invention can be understood according to specific situations.

[0165] In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0166] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment or part of the code including one or more executable instructions for realizing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0167] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. An eyeball positioning method, characterized in that, The described eyeball positioning method includes: Obtaining the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model. The image coordinates of the multiple feature points are determined based on the face image of the target user captured by the camera. The image coordinates of the multiple feature points include the left eye image coordinates and the right eye image coordinates of the target user; Determining the face pose angles of the target user in the head coordinate system according to the image coordinates of the multiple feature points and the human head model; Determining the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model. The camera coordinate system is determined based on the camera; Determining the face pose angles of the target user in the camera coordinate system according to the transformation matrix and the face pose angles of the target user in the head coordinate system; Determining the left eye camera coordinates and the right eye camera coordinates of the target user according to the left eye image coordinates, the right eye image coordinates, the internal parameters of the camera, and the face pose angles of the target user in the camera coordinate system.

2. The eyeball positioning method according to claim 1, characterized in that, The image coordinates of the multiple feature points further include the nose tip image coordinates of the target user, the left mouth corner image coordinates of the target user, and the right mouth corner image coordinates of the target user.

3. The eye positioning method according to claim 1, wherein The face pose angles include the yaw angle, the pitch angle, and the roll angle of the face.

4. The eyeball positioning method according to claim 1, characterized in that The determining of the face pose angles of the target user in the head coordinate system according to the image coordinates of the multiple feature points and the human head model includes: Determining the coordinates of the multiple feature points in the head coordinate system according to the image coordinates of the multiple feature points and the human head model; Determining the face pose angles of the target user in the head coordinate system according to the coordinates of the multiple feature points in the head coordinate system, the image coordinates of the multiple feature points, and the internal parameters of the camera.

5. The eye positioning method according to claim 1, characterized in that, The determining of the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model includes: Determining the coordinates of the multiple feature points in the camera coordinate system according to the image coordinates of the multiple feature points and the internal parameters of the camera; Determining the coordinates of the multiple feature points in the head coordinate system according to the image coordinates of the multiple feature points and the human head model; Determining the transformation matrix between the head coordinate system and the camera coordinate system according to the coordinates of the multiple feature points in the camera coordinate system and the coordinates of the multiple feature points in the head coordinate system.

6. The eye positioning method according to claim 1, characterized in that The determining of the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model includes: Determining the gaze point image coordinates of the target user according to the weighted average of the left eye image coordinates and the right eye image coordinates of the target user; Determining the transformation matrix between the head coordinate system and the camera coordinate system according to the gaze point image coordinates, the internal parameters of the camera, and the human head model.

7. The eye positioning method according to claim 1, wherein Determining the left-eye camera coordinates and the right-eye camera coordinates based on the left-eye image coordinates, the right-eye image coordinates, the internal parameters of the camera, and the face pose angle of the target user in the camera coordinate system includes: Determining the pupil distance of the target user according to the difference between the left-eye image coordinates and the right-eye image coordinates; Determining the left-eye camera coordinates and the right-eye camera coordinates according to the left-eye image coordinates, the right-eye image coordinates, the internal parameters of the camera, the face pose angle of the target user in the camera coordinate system, and the pupil distance of the target user.

8. The eye positioning method according to claim 1, characterized in that, After determining the left-eye camera coordinates and the right-eye camera coordinates, the eye positioning method further includes: Obtaining the external parameters of the camera; Determining the left-eye world coordinates of the target user according to the external parameters of the camera and the left-eye camera coordinates; Determining the right-eye world coordinates of the target user according to the external parameters of the camera and the right-eye camera coordinates.

9. An anti-peeping control method for a display device, characterized in that, The anti-peeping control method includes: Determining the left-eye camera coordinates and the right-eye camera coordinates of the target user according to the eye positioning method according to any one of claims 1 to 8; Determining whether the target user is within the visible range of the display device according to the left-eye camera coordinates and the right-eye camera coordinates; When the target user is within the visible range of the display device and the target user is not an authorized user of the display device, controlling the display device to trigger the anti-peeping protection function.

10. The anti-peeping control method according to claim 9, characterized in that, The determining whether the target user is within the visible range of the display device according to the left-eye camera coordinates and the right-eye camera coordinates includes: Determining the distances and relative angles between the left eye and the right eye of the target user and the display device according to the left-eye camera coordinates and the right-eye camera coordinates; When the distances between the left eye and the right eye of the target user and the display device are less than the visible distance of the display device and the relative angles between the left eye and the right eye of the target user and the display device are less than the visible angle of the display device, determining that the target user is within the visible range of the display device.

11. An eyeball positioning device, characterized in that, The eye positioning device includes: An acquisition module, which is used to acquire the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model. The image coordinates of the multiple feature points are determined based on the face image of the target user captured by the camera. The image coordinates of the multiple feature points include the left-eye image coordinates and the right-eye image coordinates of the target user; A calculation module, which is used to determine the face pose angle of the target user in the head coordinate system according to the image coordinates of the multiple feature points and the human head model; The calculation module is further used to determine the transformation matrix between the head coordinate system and the camera coordinate system according to the image coordinates of multiple feature points, the internal parameters of the camera, and the human head model. The camera coordinate system is determined based on the camera; The calculation module is further used to determine the face pose angle of the target user in the camera coordinate system according to the transformation matrix and the face pose angle of the target user in the head coordinate system; The calculation module is further configured to determine the left-eye camera coordinates and right-eye camera coordinates of the target user according to the left-eye image coordinates, the right-eye image coordinates, the internal parameters of the camera, and the face pose angle of the target user in the camera coordinate system.

12. A display device, characterized in that, The display device includes: a camera configured to capture a face image of a target user; the eyeball positioning device according to claim 11, the eyeball positioning device being configured to determine the left-eye camera coordinates and right-eye camera coordinates of the target user; a determination module configured to determine whether the target user is within the visible range of the display device according to the left-eye camera coordinates and right-eye camera coordinates of the target user, and the determination module is further configured to determine whether the target user is an authorized user of the display device; an anti-peeping protection module configured to trigger the anti-peeping protection function of the display device when the target user is within the visible range of the display device and the target user is not an authorized user of the display device.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store a computer program, and when the computer-readable storage medium is executed, it implements the eyeball positioning method according to any one of claims 1-8 or the anti-peeping control method according to claim 9 or 10.

Images