Depth detection method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a depth detection method and device, a storage medium and electronic equipment, and the method comprises the steps: firstly, carrying out the detection of a current head posture of a user, obtaining the first coordinates of a plurality of total feature points of a user face in a reference coordinate system, and then, carrying out the detection of the first coordinates; and calculating a difference value between a preset axial numerical value in the first coordinate of the first feature point and a preset axial numerical value in the first coordinate of the second feature point, determining a target feature point in the plurality of feature points based on the difference value, and finally determining the target feature point in the plurality of feature points based on the first coordinate of the target feature point and a pinhole camera model imaging equation. And calculating the depth from the eyes of the user to the terminal screen. According to the embodiment of the invention, the current head posture of the user is detected to select the target feature points for calculating the depth from the human eyes to the screen, and then the depth from the human eyes to the terminal screen is calculated according to the target feature points, so that the accuracy of measuring the depth from the human eyes to the screen is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a depth detection method, device, storage medium and electronic device. Background Art

[0002] With the popularity of smart devices, people are increasingly relying on them in their daily lives. Long-term use of smart devices or poor usage habits can cause damage to people's eyesight.

[0003] In the related art, in order to protect the user's eyesight, some smart devices will intelligently remind the user to move away from the screen when the user is close to the screen. In this process, the smart device mainly monitors the distance between the user and the screen, and when the distance between the user and the screen exceeds a set threshold, it will remind the user to move away from the screen. Summary of the invention

[0004] The embodiments of the present application provide a depth detection method, device, computer storage medium and electronic device, which detect the current head posture of the user to select the best target feature points for calculating the depth of the human eye from the screen, and then calculate the depth of the human eye from the terminal screen based on these target feature points, thereby improving the accuracy of measuring the depth of the human eye from the screen. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a depth detection method, the method comprising:

[0006] Detecting the current head posture of the user to obtain first coordinates of a plurality of feature points in the user's face in a reference coordinate system, wherein the reference coordinate system is parallel to the camera coordinate system, wherein the plurality of feature points include at least a first feature point and a second feature point, wherein the first feature point and the second feature point are symmetrical feature points in two eye regions;

[0007] Calculate the difference between the value of the preset axial direction in the first coordinate of the first feature point and the value of the preset axial direction in the first coordinate of the second feature point, and determine the target feature point from the plurality of feature points based on the difference;

[0008] Based on the first coordinates of the target feature point and the pinhole camera model imaging equation, the depth of the user's human eye from the terminal is calculated.

[0009] In a second aspect, an embodiment of the present application provides a depth detection device, the device comprising:

[0010] A first processing module is used to detect the current head posture of the user, and obtain first coordinates of a plurality of feature points in the user's face in a reference coordinate system, wherein the reference coordinate system is parallel to the camera coordinate system, and the plurality of feature points include at least a first feature point and a second feature point, wherein the first feature point and the second feature point are symmetrical feature points in two eye regions;

[0011] A second processing module, configured to calculate a difference between a value of a preset axial direction in the first coordinate of the first feature point and a value of a preset axial direction in the first coordinate of the second feature point, and determine a target feature point from among the plurality of feature points based on the difference;

[0012] The depth calculation module is used to calculate the depth of the user's human eye from the terminal based on the first coordinate of the target feature point and the pinhole camera model imaging equation.

[0013] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium has a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0014] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a memory and a processor; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the memory and executing the above-mentioned method steps.

[0015] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0016] In an embodiment of the present application, the current head posture of the user is first detected to obtain the first coordinates of the multiple feature points of the user's face in the reference coordinate system, and then the difference between the value of the preset axis in the first coordinate of the first feature point and the value of the preset axis in the first coordinate of the second feature point is calculated, and the target feature point is determined from the multiple feature points based on the difference, and finally, the depth of the user's eye from the terminal screen is calculated based on the first coordinate of the target feature point and the pinhole camera model imaging equation. The embodiment of the present application detects the current head posture of the user to select the target feature points that are preferably used to calculate the depth of the eye from the screen, and then calculates the depth of the eye from the terminal screen based on these target feature points, thereby improving the accuracy of measuring the depth of the eye from the screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flowchart of a depth detection method provided in an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a pinhole camera model imaging equation provided in an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of another pinhole camera model imaging equation provided in an embodiment of the present application;

[0021] Figure 4 It is a flowchart of another depth detection method provided in an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a depth calculation equation provided in an embodiment of the present application;

[0023] Figure 6 It is a flowchart of another depth detection method provided in an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of another depth calculation equation provided in an embodiment of the present application;

[0025] Figure 8 It is a flowchart of another depth detection method provided in an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of another depth calculation equation provided in an embodiment of the present application;

[0027] Fig.10 It is a flowchart of another depth detection method provided in an embodiment of the present application;

[0028] Fig.11 is a schematic structural diagram of a depth detection device provided in an embodiment of the present application;

[0029] Fig.12 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0031] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0032] In the related art, in some application scenarios, a smart device can monitor the distance between a user and a device screen, and the smart device may include but is not limited to a mobile phone, a tablet, and a laptop computer.

[0033] For example, in some vision protection scenarios, such as students watching course videos through smart devices, students completing course assignments through smart devices, and users watching books and entertainment videos through smart devices, smart devices can monitor the distance between the user's eyes and the device screen in real time. When the distance between the user's eyes and the device screen is detected to be less than the set threshold, the smart device can trigger a corresponding reminder operation to remind the user to keep a certain distance from the device screen, thereby protecting eyesight.

[0034] In summary, the relevant technology can realize the function of monitoring the distance between the user's eyes and the device screen. Therefore, how to improve the accuracy of detecting the distance between the user's eyes and the device screen is a technical problem that needs to be solved urgently.

[0035] In view of the above problems, the present application proposes a technical solution of a depth detection method to improve the accuracy of detecting the distance between the user's eyes and the device screen. The specific concept of the technical solution is as follows:

[0036] When the user's head posture changes, due to occlusion and viewing angle, the detection accuracy of facial feature points far away from the camera lens is poor. Therefore, the present application designs a technical solution based on detecting the user's head posture, that is, determining the target feature points selected for the depth between the human eye and the device screen under the current head posture based on the user's head posture, and then solving the depth between the human eye and the device screen based on the target feature points.

[0037] When the user's head posture changes, the coordinates (X1, Y1, Z1) and (X2, Y2, Z2) of the two facial feature points in the two eye regions in symmetrical positions in the camera coordinate system are used to calculate (X1-X2). 2 +(Y1-Y2) 2 It is not equal to the square of the Euclidean distance between the two facial feature points. This is only true if the depths of the two facial feature points in the camera coordinate system are the same, (X1-X2) 2 +(Y1-Y2) 2 It is equal to the square of the Euclidean distance between the two facial feature points. Therefore, it is necessary to determine the target feature points involved in the solution based on the head posture.

[0038] The depth between the left eye feature point in the left eye area and the screen can be called the first depth, and the depth between the right eye feature point in the right eye area and the screen can be called the second depth. The left eye feature point and the right eye feature point are feature points in symmetrical positions in the two eye areas. When the difference between the first depth and the second depth is not large, it means that the distance between the two eyes and the screen is not much different. The symmetrical feature points in the two eye areas can be determined as target feature points to solve the depth between the human eye and the screen. When the difference between the first depth and the second depth is large, it means that the distance between the two eyes and the screen is large. Feature points closer to the screen can be selected as target feature points to solve the depth between the human eye and the screen. Through the above feature point selection strategy, the depth between the human eye and the screen can be solved according to the selected accurate feature points, thereby achieving the effect of improving the accuracy of the depth.

[0039] Based on the above technical concept, the depth detection method of the embodiment of the present application is further explained below in combination with specific embodiments.

[0040] See also Figure 1 , is a flow chart of a depth detection method provided in an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0041] S101, detecting the current head posture of the user to obtain first coordinates of a plurality of feature points in the user's face in a reference coordinate system.

[0042] It is easy to understand that the current head posture may refer to the head posture of the user at the current moment.

[0043] A feature point may refer to a facial feature point extracted according to the position of the facial features. For example, the feature point may be a feature point in the eye region, a feature point in the nose region, or a feature point in the mouth region. The multiple feature points in the present embodiment may include at least a first feature point and a second feature point, and the first feature point and the second feature point may be symmetrical feature points in the two eye regions. That is, the first feature point may be a feature point in the left eye region, and the second feature point may be a feature point in the right eye region that is symmetrical to the first feature point. Alternatively, the first feature point may be a feature point in the right eye region, and the second feature point may be a feature point in the left eye region that is symmetrical to the first feature point. For example, the first feature point may refer to a feature point at the left corner of the eye, and the second feature point may refer to a feature point at the right corner of the eye; or the first feature point may refer to a feature point at the right corner of the eye, and the first feature point may refer to a feature point at the left corner of the eye.

[0044] The reference coordinate system may refer to a three-dimensional Cartesian coordinate system parallel to the camera coordinate system. The camera coordinate system may refer to a three-dimensional Cartesian coordinate system with the center point of the camera on the terminal as the origin and composed of set X-axis, Y-axis and Z-axis. In the camera coordinate system, the plane where the terminal screen is located can be regarded as the plane composed of the X-axis and the Y-axis, the straight line perpendicular to the terminal screen can be regarded as the Z-axis, and the direction of the camera toward the object can be regarded as the positive direction of the Z-axis. It should be noted that the terminal in the embodiment of the present application may include but is not limited to mobile phones, personal computers, laptops, interactive tablets and other terminal devices with camera functions.

[0045] In some embodiments, the current head posture of the user can be detected to obtain the second coordinates of multiple feature points in the user's face in the current coordinate system corresponding to the current head posture; further, the rotation matrix of the current coordinate system to the reference coordinate system can be obtained, and the second coordinates of the multiple feature points can be transformed according to the rotation matrix to obtain the first coordinates of the multiple feature points in the reference coordinate system, that is, the second coordinates of the feature points in the current coordinate system are converted to the first coordinates in the reference coordinate system according to the rotation matrix. Among them, the current coordinate system refers to a three-dimensional Cartesian coordinate system, and the current coordinate system can change with the change of the user's head posture. Whenever the user's head posture changes from one posture to another, the current coordinate system also changes accordingly.

[0046] In a feasible implementation, the multiple feature points may refer to three feature points, which may be a first feature point, a second feature point and a third feature point. The third feature point may refer to a feature point of the root of the nose. The first feature point may be a feature point of the left corner of the eye, and the second feature point may be a feature point of the right corner of the eye.

[0047] In another feasible implementation, the multiple feature points may refer to more than three feature points, which may include a first feature point, a feature point belonging to the same eye region as the first feature point, a second feature point, a feature point belonging to the same eye region as the second feature point, and a third feature point. Among them, the feature point belonging to the same eye region as the first feature point and the feature point belonging to the same eye region as the second feature point are feature points with symmetrical positions; when the number of feature points belonging to the same eye region as the first feature point is N, and N is an integer greater than 1, correspondingly, the number of feature points belonging to the same eye region as the second feature point is also N, and then there are N pairs of feature points with mutually symmetrical positions.

[0048] S102, calculating a difference between a value of a preset axial direction in a first coordinate of a first feature point and a value of a preset axial direction in a first coordinate of a second feature point, and determining a target feature point from a plurality of feature points based on the difference.

[0049] It is easy to understand that the preset axis direction may refer to the Z axis in the reference coordinate system.

[0050] The first feature point may refer to the feature point of the left eye corner in the left eye region, and the second feature point may refer to the feature point of the right eye corner in the right eye region. Alternatively, the first feature point may refer to the feature point of the right eye corner in the right eye region, and the second feature point may refer to the feature point of the left eye corner in the left eye region. In the following embodiments of this specification, the first feature point is the feature point of the left eye corner, and the second feature point is the feature point of the right eye corner. In this case, the first feature point is the feature point of the right eye corner, and the second feature point is the feature point of the left eye corner. The corresponding result can be inferred when the first feature point is the feature point of the right eye corner and the second feature point is the feature point of the left eye corner.

[0051] In some embodiments, the difference between the value of the Z axis in the first coordinate of the first feature point and the value of the Z axis in the first coordinate of the second feature point can be calculated, and further, the target feature point can be determined from the multiple feature points according to the size relationship between the difference and the preset first threshold and second threshold, wherein the first threshold is smaller than the second threshold.

[0052] Specifically, when the multiple feature points described in S101 refer to three feature points, when it is determined that the absolute value of the difference is less than or equal to the first threshold, the first feature point and the second feature point can be determined as the target feature point. When the multiple feature points described in S101 refer to more than three feature points, when it is determined that the absolute value of the difference is less than or equal to the first threshold, the first feature point, the feature point belonging to the same eye area as the first feature point, the second feature point, and the feature point belonging to the same eye area as the second feature point can be determined as the target feature point.

[0053] Optionally, in the case where the multiple feature points described in S101 refer to three feature points, when it is determined that the absolute value of the difference is greater than the second threshold value and the difference is a positive number, the second feature point and the third feature point can be determined as target feature points. In the case where the multiple feature points described in S101 refer to more than three feature points, when it is determined that the absolute value of the difference is greater than or equal to the second threshold value and the difference is a positive number, the second feature point, the feature point belonging to the same eye area as the second feature point, and the third feature point can be determined as target feature points.

[0054] Optionally, in the case where the multiple feature points described in S101 refer to three feature points, when it is determined that the absolute value of the difference is greater than the second threshold value, and the difference is a negative number, the first feature point and the third feature point can be determined as target feature points. In the case where the multiple feature points described in S101 refer to more than three feature points, when it is determined that the absolute value of the difference is greater than or equal to the second threshold value, and the difference is a negative number, the first feature point, the feature point belonging to the same eye area as the first feature point, and the third feature point can be determined as target feature points.

[0055] Optionally, in the case where the multiple feature points described in S101 refer to three feature points, when it is determined that the absolute value of the difference is less than or equal to the second threshold, and the absolute value is greater than the first threshold, the first feature point, the second feature point, and the third feature point may be determined as target feature points. In the case where the multiple feature points described in S101 refer to more than three feature points, when it is determined that the absolute value of the difference is less than or equal to the second threshold, and the absolute value is greater than the first threshold, the first feature point, the feature point belonging to the same eye area as the first feature point, the second feature point, the feature point belonging to the same eye area as the second feature point may be determined as the target feature point, and the third feature point may be determined as the target feature point.

[0056] S103, calculating the depth of the user's eyes from the terminal screen based on the first coordinates of the target feature point and the pinhole camera model imaging equation.

[0057] It is easy to understand that the pinhole camera model imaging equation may refer to an equation derived according to the pinhole camera imaging model. The pinhole camera imaging model may refer to a pinhole imaging model. The pinhole camera imaging model equation in the embodiment of the present application can be found in Figure 2 The equation shown. Figure 2 In the equation shown, f x 、f y , u0 and v0 are the internal parameters of the camera, f x and f y is the focal length, u0 and v0 are the principal point coordinates; X c '+X c Indicates the value of the target feature point in the X-axis in the camera coordinate system, Y c '+Y c Indicates the value of the target feature point on the Y axis in the camera coordinate system, c '+Z c Indicates the value of the target feature point in the Z axis in the camera coordinate system; X c ' represents the value of the target feature point in the reference coordinate system along the X axis, Y c ' represents the value of the target feature point on the Y axis in the reference coordinate system, Z c ' represents the value of the target feature point in the Z axis in the reference coordinate system; u and v represent the values ​​of the target feature point in the two axes in the pixel coordinate system.

[0058] In some embodiments, Figure 2 The pinhole camera imaging model equation shown can be converted into Figure 3 As shown in the equation, due to X c ', Y c ' and Z c ' is the value of the three axes in the first coordinate of the target feature point, that is, X c ', Y c ' and Z c ' is also known, then Figure 3 In the equation shown, only X c , Y c and Z c Three unknowns. The first coordinate of the target feature point can be substituted into Figure 3 The equation shown in Figure 2 is used to solve for Z c For each target feature point, substitute its first coordinate into Figure 3 The equation shown in the figure can be used to obtain two equations. When the first coordinates of two or more target feature points are substituted into Figure 3 The equation shown in the figure can give 4 or more equations, so Z can be solved based on 4 or more equations. c .

[0059] Specifically, when it is determined that the absolute value of the difference in S102 is less than or equal to the first threshold, and when it is determined that the absolute value of the difference in S102 is greater than the second threshold, Z obtained by solving the multiple equations is c That is, the depth of the user's eyes from the terminal screen.

[0060] Optionally, when it is determined in S102 that the absolute value of the difference is less than or equal to the second threshold and the absolute value is greater than the first threshold, a linear weighted method can be used to calculate the solved depth to obtain a target depth, which is the depth of the user's eyes from the terminal screen.

[0061] In an embodiment of the present application, the current head posture of the user is first detected to obtain the first coordinates of the multiple feature points of the user's face in the reference coordinate system, and then the difference between the value of the preset axis in the first coordinate of the first feature point and the value of the preset axis in the first coordinate of the second feature point is calculated, and the target feature point is determined from the multiple feature points based on the difference, and finally, the depth of the user's eye from the terminal screen is calculated based on the first coordinate of the target feature point and the pinhole camera model imaging equation. The embodiment of the present application detects the current head posture of the user to select the optimal target feature point for calculating the depth of the eye from the screen, and then calculates the depth of the eye from the terminal screen based on these target feature points, thereby improving the accuracy of measuring the depth of the eye from the screen.

[0062] See also Figure 4 , is a flow chart of a depth detection method provided in an embodiment of the present application. Figure 4 As shown, the method of the embodiment of the present application may include the following steps:

[0063] S401, detecting the current head posture of the user to obtain second coordinates of a plurality of feature points in the user's face in a current coordinate system corresponding to the current head posture.

[0064] It should be noted that the definitions of the current head posture and feature points may be specifically referred to the description in S101 and will not be repeated here.

[0065] It is easy to understand that the current coordinate system may refer to a three-dimensional Cartesian coordinate system determined according to the current head posture. In an embodiment of the present application, the current coordinate system may be obtained by rotating the initial coordinate system by a certain angle. The initial coordinate system refers to a three-dimensional Cartesian coordinate system determined according to the initial head posture of the user, and the initial head posture refers to the head posture of the user's face facing the camera.

[0066] In the initial coordinate system, the feature point of the root of the nose can be used as the origin of the coordinate system, and a vertical plane can be established with the origin, that is, a plane parallel to the plane where the terminal screen is located, and the plane is used as the plane composed of the X-axis and the Y-axis in the initial coordinate system. For this plane, its vertical direction can be the direction where the X-axis is located, its horizontal direction can be the direction where the Y-axis is located, and the direction perpendicular to the plane can be the direction where the Z-axis is located. The direction toward the terminal screen can be used as the positive direction of the Z-axis. The current head posture can be understood as the initial head posture after rotating a certain angle, so the current coordinate system can be obtained by rotating the initial coordinate system in the same direction by the angle.

[0067] In some embodiments, the user's current head posture can be detected to obtain an initial rotation angle of the current head posture relative to the initial head posture in the initial coordinate system, and based on the initial rotation angle, the second coordinates of multiple feature points in the user's face in the current coordinate system corresponding to the current head posture are calculated.

[0068] Specifically, a pre-trained head posture estimation model can be used to detect the user's current head posture to obtain an initial rotation angle of the current head posture relative to the initial head posture in the initial coordinate system. The initial rotation angle may refer to the rotation angle of the head around three axes. For example, the head posture estimation model may use the FSA-Net model.

[0069] Furthermore, before calculating the second coordinate, a facial feature point detection model can be used to detect the user's face to obtain the initial coordinates of multiple feature points in the user's face in the initial coordinate system; then, based on the initial rotation angle, the initial coordinates are calculated to obtain the second coordinates of multiple feature points in the current coordinate system corresponding to the current head posture.

[0070] S402, obtaining a rotation matrix corresponding to the current coordinate system and the reference coordinate system, and calculating first coordinates of a plurality of feature points in the reference coordinate system based on the second coordinates and the rotation matrix.

[0071] It is easy to understand that the rotation matrix can refer to the coordinate transformation matrix. The coordinates of the feature points are transformed from the current coordinate system to the reference coordinate system. The coordinates before the transformation of the coordinate system can be calculated according to the rotation matrix to obtain the coordinates after the transformation of the coordinate system.

[0072] In some embodiments, the rotation matrix for coordinate transformation between two coordinate systems can be inferred in advance, and the inferred rotation matrix can be stored in a database. The rotation matrix can be read from the database, and then the rotation matrix can be used to simultaneously calculate the second coordinates of multiple feature points to obtain the first coordinates of the multiple feature points in the reference coordinate system.

[0073] S403, calculating a difference between a value of a preset axial direction in the first coordinate of the first feature point and a value of a preset axial direction in the first coordinate of the second feature point.

[0074] It is easy to understand that the first feature point refers to the feature point of the left eye corner, and the second feature point refers to the feature point of the right eye corner. The preset axis refers to the Z axis.

[0075] In some embodiments, a difference between a value of the Z axis in the first coordinate of the first feature point and a value of the Z axis in the first coordinate of the second feature point may be calculated.

[0076] S404: If it is determined that the absolute value of the difference is less than or equal to the first threshold, the first feature point and the second feature point are determined as target feature points.

[0077] It is easy to understand that the first threshold and the second threshold can be preset, the second threshold is greater than the first threshold, and both thresholds are positive numbers.

[0078] In some embodiments, if the absolute value of the difference is determined to be less than or equal to the first threshold, the first feature point and the second feature point may be determined as the target feature point. In this case, the set multiple feature points refer to the first feature point, the second feature point and the third feature point; the third feature point refers to the feature point of the root of the nose described in S401 above.

[0079] Optionally, in another embodiment, if the absolute value of the difference is determined to be less than or equal to the first threshold, the first feature point, the feature point belonging to the same eye area as the first feature point, the second feature point, and the feature point belonging to the same eye area as the second feature point can be determined as the target feature point. In this case, the set multiple feature points refer to the first feature point, the feature point belonging to the same eye area as the first feature point, the second feature point, the feature point belonging to the same eye area as the second feature point, and the third feature point. Among them, the third feature point refers to the feature point of the root of the nose described in S401 above. The number of feature points belonging to the same eye area as the first feature point can be one or more, and correspondingly, the number of feature points belonging to the same eye area as the second feature point can be one or more, because the feature point belonging to the same eye area as the first feature point and the feature point belonging to the same eye area as the second feature point can also be feature points whose positions are symmetrical to each other. For example, the feature point belonging to the same eye area as the first feature point is the feature point of the center point of the left eye, and the feature point belonging to the same eye area as the second feature point is the feature point of the center point of the right eye.

[0080] S405, substituting the first coordinates of each target feature point into the pinhole camera model imaging equation to obtain the calculation equation corresponding to each target feature point.

[0081] It should be noted that the definition of the pinhole camera model imaging equation can be specifically referred to the description in S103, which will not be repeated here.

[0082] In some embodiments, the first coordinates of the first feature point and the first coordinates of the second feature point can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure can be used to obtain two pairs of Figure 3 The calculation equations shown in FIG. One pair of calculation equations can be the equations after the first coordinate of the first feature point is substituted into the calculation. In the pair of equations, X c ' represents the value of the X axis in the first coordinate of the first feature point, Y c ' represents the value of the Y axis in the first coordinate of the first feature point, Z c ' represents the value of the Z axis in the first coordinate of the first feature point. Another pair of calculation equations can be the equations after substituting the first coordinate of the second feature point into the calculation. In this pair of equations, X c ' represents the value of the X axis in the first coordinate of the second feature point, Y c ' represents the value of the Y axis in the first coordinate of the second feature point, Z c ' represents the value of the Z axis in the first coordinate of the second feature point.

[0083] Optionally, in another embodiment, when the first feature point, the feature point belonging to the same eye region as the first feature point, the second feature point, and the feature point belonging to the same eye region as the second feature point are target feature points, the first coordinates of these target feature points can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure gives several pairs of Figure 3 The calculation equation is shown. In each pair of Figure 3 In the calculation equation shown, X c 'Represents the value of the X axis in the first coordinate of each target feature point, Y c 'Represents the value of the Y axis in the first coordinate of each target feature point, Z c 'Represents the Z-axis value of the first coordinate of each target feature point.

[0084] S406: Calculate the depth of the user's eyes from the terminal screen based on all calculation equations.

[0085] In some embodiments, when the first feature point and the second feature point are target feature points, the calculated Figure 3 The two pairs of calculation equations shown in the figure are processed and the following is obtained: Figure 5 The target equation is shown. Figure 5 In the target equation shown, X c '1 represents the value of the X axis in the first coordinate of the first feature point, Y c'1 represents the value of the Y axis in the first coordinate of the first feature point, X c '2 represents the value of the X axis in the first coordinate of the second feature point, Y c '2 represents the value of the Y axis in the first coordinate of the second feature point; u1 and v1 represent the values ​​of the two axes of the first feature point in the pixel coordinate system; u2 and v2 represent the values ​​of the two axes of the second feature point in the pixel coordinate system. Figure 5 It can be seen that the value on the left side of the equation is Z c 2 , we can perform square root operation on this value to get Z c The value of Z c That is, the depth between the user's glasses and the terminal screen.

[0086] Optionally, in another embodiment, when the first feature point, the feature point belonging to the same eye region as the first feature point, the second feature point, and the feature point belonging to the same eye region as the second feature point are target feature points, the first coordinates of these target feature points can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure gives several pairs of Figure 3 The calculation equation shown in the figure can be solved to get Z after operation. c The value of Z c That is, the depth between the user's glasses and the terminal screen.

[0087] In an embodiment of the present application, the current head posture of the user is detected to obtain the second coordinates of multiple feature points in the user's face in the current coordinate system corresponding to the current head posture, obtain the rotation matrix corresponding to the current coordinate system and the reference coordinate system, calculate the first coordinates of the multiple feature points in the reference coordinate system based on the second coordinates and the rotation matrix, calculate the difference between the numerical value of the preset axis in the first coordinate of the first feature point and the numerical value of the preset axis in the first coordinate of the second feature point, and if the absolute value of the difference is determined to be less than or equal to the first threshold, determine the first feature point and the second feature point as target feature points, and finally, calculate the depth of the user's eyes from the terminal screen based on the first coordinates of the target feature points and the pinhole camera model imaging equation. In the embodiment of the present application, when the difference in the depth of the two eyes from the terminal screen is small, the feature points of the two eyes are used as target feature points to calculate the depth of the eyes from the screen, which can ensure the accuracy of the final measured depth, thereby improving the accuracy of measuring the depth of the eyes from the screen.

[0088] See also Figure 6 , is a flow chart of a depth detection method provided in an embodiment of the present application. Figure 6 As shown, the method of the embodiment of the present application may include the following steps:

[0089] S601, detecting the current head posture of the user to obtain second coordinates of a plurality of feature points in the user's face in a current coordinate system corresponding to the current head posture.

[0090] S602, obtaining a rotation matrix corresponding to the current coordinate system and the reference coordinate system, and calculating first coordinates of a plurality of feature points in the reference coordinate system based on the second coordinates and the rotation matrix.

[0091] S603, calculating a difference between a value of a preset axial direction in the first coordinate of the first feature point and a value of a preset axial direction in the first coordinate of the second feature point.

[0092] Specifically, S601-S603 can be found in Figure 4 The description of S401 - S403 in the illustrated embodiment will not be repeated here.

[0093] S604: If it is determined that the absolute value of the difference is greater than the second threshold and the difference is a positive number, determine the target feature point based on the second area feature point of the second area where the second feature point is located, obtain a preset third feature point, and configure the third feature point as the target feature point.

[0094] It is easy to understand that the first threshold and the second threshold can be preset, the second threshold is greater than the first threshold, and both thresholds are positive numbers.

[0095] In some embodiments, if the absolute value of the difference is determined to be greater than the second threshold value and the difference is a positive number, the second feature point and the third feature point may be determined as target feature points. In this case, the set multiple feature points refer to the first feature point, the second feature point and the third feature point; the third feature point refers to the feature point of the root of the nose described in S401 above.

[0096] Optionally, in another embodiment, if the absolute value of the difference is determined to be greater than the second threshold value, and the difference is a positive number, the second feature point, the feature point belonging to the same eye area as the second feature point (i.e., the feature point of the second area other than the second feature point in the second area where the second feature point is located) and the third feature point can be determined as the target feature point. In this case, the multiple feature points set refer to the first feature point, the feature point belonging to the same eye area as the first feature point, the second feature point, the feature point belonging to the same eye area as the second feature point, and the third feature point. Among them, the third feature point refers to the feature point of the root of the nose described in S401 above. The number of feature points belonging to the same eye area as the first feature point can be one or more, and accordingly, the number of feature points belonging to the same eye area as the second feature point can be one or more, because the feature point belonging to the same eye area as the first feature point and the feature point belonging to the same eye area as the second feature point can also be feature points whose positions are symmetrical to each other. For example, the feature point belonging to the same eye region as the first feature point is the feature point of the left eye center point, and the feature point belonging to the same eye region as the second feature point is the feature point of the right eye center point.

[0097] S605, substituting the first coordinates of each target feature point into the pinhole camera model imaging equation to obtain the calculation equation corresponding to each target feature point.

[0098] It should be noted that the definition of the pinhole camera model imaging equation can be specifically referred to the description in S103, which will not be repeated here.

[0099] In some embodiments, the first coordinates of the second feature point and the third feature point can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure can be used to obtain two pairs of Figure 3 The calculation equations shown in FIG. One pair of calculation equations can be the equations after the first coordinate of the second feature point is substituted into the calculation. In the pair of equations, X c ' represents the value of the X axis in the first coordinate of the second feature point, Y c ' represents the value of the Y axis in the first coordinate of the second feature point, Z c ' represents the value of the Z axis in the first coordinate of the second feature point. Another pair of calculation equations can be the equations after substituting the first coordinate of the third feature point into the calculation. In this pair of equations, X c ' represents the value of the X axis in the first coordinate of the third feature point, Y c ' represents the value of the Y axis in the first coordinate of the third feature point, Z c ' represents the value of the Z axis in the first coordinate of the third feature point.

[0100] Optionally, in another embodiment, when the second feature point, the feature point belonging to the same eye region as the second feature point (i.e., the second region feature point other than the second feature point in the second region where the second feature point is located), and the third feature point are target feature points, the first coordinates of these target feature points can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure gives several pairs of Figure 3 The calculation equation is shown. In each pair of Figure 3 In the calculation equation shown, X c 'Represents the value of the X axis in the first coordinate of each target feature point, Y c 'Represents the value of the Y axis in the first coordinate of each target feature point, Z c 'Represents the Z-axis value of the first coordinate of each target feature point.

[0101] S606: Calculate the depth of the user's eyes from the terminal screen based on all calculation equations.

[0102] In some embodiments, when the second feature point and the third feature point are target feature points, the calculated Figure 3 The two pairs of calculation equations shown in the figure are processed and the following is obtained: Figure 7 The target equation is shown. Figure 7 In the target equation shown in FIG. 1 , since the third feature point, i.e., the feature point of the root of the nose, is defined as the origin in the reference coordinate system, the values ​​of each axis in the first coordinate of the third feature point are all 0; X c '2 represents the value of the X axis in the first coordinate of the second feature point, Y c '2 represents the value of the Y axis in the first coordinate of the second feature point; u3 and v3 represent the values ​​of the two axes of the third feature point in the pixel coordinate system; u2 and v2 represent the values ​​of the two axes of the second feature point in the pixel coordinate system. Figure 7 It can be seen that the value on the left side of the equation is Z c 2 , we can perform square root operation on this value to get Z c The value of Z c That is, the depth between the user's glasses and the terminal screen.

[0103] Optionally, in another embodiment, when the second feature point, the feature point belonging to the same eye region as the second feature point (i.e., the second region feature point other than the second feature point in the second region where the second feature point is located), and the third feature point are target feature points, the first coordinates of these target feature points can be substituted into Figure 2 After the pinhole camera model imaging equation is shown, all the calculation equations are obtained, and Z can be solved by calculation. c The value of Z cThat is, the depth between the user's glasses and the terminal screen.

[0104] In an embodiment of the present application, the current head posture of the user is detected, and the second coordinates of multiple feature points in the user's face in the current coordinate system corresponding to the current head posture are obtained, and the rotation matrix corresponding to the current coordinate system and the reference coordinate system is obtained. Based on the second coordinates and the rotation matrix, the first coordinates of the multiple feature points in the reference coordinate system are calculated, and the difference between the value of the preset axis in the first coordinate of the first feature point and the value of the preset axis in the first coordinate of the second feature point is calculated. If the absolute value of the difference is determined to be greater than the second threshold value, and the difference is a positive number, the target feature point is determined based on the second area feature point of the second area where the second feature point is located, and the preset third feature point is obtained, and the third feature point is configured as the target feature point. Finally, based on the first coordinate of the target feature point and the pinhole camera model imaging equation, the depth of the user's human eye from the terminal screen is calculated. In the embodiment of the present application, when the difference in the depth of the two human eyes from the terminal screen is large, the feature point closer to the terminal screen is selected as the target feature point to calculate the depth of the human eye from the screen, which can ensure the accuracy of the final measured depth, thereby improving the accuracy of measuring the depth of the human eye from the screen.

[0105] See also Figure 8 , is a flow chart of a depth detection method provided in an embodiment of the present application. Figure 8 As shown, the method of the embodiment of the present application may include the following steps:

[0106] S801, detecting the current head posture of the user to obtain second coordinates of a plurality of feature points in the user's face in a current coordinate system corresponding to the current head posture.

[0107] S802, obtaining a rotation matrix corresponding to the current coordinate system and the reference coordinate system, and calculating first coordinates of a plurality of feature points in the reference coordinate system based on the second coordinates and the rotation matrix.

[0108] S803, calculating a difference between a value of a preset axial direction in the first coordinate of the first feature point and a value of a preset axial direction in the first coordinate of the second feature point.

[0109] Specifically, S801-S803 can be found in Figure 4 The description of S401 - S403 in the illustrated embodiment will not be repeated here.

[0110] S804: If it is determined that the absolute value of the difference is greater than the second threshold value and the difference is a negative number, a target feature point is determined based on the first area feature point of the first area where the first feature point is located, and a preset third feature point is obtained, and the third feature point is configured as the target feature point.

[0111] It is easy to understand that the first threshold and the second threshold can be preset, the second threshold is greater than the first threshold, and both thresholds are positive numbers.

[0112] In some embodiments, if the absolute value of the difference is determined to be greater than the second threshold value and the difference is negative, the first feature point and the third feature point may be determined as target feature points. In this case, the set multiple feature points refer to the first feature point, the second feature point and the third feature point; the third feature point refers to the feature point of the root of the nose described in S401 above.

[0113] Optionally, in another embodiment, if the absolute value of the difference is determined to be greater than the second threshold value, and the difference is a negative number, the first feature point, the feature point belonging to the same eye area as the first feature point (i.e., the first area feature point in the first area where the first feature point is located except the first feature point) and the third feature point can be determined as the target feature point. In this case, the multiple feature points set refer to the first feature point, the feature point belonging to the same eye area as the first feature point, the second feature point, the feature point belonging to the same eye area as the second feature point, and the third feature point. Among them, the third feature point refers to the feature point of the root of the nose described in S401 above. The number of feature points belonging to the same eye area as the first feature point can be one or more, and correspondingly, the number of feature points belonging to the same eye area as the second feature point can be one or more, because the feature point belonging to the same eye area as the first feature point and the feature point belonging to the same eye area as the second feature point can also be feature points whose positions are symmetrical to each other. For example, the feature point belonging to the same eye region as the first feature point is the feature point of the left eye center point, and the feature point belonging to the same eye region as the second feature point is the feature point of the right eye center point.

[0114] S805, substituting the first coordinates of each target feature point into the pinhole camera model imaging equation to obtain the calculation equation corresponding to each target feature point.

[0115] It should be noted that the definition of the pinhole camera model imaging equation can be specifically referred to the description in S103, which will not be repeated here.

[0116] In some embodiments, the first coordinates of the first feature point and the third feature point can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure can be used to obtain two pairs of Figure 3 The calculation equations shown in FIG. One pair of calculation equations can be the equations after the first coordinate of the first feature point is substituted into the calculation. In the pair of equations, X c ' represents the value of the X axis in the first coordinate of the first feature point, Y c ' represents the value of the Y axis in the first coordinate of the first feature point, Z c' represents the value of the Z axis in the first coordinate of the first feature point. Another pair of calculation equations can be the equations after substituting the first coordinate of the third feature point into the calculation. In this pair of equations, X c ' represents the value of the X axis in the first coordinate of the third feature point, Y c ' represents the value of the Y axis in the first coordinate of the third feature point, Z c ' represents the value of the Z axis in the first coordinate of the third feature point.

[0117] Optionally, in another embodiment, when the first feature point, the feature point belonging to the same eye region as the first feature point (i.e., the first region feature point other than the first feature point in the first region where the first feature point is located), and the third feature point are target feature points, the first coordinates of these target feature points can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure gives several pairs of Figure 3 The calculation equation is shown. In each pair of Figure 3 In the calculation equation shown, X c 'Represents the value of the X axis in the first coordinate of each target feature point, Y c 'Represents the value of the Y axis in the first coordinate of each target feature point, Z c 'Represents the Z-axis value of the first coordinate of each target feature point.

[0118] S806: Calculate the depth of the user's eyes from the terminal screen based on all calculation equations.

[0119] In some embodiments, when the first feature point and the third feature point are target feature points, the calculated Figure 3 The two pairs of calculation equations shown in the figure are processed and the following is obtained: Fig. 9 The target equation is shown. Fig. 9 In the target equation shown in the figure, since the third feature point, that is, the feature point of the root of the nose, is defined as the origin in the reference coordinate system, the values ​​of each axis in the first coordinate of the third feature point are all 0, X c '1 represents the value of the X axis in the first coordinate of the first feature point, Y c '1 represents the value of the Y axis in the first coordinate of the first feature point; u3 and v3 represent the values ​​of the two axes of the third feature point in the pixel coordinate system; u1 and v1 represent the values ​​of the two axes of the second feature point in the pixel coordinate system. Fig. 9 It can be seen that the value on the left side of the equation is Z c 2 , we can perform square root operation on this value to get Z c The value of Z c That is, the depth between the user's glasses and the terminal screen.

[0120] Optionally, in another embodiment, when the first feature point, the feature point belonging to the same eye region as the first feature point (i.e., the first region feature point other than the first feature point in the first region where the first feature point is located), and the third feature point are target feature points, the first coordinates of these target feature points can be substituted into Figure 2 After the pinhole camera model imaging equation is shown, all the calculation equations are obtained, and Z can be solved by calculation. c The value of Z c That is, the depth between the user's glasses and the terminal screen.

[0121] In an embodiment of the present application, the current head posture of the user is detected, and the second coordinates of multiple feature points in the user's face in the current coordinate system corresponding to the current head posture are obtained, and the rotation matrix corresponding to the current coordinate system and the reference coordinate system is obtained. Based on the second coordinates and the rotation matrix, the first coordinates of the multiple feature points in the reference coordinate system are calculated, and the difference between the value of the preset axis in the first coordinate of the first feature point and the value of the preset axis in the first coordinate of the second feature point is calculated. If the absolute value of the difference is determined to be greater than the second threshold value, and the difference is negative, the target feature point is determined based on the second area feature point of the second area where the second feature point is located, and the preset third feature point is obtained, and the third feature point is configured as the target feature point. Finally, based on the first coordinate of the target feature point and the pinhole camera model imaging equation, the depth of the user's human eye from the terminal screen is calculated. In the embodiment of the present application, when the difference in the depth of the two human eyes from the terminal screen is large, the feature point closer to the terminal screen is selected as the target feature point to calculate the depth of the human eye from the screen, which can ensure the accuracy of the final measured depth, thereby improving the accuracy of measuring the depth of the human eye from the screen.

[0122] See also Fig.10 , is a flow chart of a depth detection method provided in an embodiment of the present application. Fig.10 As shown, the method of the embodiment of the present application may include the following steps:

[0123] S1001, detecting the current head posture of the user to obtain second coordinates of a plurality of feature points in the user's face in a current coordinate system corresponding to the current head posture.

[0124] S1002, obtaining a rotation matrix corresponding to the current coordinate system and the reference coordinate system, and calculating first coordinates of a plurality of feature points in the reference coordinate system based on the second coordinates and the rotation matrix.

[0125] S1003, calculating a difference between a value of a preset axial direction in the first coordinate of the first feature point and a value of a preset axial direction in the first coordinate of the second feature point.

[0126] Specifically, S1001-S1003 can be found in Figure 4 The description of S401 - S403 in the illustrated embodiment will not be repeated here.

[0127] S1004, if it is determined that the absolute value of the difference is less than or equal to the second threshold, and the absolute value is greater than the first threshold, determine the first target feature point based on the first area feature point of the first area where the first feature point is located, determine the second target feature point based on the second area feature point of the second area where the second feature point is located, and obtain a preset third feature point, and configure the third feature point as the third target feature point.

[0128] It is easy to understand that the first threshold and the second threshold can be preset, the second threshold is greater than the first threshold, and both thresholds are positive numbers.

[0129] In some embodiments, if the absolute value of the difference is determined to be less than or equal to the second threshold value, and the absolute value is greater than the first threshold value, the first feature point can be determined as the first target feature point, the second feature point can be determined as the second target feature point, and the third feature point can be determined as the third target feature point, and the first target feature point, the second target feature point and the third target feature point form the target feature point. In this case, the set multiple feature points refer to the first feature point, the second feature point and the third feature point; the third feature point refers to the feature point of the root of the nose described in S401 above.

[0130] Optionally, in another embodiment, if the absolute value of the difference is determined to be less than or equal to the second threshold value, and the absolute value is greater than the first threshold value, the first feature point and the feature point belonging to the same eye area as the first feature point (the first area feature point in the first area where the first feature point is located except the first feature point) can be determined as the first target feature point, the second feature point and the feature point belonging to the same eye area as the second feature point can be determined as the target feature point (the second area feature point in the second area where the second feature point is located except the second area feature point) can be determined as the target feature point, and the third feature point can be determined as the third target feature point, and the target feature point is composed of the first target feature point, the second target feature point and the third target feature point. In this case, the set multiple feature points refer to the first feature point, the feature point belonging to the same eye area as the first feature point, the second feature point, the feature point belonging to the same eye area as the second feature point, and the third feature point. Among them, the third feature point refers to the feature point of the root of the nose described in S401 above. The number of feature points belonging to the same eye region as the first feature point may be one or more, and correspondingly, the number of feature points belonging to the same eye region as the second feature point may be one or more, because the feature points belonging to the same eye region as the first feature point and the feature points belonging to the same eye region as the second feature point may also be feature points whose positions are symmetrical to each other. For example, if the feature point belonging to the same eye region as the first feature point is the feature point of the center point of the left eye, then the feature point belonging to the same eye region as the second feature point is the feature point of the center point of the right eye.

[0131] S1005: Calculate a first weight and a second weight based on the absolute value, the first threshold and the second threshold.

[0132] In some embodiments, the difference between the absolute value and the first threshold can be calculated to obtain a first difference, and the difference between the second threshold and the first threshold can be calculated to obtain a second difference. Further, the ratio of the first difference to the second difference can be calculated to obtain a first weight. Further, the second weight is equal to the difference between 1 and the first weight.

[0133] S1006: If the difference is a positive number, calculate the first depth based on the first coordinate of the first target feature point, the first coordinate of the second target feature point, and a pinhole camera model imaging equation.

[0134] In the embodiment of the present application, the first target feature point is the first feature point, and the second target feature point is the second feature point. If the difference is a positive number, the first coordinates of the first feature point and the first coordinates of the second feature point can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure can be used to obtain two pairs of Figure 3 The calculation equations shown in FIG. One pair of calculation equations can be the equations after the first coordinate of the first feature point is substituted into the calculation. In the pair of equations, Xc ' represents the value of the X axis in the first coordinate of the first feature point, Y c ' represents the value of the Y axis in the first coordinate of the first feature point, Z c ' represents the value of the Z axis in the first coordinate of the first feature point. Another pair of calculation equations can be the equations after substituting the first coordinate of the second feature point into the calculation. In this pair of equations, X c ' represents the value of the X axis in the first coordinate of the second feature point, Y c ' represents the value of the Y axis in the first coordinate of the second feature point, Z c ' represents the value of the Z axis in the first coordinate of the second feature point. Further, the above two pairs of calculation equations are processed and the following is obtained: Figure 5 The target equation is shown. Figure 5 In the target equation shown, X c '1 represents the value of the X axis in the first coordinate of the first feature point, Y c '1 represents the value of the Y axis in the first coordinate of the first feature point, X c '2 represents the value of the X axis in the first coordinate of the second feature point, Y c '2 represents the value of the Y axis in the first coordinate of the second feature point; u1 and v1 represent the values ​​of the two axes of the first feature point in the pixel coordinate system; u2 and v2 represent the values ​​of the two axes of the second feature point in the pixel coordinate system. Figure 5 It can be seen that the value on the left side of the equation is Z c 2 , we can perform square root operation on this value to get Z c The value of Z c This is the first depth.

[0135] Optionally, in another embodiment, when the first feature point, the feature point belonging to the same eye region as the first feature point, the second feature point, and the feature point belonging to the same eye region as the second feature point are target feature points, if the difference is a positive number, the first coordinates of these target feature points can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure gives several pairs of Figure 3 The calculation equation is shown. In each pair of Figure 3 In the calculation equation shown, X c 'Represents the value of the X axis in the first coordinate of each target feature point, Y c 'Represents the value of the Y axis in the first coordinate of each target feature point, Z c ' represents the value of the Z axis in the first coordinate of each target feature point. Further, after operating the above multiple pairs of calculation equations, Z can be solved c The value of Z c This is the first depth.

[0136] S1007, calculating a second depth based on the first coordinates of the second target feature point, the first coordinates of the third target feature point, and a pinhole camera model imaging equation.

[0137] In the embodiment of the present application, the second target feature point is the second feature point, and the third target feature point is the third feature point. The first coordinates of the second feature point and the first coordinates of the third feature point can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure can be used to obtain two pairs of Figure 3 The calculation equations shown in FIG. One pair of calculation equations can be the equations after the first coordinate of the second feature point is substituted into the calculation. In the pair of equations, X c ' represents the value of the X axis in the first coordinate of the second feature point, Y c ' represents the value of the Y axis in the first coordinate of the second feature point, Z c ' represents the value of the Z axis in the first coordinate of the second feature point. Another pair of calculation equations can be the equations after substituting the first coordinate of the third feature point into the calculation. In this pair of equations, X c ' represents the value of the X axis in the first coordinate of the third feature point, Y c ' represents the value of the Y axis in the first coordinate of the third feature point, Z c ' represents the value of the Z axis in the first coordinate of the third feature point. Further, the above two pairs of calculation equations are processed and the following is obtained: Figure 7 The target equation is shown. Figure 7 In the target equation shown in FIG. 1 , since the third feature point, i.e., the feature point of the root of the nose, is defined as the origin in the reference coordinate system, the values ​​of each axis in the first coordinate of the third feature point are all 0; X c '2 represents the value of the X axis in the first coordinate of the second feature point, Y c '2 represents the value of the Y axis in the first coordinate of the second feature point; u3 and v3 represent the values ​​of the two axes of the third feature point in the pixel coordinate system; u2 and v2 represent the values ​​of the two axes of the second feature point in the pixel coordinate system. Figure 7 It can be seen that the value on the left side of the equation is Z c 2 , we can perform square root operation on this value to get Z c The value of Z c This is the second depth.

[0138] Optionally, in another embodiment, when the second feature point, the feature point belonging to the same eye region as the second feature point, and the third feature point are target feature points, the first coordinates of these target feature points can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure gives several pairs of Figure 3The calculation equation is shown. In each pair of Figure 3 In the calculation equation shown, X c 'Represents the value of the X axis in the first coordinate of each target feature point, Y c 'Represents the value of the Y axis in the first coordinate of each target feature point, Z c ' represents the value of the Z axis in the first coordinate of each target feature point. Further, after operating the above multiple pairs of calculation equations, Z can be solved c The value of Z c This is the second depth.

[0139] S1008 , performing weighted summation based on the first weight, the second weight, the first depth, and the second depth to obtain the depth of the user's eyes from the terminal screen.

[0140] In some embodiments, the first weight can be used as the weight of the second depth, and the second weight can be used as the weight of the first depth. The first depth and the second depth are weighted to obtain a sum value, which can be the depth of the user's eyes from the screen.

[0141] S1009: If the difference is a negative number, calculating a third depth based on the first coordinate of the first target feature point, the first coordinate of the second target feature point, and a pinhole camera model imaging equation.

[0142] Specifically, the third depth is calculated in the same manner as the first depth in S1006 . For details, please refer to the description in S1006 , which will not be repeated here.

[0143] S1010, calculating a fourth depth based on the first coordinates of the first target feature point, the first coordinates of the third target feature point, and a pinhole camera model imaging equation.

[0144] In the embodiment of the present application, the first target feature point is the first feature point, and the third target feature point is the third feature point. The first coordinates of the first feature point and the first coordinates of the third feature point can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure can be used to obtain two pairs of Figure 3 The calculation equations shown in FIG. One pair of calculation equations can be the equations after the first coordinate of the first feature point is substituted into the calculation. In the pair of equations, X c ' represents the value of the X axis in the first coordinate of the first feature point, Y c ' represents the value of the Y axis in the first coordinate of the first feature point, Z c ' represents the value of the Z axis in the first coordinate of the first feature point. Another pair of calculation equations can be the equations after substituting the first coordinate of the third feature point into the calculation. In this pair of equations, X c ' represents the value of the X axis in the first coordinate of the third feature point, Yc ' represents the value of the Y axis in the first coordinate of the third feature point, Z c ' represents the value of the Z axis in the first coordinate of the third feature point. Further, the above two pairs of calculation equations are processed and the following is obtained: Fig. 9 The target equation is shown. Fig. 9 In the target equation shown in the figure, since the third feature point, that is, the feature point of the root of the nose, is defined as the origin in the reference coordinate system, the values ​​of each axis in the first coordinate of the third feature point are all 0, X c '1 represents the value of the X axis in the first coordinate of the first feature point, Y c '1 represents the value of the Y axis in the first coordinate of the first feature point; u3 and v3 represent the values ​​of the two axes of the third feature point in the pixel coordinate system; u1 and v1 represent the values ​​of the two axes of the second feature point in the pixel coordinate system. Figure 5 It can be seen that the value on the left side of the equation is Z c 2 , we can perform square root operation on this value to get Z c The value of Z c This is the second depth.

[0145] Optionally, in another embodiment, when the first feature point, the feature point belonging to the same eye region as the first feature point, and the third feature point are target feature points, the first coordinates of these target feature points can be substituted into Figure 2 The pinhole camera model imaging equation shown in the figure gives several pairs of Figure 3 The calculation equation is shown. In each pair of Figure 3 In the calculation equation shown, X c 'Represents the value of the X axis in the first coordinate of each target feature point, Y c 'Represents the value of the Y axis in the first coordinate of each target feature point, Z c ' represents the value of the Z axis in the first coordinate of each target feature point. Further, after operating the above multiple pairs of calculation equations, Z can be solved c The value of Z c This is the fourth depth.

[0146] S1011, performing weighted summation based on the first weight, the second weight, the third depth, and the fourth depth to obtain the depth of the user's eyes from the terminal screen.

[0147] In some embodiments, the first weight can be used as the weight of the fourth depth, and the second weight can be used as the weight of the third depth. The third depth and the fourth depth are weighted to obtain a sum value, which can be the depth of the user's eyes from the screen.

[0148] In an embodiment of the present application, the user's current head posture is detected to obtain second coordinates of multiple feature points in the user's face in the current coordinate system corresponding to the current head posture, and the rotation matrix corresponding to the current coordinate system and the reference coordinate system is obtained. Based on the second coordinates and the rotation matrix, the first coordinates of the multiple feature points in the reference coordinate system are calculated, and the difference between the numerical value of a preset axis in the first coordinate of the first feature point and the numerical value of a preset axis in the first coordinate of the second feature point is calculated. If it is determined that the absolute value of the difference is less than or equal to the second threshold, and the absolute value is greater than the first threshold, that is, it is determined that the absolute value of the difference is between the larger difference and the smaller difference, all feature points are determined as target feature points, and then the depth of the human eye from the screen is calculated using a linear weighted method, which can ensure the accuracy of the final measured depth, thereby improving the accuracy of measuring the depth of the human eye from the screen.

[0149] See also Fig.11 , is a schematic diagram of the structure of a depth detection device provided in an embodiment of the present application. The depth detection device 1100 can be implemented as all or part of the terminal through software, hardware, or a combination of both. The device 1100 includes:

[0150] A first processing module 1110 is used to detect the current head posture of the user, and obtain first coordinates of a plurality of feature points in the user's face in a reference coordinate system, wherein the reference coordinate system is parallel to the camera coordinate system, and the plurality of feature points include at least a first feature point and a second feature point, wherein the first feature point and the second feature point are symmetrical feature points in two eye regions;

[0151] A second processing module 1120 is used to calculate a difference between a value of a preset axial direction in the first coordinate of the first feature point and a value of a preset axial direction in the first coordinate of the second feature point, and determine a target feature point from the plurality of feature points based on the difference;

[0152] The depth calculation module 1130 is used to calculate the depth of the user's eyes from the terminal based on the first coordinates of the target feature point and the pinhole camera model imaging equation.

[0153] Optionally, the first processing module includes:

[0154] A first processing unit is used to detect the current head posture of the user and obtain second coordinates of a plurality of feature points in the user's face in a current coordinate system corresponding to the current head posture;

[0155] The second processing unit is used to obtain a rotation matrix corresponding to the current coordinate system and the reference coordinate system, and calculate the first coordinates of the multiple feature points in the reference coordinate system based on the second coordinates and the rotation matrix.

[0156] Optionally, the first processing unit includes:

[0157] A first subunit is used to detect the current head posture of the user and obtain an initial rotation angle of the current head posture relative to the initial head posture in the initial coordinate system;

[0158] The second subunit is used to calculate the second coordinates of multiple feature points in the user's face in the current coordinate system corresponding to the current head posture based on the initial rotation angle.

[0159] Optionally, the second processing module includes:

[0160] The first determining unit is configured to determine the first feature point and the second feature point as target feature points if it is determined that the absolute value of the difference is less than or equal to a first threshold, and the first threshold is less than a second threshold.

[0161] Optionally, the second processing module includes:

[0162] A second determining unit is configured to determine a target feature point based on a second area feature point of a second area where the second feature point is located, and obtain a preset third feature point, and configure the third feature point as the target feature point if it is determined that the absolute value of the difference is greater than a second threshold value and the difference is a positive number, and the second threshold value is greater than the first threshold value; or

[0163] A third determination unit is used to determine the target feature point based on the first area feature point of the first area where the first feature point is located, and obtain a preset third feature point, and configure the third feature point as the target feature point if it is determined that the absolute value of the difference is greater than a second threshold and the difference is a negative number, and the second threshold is greater than the first threshold.

[0164] Optionally, the depth calculation module includes:

[0165] A first calculation unit, used for respectively substituting the first coordinates of each of the target feature points into the pinhole camera model imaging equation to obtain a calculation equation corresponding to each of the target feature points;

[0166] The second calculation unit is used to calculate the depth of the user's eyes from the terminal screen based on all calculation equations.

[0167] Optionally, the second processing module includes:

[0168] A fourth determining unit is configured to determine a target feature point based on the plurality of feature points if an absolute value of the difference is less than or equal to a second threshold and the absolute value is greater than a first threshold, wherein the first threshold is less than the second threshold.

[0169] Optionally, the fourth determining unit includes:

[0170] a first determining subunit, configured to determine a first target feature point based on a first regional feature point of a first region where the first feature point is located, determine a second target feature point based on a second regional feature point of a second region where the second feature point is located, and obtain a preset third feature point, and configure the third feature point as a third target feature point, wherein the third feature point is a feature point that does not belong to the same region as the first feature point and does not belong to the same region as the second feature point;

[0171] A first calculation subunit, configured to calculate a first weight and a second weight based on the absolute value, the first threshold and the second threshold;

[0172] a second calculation subunit, configured to calculate a first depth based on the first coordinate of the first target feature point, the first coordinate of the second target feature point, and a pinhole camera model imaging equation if the difference is a positive number;

[0173] A third calculation subunit, configured to calculate a second depth based on the first coordinates of the second target feature point, the first coordinates of the third feature point, and a pinhole camera model imaging equation;

[0174] The fourth calculation subunit is used to perform weighted summation based on the first weight, the second weight, the first depth and the second depth to obtain the depth of the user's eyes from the terminal screen.

[0175] Optionally, the depth detection device further includes:

[0176] a fifth calculation subunit, configured to calculate a third depth based on the first coordinate of the first target feature point, the first coordinate of the second target feature point, and a pinhole camera model imaging equation if the difference is a negative number;

[0177] a sixth calculation subunit, configured to calculate a fourth depth based on the first coordinate of the first target feature point, the first coordinate of the third target feature point, and a pinhole camera model imaging equation;

[0178] The seventh calculation subunit is used to perform weighted summation based on the first weight, the second weight, the third depth and the fourth depth to obtain the depth of the user's eyes from the terminal screen.

[0179] Please refer to Fig.12 , Fig.12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.12 As shown, the electronic device 1200 may include: at least one processor 1201 , at least one network interface 1204 , a user interface 1203 , a memory 1205 , a display screen component 1206 , and at least one communication bus 602 .

[0180] The communication bus 1202 is used to realize the connection and communication between these components.

[0181] The user interface 1203 may include a display screen (Display) and a camera (Camera), and the optional user interface 1203 may also include a standard wired interface and a wireless interface.

[0182] The network interface 1204 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0183] Among them, the processor 1201 may include one or more processing cores. The processor 1201 uses various interfaces and lines to connect various parts of the entire electronic device 1200, and executes various functions and processes data of the electronic device 1200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1205, and calling data stored in the memory 1205. Optionally, the processor 1201 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1201 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1201, and it can be implemented separately through a chip.

[0184] Among them, the memory 1205 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1205 includes a non-transitory computer-readable storage medium. The memory 1205 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1205 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1205 may also be optionally at least one storage device located away from the aforementioned processor 1201. As Fig.12 As shown, the memory 1205 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a program of a depth detection method.

[0185] exist Fig.12 In the electronic device 1200 shown, the user interface 1203 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 1201 can be used to call the program of the depth detection method stored in the memory 1205, and specifically perform the following operations:

[0186] Detecting the current head posture of the user to obtain first coordinates of a plurality of feature points in the user's face in a reference coordinate system, wherein the reference coordinate system is parallel to the camera coordinate system, wherein the plurality of feature points include at least a first feature point and a second feature point, wherein the first feature point and the second feature point are symmetrical feature points in two eye regions;

[0187] Calculate the difference between the value of the preset axial direction in the first coordinate of the first feature point and the value of the preset axial direction in the first coordinate of the second feature point, and determine the target feature point from the plurality of feature points based on the difference;

[0188] Based on the first coordinates of the target feature point and the pinhole camera model imaging equation, the depth of the user's eyes from the terminal screen is calculated.

[0189] In addition, those skilled in the art can understand that the structure of the electronic device 1200 shown in the above figures does not constitute a limitation on the electronic device 1200, and the user terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the electronic device 1200 also includes components such as a radio frequency circuit, an audio circuit, a WiFi component, a power supply, and a Bluetooth component, which will not be described in detail here.

[0190] An embodiment of the present application further provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the depth detection method as described in the above embodiments.

[0191] An embodiment of the present application further provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the depth detection method described in the above embodiments.

[0192] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0193] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A depth detection method, characterized in that: The method comprises: Detecting the current head posture of the user to obtain first coordinates of a plurality of feature points in the user's face in a reference coordinate system, wherein the reference coordinate system is parallel to the camera coordinate system, wherein the plurality of feature points include at least a first feature point and a second feature point, wherein the first feature point and the second feature point are symmetrical feature points in two eye regions; Calculate the difference between the value of the preset axial direction in the first coordinate of the first feature point and the value of the preset axial direction in the first coordinate of the second feature point, and determine the target feature point from the plurality of feature points based on the difference; Based on the first coordinate of the target feature point and the pinhole camera model imaging equation, the depth of the user's eyes from the terminal screen is calculated.

2. The method according to claim 1, characterized in that The user's current head posture is detected to obtain the first coordinates of multiple feature points on the user's face in the reference coordinate system, including: Detecting the current head posture of the user to obtain second coordinates of a plurality of feature points in the user's face in a current coordinate system corresponding to the current head posture; A rotation matrix corresponding to the current coordinate system and the reference coordinate system is obtained, and first coordinates of the plurality of feature points in the reference coordinate system are calculated based on the second coordinates and the rotation matrix.

3. The method according to claim 2, characterized in that The detecting the current head posture of the user to obtain second coordinates of a plurality of feature points in the user's face in a current coordinate system corresponding to the current head posture includes: Detecting the current head posture of the user to obtain an initial rotation angle of the current head posture relative to the initial head posture in the initial coordinate system; Based on the initial rotation angle, second coordinates of a plurality of feature points in the user's face in a current coordinate system corresponding to the current head posture are calculated.

4. The method according to claim 1, characterized in that: The determining a target feature point from among the plurality of feature points based on the difference comprises: If it is determined that the absolute value of the difference is less than or equal to a first threshold, the first feature point and the second feature point are determined as target feature points, and the first threshold is less than the second threshold.

5. The method according to claim 1, characterized in that The determining a target feature point from among the plurality of feature points based on the difference comprises: If it is determined that the absolute value of the difference is greater than the second threshold value, and the difference is a positive number, a target feature point is determined based on a second area feature point of the second area where the second feature point is located, and a preset third feature point is obtained, and the third feature point is configured as the target feature point, and the second threshold value is greater than the first threshold value; or, If it is determined that the absolute value of the difference is greater than the second threshold and the difference is a negative number, the target feature point is determined based on the first area feature point of the first area where the first feature point is located, and a preset third feature point is obtained, and the third feature point is configured as the target feature point, and the second threshold is greater than the first threshold.

6. The method according to any one of claims 1 to 5, characterized in that: The calculating the depth of the user's eyes from the terminal screen based on the first coordinate of the target feature point and the pinhole camera model imaging equation includes: Substituting the first coordinates of each of the target feature points into the pinhole camera model imaging equation to obtain the calculation equation corresponding to each of the target feature points; Based on all the calculation equations, solve the depth of the user's eyes from the terminal screen.

7. The method according to claim 1, characterized in that The determining a target feature point from among the plurality of feature points based on the difference comprises: If the absolute value of the difference is less than or equal to a second threshold, and the absolute value is greater than a first threshold, a target feature point is determined based on the multiple feature points, and the first threshold is less than the second threshold.

8. The method according to claim 7, characterized in that The determining the target feature point based on the multiple feature points comprises: Determine a first target feature point based on a first regional feature point of the first region where the first feature point is located, determine a second target feature point based on a second regional feature point of the second region where the second feature point is located, obtain a preset third feature point, and configure the third feature point as a third target feature point, where the third feature point is a feature point that does not belong to the same region as the first feature point and does not belong to the same region as the second feature point; The calculating the depth of the user's eyes from the terminal screen based on the first coordinate of the target feature point and the pinhole camera model imaging equation includes: Calculating a first weight and a second weight based on the absolute value, the first threshold, and the second threshold; If the difference is a positive number, calculating a first depth based on the first coordinate of the first target feature point, the first coordinate of the second target feature point, and a pinhole camera model imaging equation; Calculate a second depth based on the first coordinate of the second target feature point, the first coordinate of the third feature point, and a pinhole camera model imaging equation; A weighted sum is performed based on the first weight, the second weight, the first depth, and the second depth to obtain the depth of the user's eyes from the terminal screen.

9. The method according to claim 8, characterized in that The method further comprises: If the difference is a negative number, calculating a third depth based on the first coordinate of the first target feature point, the first coordinate of the second target feature point, and a pinhole camera model imaging equation; Calculate a fourth depth based on the first coordinate of the first target feature point, the first coordinate of the third target feature point, and a pinhole camera model imaging equation; A weighted sum is performed based on the first weight, the second weight, the third depth, and the fourth depth to obtain the depth of the user's eyes from the terminal screen.

10. A depth detection device, characterized in that: The device comprises: A first processing module is used to detect the current head posture of the user, and obtain first coordinates of a plurality of feature points in the user's face in a reference coordinate system, wherein the reference coordinate system is parallel to the camera coordinate system, and the plurality of feature points include at least a first feature point and a second feature point, wherein the first feature point and the second feature point are symmetrical feature points in two eye regions; A second processing module, configured to calculate a difference between a value of a preset axial direction in the first coordinate of the first feature point and a value of a preset axial direction in the first coordinate of the second feature point, and determine a target feature point from among the plurality of feature points based on the difference; The depth calculation module is used to calculate the depth of the user's eyes from the terminal based on the first coordinates of the target feature point and the pinhole camera model imaging equation.

11. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 9.

Images