Method, device and storage medium for line of sight estimation based on dual liquid lens

Abstract

Embodiments of the present application provide a line-of-sight estimation method and device based on a binocular liquid lens and a storage medium, the method comprising: determining whether a first distance and a second distance are within a current depth of field of a target binocular camera; if not, adjusting a current of the binocular liquid lens based on a joint adjustment model to make the first distance and the second distance within an adjusted depth of field of the target binocular camera, and adjusting intrinsic parameters of the target binocular camera according to the adjusted current to re-perform line-of-sight estimation. The present application establishes a joint adjustment model to determine the adjusted current of the binocular liquid lens according to the distance from a target point to the binocular optical center, to ensure that the target point is within the adjusted depth of field, and to determine the intrinsic parameters of the target binocular camera by using the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera, and then to re-perform line-of-sight estimation. Thus, panoramic depth eye movement tracking is achieved, and the application and development of liquid lenses in the field of line-of-sight estimation are promoted.

Description

Technical Field

[0001] This invention relates to the field of imaging technology, and in particular to a method, device, and storage medium for estimating line of sight based on a binocular liquid lens. Background Technology

[0002] Eye-tracking technology has become a key tool in various fields, including psychology, neuromarketing, neurocognition, user experience, basic research, and market research, for assessing visual and human behavior. It is also widely recognized as a highly accurate and precise method for estimating gaze using traditional gaze estimation devices. However, traditional gaze estimation devices, based on eye-tracking technology, have fixed parameters after gaze estimation. Furthermore, their interaction distance and working range are limited; effective gaze estimation can only be performed within a narrow, defined area, significantly reducing the user experience.

[0003] A liquid lens uses a liquid as a lens, changing its curvature to alter the focal length. Therefore, mounting a liquid lens in front of the lens of a stereo camera can significantly extend the range and distance of line-of-sight estimation. However, any change in the liquid within the lens, such as movement or a change in tilt angle, will cause optical center drift in the stereo camera with the liquid lens mounted. Using fixed parameters will undoubtedly severely impact the accuracy of line-of-sight estimation, potentially leading to estimation failure and negating the purpose of using a liquid lens to extend the estimated distance and range.

[0004] Therefore, how to enable the intrinsic parameters of a binocular camera equipped with a liquid lens to be dynamically adjusted according to the changes in the liquid lens, so as to ensure the accuracy of line-of-sight estimation, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] In view of the technical problems existing in the prior art, the present invention provides a method, device and storage medium for estimating line of sight based on binocular liquid lens.

[0006] In a first aspect, the present invention provides a line-of-sight estimation method based on a binocular liquid lens, comprising:

[0007] Determine whether the first distance and the second distance are within the current depth of field of the target binocular camera; the first distance is the distance between the target point and the center of gravity of the left eye of the target binocular camera; the second distance is the distance between the target point and the center of gravity of the right eye of the target binocular camera; the target point is the binocular center point determined based on the facial image acquired by the target binocular camera;

[0008] If the distance is not within the current depth of field of the target camera, the current of the binocular liquid lens is adjusted based on the joint adjustment model so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera. The intrinsic parameters of the target binocular camera are adjusted according to the adjusted current, and the line of sight is re-estimated. The binocular liquid lens is disposed at the front end of the lens of the target binocular camera.

[0009] The joint adjustment model is used to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, as well as the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera.

[0010] Optionally, before determining whether the first distance and the second distance are within the current depth of field range of the target stereo camera, the process includes:

[0011] Based on the generalized depth estimation model and the key points of the eyes in the facial image, the z-axis coordinates of the center points of both eyes in the coordinate system of the left eye camera are determined and used as the z-axis coordinates of the target point.

[0012] Based on the transformation relationship of the three-dimensional coordinates in the left eye camera coordinate system and the z-axis coordinate of the target point, the three-dimensional coordinates of the target point in the left eye camera coordinate system are determined;

[0013] Based on the three-dimensional coordinates of the target point in the coordinate system of the left eye camera, the distance between the optical axes of the target binocular camera and the distance between the optical centers, the distance between the target point and the left eye center of the target binocular camera, and the distance between the target point and the right eye center of the target binocular camera are determined, and are respectively used as the first distance and the second distance.

[0014] The formula for the optical center distance is:

[0015] Among them, s L The distance s represents the distance between the target point and the left eye center of the target binocular camera. R d' represents the distance between the target point and the right optical center of the target binocular camera, d' represents the distance between the optical axes of the target binocular camera, and x0, y0, and z0 represent the three-dimensional coordinates of the target point in the coordinate system of the left eye camera.

[0016] The generalized depth estimation model is used to determine the z-axis coordinate of any point in the left-eye camera coordinate system based on the coordinates of any point in the left-eye image pixel coordinate system and the intrinsic parameters of the target binocular camera.

[0017] Optionally, if the distance is not within the current depth of field of the target camera, then based on the joint adjustment model, the current of the binocular liquid lens is adjusted so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera, and the intrinsic parameters of the target binocular camera are adjusted according to the adjusted current, and the line of sight estimation is performed again, including:

[0018] Based on the distance-current relationship model, and the first distance and the second distance, the adjusted current of the binocular liquid lens is determined; the adjusted current includes the adjusted current of the left eye lens of the target binocular camera and the adjusted current of the right eye lens of the target binocular camera.

[0019] Based on the current camera intrinsic parameter relationship model and the current after adjustment of the binocular liquid lens, the adjusted intrinsic parameters of the target binocular camera are determined.

[0020] Based on the camera depth-of-field model and the adjusted intrinsic parameters of the target binocular camera, the adjusted depth-of-field range of the target binocular camera is determined.

[0021] The joint adjustment model includes the distance-current relationship model, the current-camera intrinsic parameter relationship model, and the camera depth-of-field model;

[0022] The current camera intrinsic parameter relationship model is used to represent the correspondence between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera.

[0023] Optionally, determining the adjusted current of the binocular liquid lens based on the distance-current relationship model, and the first distance and the second distance, includes:

[0024] Based on the first correlation in the distance-current relationship model, the current of the left eye lens of the binocular liquid lens after adjustment is determined;

[0025] Based on the second correlation in the second distance and the distance-current relationship model, the current of the right eye lens of the binocular liquid lens after adjustment is determined;

[0026] The formula corresponding to the first association relationship is:

[0027] The formula corresponding to the second association relationship is:

[0028] Among them, I L I represents the current of the left eye lens. R The current s represents the current of the right eye lens. L The distance s represents the distance between the target point and the left eye center of the stereo camera. RThis represents the distance between the target point and the right eye center of the stereo camera, where a1, a2, a3, and a4 represent s. L to I L The relationship coefficients, b1, b2, b3, b4 represent s R to I R The relationship coefficient.

[0029] Optionally, the method for determining the intrinsic parameter relationship model of the current camera includes:

[0030] A third correlation relationship is established between the intrinsic parameter matrix of the left eye camera in the target binocular camera, the current of the left eye lens, the horizontal coordinate of the principal point of the left eye camera during calibration, the vertical offset of the principal point of the left eye camera, and the first constant coefficient. A fourth correlation relationship is established between the intrinsic parameter matrix of the right eye camera in the target binocular camera, the current of the right eye lens, the horizontal coordinate of the principal point of the right eye camera during calibration, the vertical offset of the principal point of the right eye camera, and the second constant coefficient. This relationship serves as the current camera intrinsic parameter relationship model.

[0031] By calibration, a first correspondence between the principal point abscissa of the left eye camera and the calibration current of the left eye camera, a second correspondence between the principal point abscissa of the right eye camera and the calibration current of the right eye camera, as well as the first constant coefficient and the second constant coefficient are determined.

[0032] Optionally, determining the adjusted intrinsic parameters of the target binocular camera based on the current camera intrinsic parameter relationship model and the current adjusted by the binocular liquid lens includes:

[0033] Based on linear interpolation and the correspondence between the principal point abscissa and the calibration current, the first calibration current corresponding to the current of the left eye lens in the current after the binocular liquid lens is adjusted, and the second calibration current corresponding to the current of the right eye lens in the current after the binocular liquid lens is adjusted.

[0034] Based on the first calibration current, the current of the left eye lens in the current after adjustment of the binocular liquid lens, and the intrinsic parameter relationship model of the current camera, the intrinsic parameters of the left eye camera after adjustment are determined.

[0035] Based on the second calibration current, the current of the right eye lens in the current after adjustment of the binocular liquid lens, and the intrinsic parameter relationship model of the current camera, the intrinsic parameters of the right eye camera after adjustment are determined.

[0036] The correspondence between the principal point abscissa and the calibration current includes a first correspondence between the principal point abscissa of the left eye camera and the calibration current of the left eye camera, and a second correspondence between the principal point abscissa of the right eye camera and the calibration current of the right eye camera.

[0037] Optionally, the offset of the principal point of the left eye camera along the vertical direction is determined based on the input current of the left eye lens and the angle between the left eye lens and the vertical direction; the offset of the principal point of the right eye camera along the vertical direction is determined based on the input current of the right eye lens and the angle between the right eye lens and the numerical direction.

[0038] Optionally, the generalized depth estimation model is further used to determine the z-axis coordinate of any point in the left-eye camera coordinate system based on the coordinates of any point in the right-eye image pixel coordinate system and the intrinsic parameters of the target binocular camera. The corresponding determination method includes:

[0039] Based on the rotation matrix R and translation matrix T of the right camera coordinate system relative to the left camera coordinate system, the coordinates of any point in the right camera coordinate system, and the transformation formula, the coordinates of the point in the left camera coordinate system are determined; the coordinates of the point in the right camera coordinate system are determined based on the coordinates of the point in the right image pixel coordinate system.

[0040] The conversion formula is C' = R -1 ·(CT), where C represents the coordinates of any point in the right eye camera coordinate system; C' represents the coordinates of the point in the left eye camera coordinate system; R represents the rotation matrix of the right eye camera coordinate system relative to the left eye camera coordinate system; and T represents the translation matrix of the right eye camera coordinate system relative to the left eye camera coordinate system.

[0041] In a second aspect, the present invention also provides a line-of-sight estimation device based on a binocular liquid lens, comprising:

[0042] The determination module is used to determine whether a first distance and a second distance are within the current depth of field range of the target binocular camera; the first distance is the distance between the target point and the center of gravity of the left eye of the target binocular camera; the second distance is the distance between the target point and the center of gravity of the right eye of the target binocular camera; the target point is the binocular center point determined based on the facial image acquired by the target binocular camera;

[0043] An adjustment module is used to adjust the current of the binocular liquid lens so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera if they are not within the current depth of field range of the target camera, based on a joint adjustment model, and adjust the intrinsic parameters of the target binocular camera according to the adjusted current to re-estimate the line of sight; the binocular liquid lens is disposed at the front end of the lens of the target binocular camera;

[0044] The joint adjustment model is used to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, as well as the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera.

[0045] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the line-of-sight estimation method based on binocular liquid lenses as described in the first aspect above.

[0046] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the line-of-sight estimation method based on binocular liquid lenses as described in the first aspect above.

[0047] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the binocular liquid lens-based line-of-sight estimation method as described in the first aspect above.

[0048] This invention provides a binocular liquid lens-based gaze estimation method, device, and storage medium. Through an established joint adjustment model, it determines the adjusted current of the binocular liquid lens based on the distance from the target to the binocular eye's center. Then, by utilizing the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera, it determines the intrinsic parameters of the target binocular camera. Finally, based on the adjusted intrinsic parameters of the target binocular camera, it determines the adjusted depth of field. In other words, through the established joint adjustment model, the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera are dynamically adjusted together based on the distance from the target to the binocular eye's center, ensuring that the target point is within the adjusted depth of field and guaranteeing the accuracy of gaze estimation by the target binocular camera. This enables full depth-of-field eye tracking, promoting the application and development of liquid lenses in the field of gaze estimation. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a schematic flowchart of the line-of-sight estimation method based on binocular liquid lenses provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the statistical results of the relationship between the binocular liquid lens and the focus position calibration provided in the embodiments of the present invention;

[0052] Figure 3 This is a schematic diagram of the statistical results of experimental data on the current and total focal length of the binocular liquid lens provided in an embodiment of the present invention;

[0053] Figure 4This is a schematic diagram of the device for implementing the line-of-sight estimation method based on binocular liquid lenses provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the gaze estimation principle based on a 3D eyeball model provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the line-of-sight estimation device based on a binocular liquid lens provided in an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0058] In this embodiment of the invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Figure 1 This is a schematic flowchart of the line-of-sight estimation method based on a binocular liquid lens provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0061] Step 101: Determine whether the first distance and the second distance are within the current depth of field of the target binocular camera; the first distance is the distance between the target point and the center of gravity of the left eye of the target binocular camera; the second distance is the distance between the target point and the center of gravity of the right eye of the target binocular camera; the target point is the center point of both eyes determined based on the facial image acquired by the target binocular camera;

[0062] Step 102: If the distance is not within the current depth of field of the target camera, then based on the joint adjustment model, adjust the current of the binocular liquid lens so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera, and adjust the intrinsic parameters of the target binocular camera according to the adjusted current, and re-estimate the line of sight; the binocular liquid lens is set at the front end of the lens of the target binocular camera.

[0063] The joint adjustment model is used to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, as well as the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera.

[0064] Specifically, by setting a binocular liquid lens on the viewfinder side of the binocular camera, the distance and range of gaze estimation can be greatly expanded, enabling full depth-of-field eye tracking. Each adjustment to the binocular liquid lens causes a drift in the light center of the image captured by the binocular camera through the liquid lens, resulting in a blurry final image. Therefore, when adjusting the binocular liquid lens, it is also necessary to adjust the intrinsic parameters of the binocular camera accordingly. That is, by adjusting the current of the binocular liquid lens while simultaneously adjusting the intrinsic parameters of the binocular camera, a clear imaging result can be obtained, ensuring the accuracy of gaze estimation. For the sake of clarity in describing the present invention, the terms "target binocular camera" and "binocular camera" have the same meaning.

[0065] This invention establishes the relationship between the current of the binocular liquid lens and the distance between the optical center of the binocular camera and any point in space, as well as the relationship between the current of the binocular liquid lens and the intrinsic parameters of the binocular camera (target binocular camera) equipped with the binocular liquid lens, through the study of a binocular camera with a binocular liquid lens installed. This is a joint adjustment model, and the relevant constant coefficients in the joint adjustment model are determined through calibration.

[0066] Based on this, the images acquired by the target binocular camera are initially screened to obtain valid facial images. For example, the validity of the acquired facial images is judged by whether the facial images are complete and clear, and whether the image size occupies more than 1 / 5 of the entire imaging area. Distortion correction is then performed on the valid facial images to locate the Region of Interest (ROI) of the eyes. Within the ROI, the pixel coordinates of the pupil and light spot are detected. Based on the position of the principal point of the left and right eye camera systems of the target binocular camera, the 3D coordinates of each image point are calculated and unified to the coordinates of the left eye camera. Finally, based on the joint adjustment model, the current of the binocular liquid lens and the intrinsic parameters of the binocular camera are adjusted.

[0067] Thus, if the center point of both eyes in the facial image acquired by the target binocular camera is not within the current depth of field of the target binocular camera, the current of the binocular liquid lens needs to be adjusted according to the joint adjustment model provided by the present invention. At the same time, the intrinsic parameters of the target binocular camera are adjusted according to the magnitude of the current to ensure the accuracy of the line of sight estimation of the target binocular camera.

[0068] This invention provides a gaze estimation method based on a binocular liquid lens. Through an established joint adjustment model, it determines the adjusted current of the binocular liquid lens based on the distance from the target to the binocular eye's center. Then, by utilizing the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera, it determines the intrinsic parameters of the target binocular camera. Finally, based on the adjusted intrinsic parameters of the target binocular camera, it determines the adjusted depth of field. In other words, by establishing a joint adjustment model, the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera are dynamically adjusted together based on the distance from the target to the binocular eye's center, ensuring that the target point is within the adjusted depth of field and guaranteeing the accuracy of gaze estimation by the target binocular camera. This enables full depth-of-field eye tracking, promoting the application and development of liquid lenses in the field of gaze estimation.

[0069] Optionally, before determining whether the first distance and the second distance are within the current depth of field range of the target stereo camera, the process includes:

[0070] Based on the generalized depth estimation model and the key points of the eyes in the facial image, the z-axis coordinates of the center points of both eyes in the coordinate system of the left eye camera are determined and used as the z-axis coordinates of the target point.

[0071] Based on the transformation relationship of the three-dimensional coordinates in the left eye camera coordinate system and the z-axis coordinate of the target point, the three-dimensional coordinates of the target point in the left eye camera coordinate system are determined;

[0072] Based on the three-dimensional coordinates of the target point in the coordinate system of the left eye camera, the distance between the optical axes of the target binocular camera and the distance between the optical centers, the distance between the target point and the left eye center of the target binocular camera, and the distance between the target point and the right eye center of the target binocular camera are determined, and are respectively used as the first distance and the second distance.

[0073] The formula for the optical center distance is:

[0074] Among them, s L The distance s represents the distance between the target point and the left eye center of the target binocular camera. R d' represents the distance between the target point and the right optical center of the target binocular camera, d' represents the distance between the optical axes of the target binocular camera, and x0, y0, and z0 represent the three-dimensional coordinates of the target point in the coordinate system of the left eye camera.

[0075] The generalized depth estimation model is used to determine the z-axis coordinate of any point in the left-eye camera coordinate system based on the coordinates of any point in the left-eye image pixel coordinate system and the intrinsic parameters of the target binocular camera.

[0076] Specifically, after acquiring the corresponding image through the target binocular camera, the depth estimation generalized model is first used to estimate the depth of the key points of the eyes in the effective facial image, and the z-axis coordinate of the center point of the eyes in the left eye camera coordinate system of the target binocular camera is determined. This depth estimation generalized model can be expressed as:

[0077]

[0078] Where z represents the Z-axis coordinate of any point Q in the left-eye camera coordinate system, and f L This indicates the focal length of the left-eye camera, in mm, f. R The focal length of the right eye camera is indicated in mm, and d represents the baseline distance of the target binocular cameras. x Represents the physical size of a pixel in the x-direction (assuming the image sensor models for the left and right cameras are the same), in mm / pixel, u L This represents the coordinates of point Q on the u-axis of the left eye image pixel coordinate system, in pixels (pixels). R This represents the coordinates of point Q on the u-axis of the right eye image pixel coordinate system, in pixels; u L0 This represents the coordinates of the principal point of the left-eye camera in the left-eye pixel coordinate system, u. R0 This represents the coordinates of the principal point of the right-eye camera in the right-eye pixel coordinate system. The physical dimension of the pixel in the x-direction here can be understood as the actual physical distance of a single pixel along the x-axis, denoted as d. x .

[0079] When any point Q in the generalized depth estimation model is the target point, i.e. the center point of both eyes, the determined z-axis coordinate is the z-axis coordinate of the target point in the coordinate system of the left eye camera.

[0080] Based on the transformation relationship of the three-dimensional coordinates in the left-eye camera coordinate system and the z-axis coordinate of the target point mentioned above, the three-dimensional coordinates of the target point in the left-eye camera coordinate system are determined. The formula corresponding to the transformation relationship of the three-dimensional coordinates in the left-eye camera coordinate system can be expressed as:

[0081]

[0082] Where x0, y0, z0 represent the three-dimensional coordinates of point Q in the left-eye camera coordinate system, z represents the Z-axis coordinate of point Q in the left-eye camera coordinate system, and u L This represents the coordinates of the imaging point of point Q in the left eye image on the u-axis of the left eye image pixel coordinate system, u L0 This represents the coordinates of the principal point of the left-eye camera on the u-axis of the left-eye pixel coordinate system, vL This represents the coordinates of the imaging point Q in the left eye image on the v-axis of the left eye image pixel coordinate system; v L0 f represents the coordinates of the principal point of the left-eye camera on the v-axis of the left-eye pixel coordinate system. L d represents the focal length of the left-eye camera. x d represents the physical size of a pixel in the x-direction. y This represents the physical size of a pixel in the y-direction. The physical size of a pixel in the y-direction can be understood as the actual physical distance of a single pixel along the y-axis, which is d. y .

[0083] When point Q is the target point, i.e. the center point of both eyes, the three-dimensional coordinates determined according to the transformation relationship of the three-dimensional coordinates in the left eye camera coordinate system are the three-dimensional coordinates of the target point in the left eye camera coordinate system.

[0084] Then, based on the optical center distance formula, the distance between the target point and the left optical center of the target binocular camera, and the distance between the target point and the right optical center of the target binocular camera, are determined. This optical center distance formula can be expressed as:

[0085]

[0086] Where, d'=d+(u L0 -u R0 )d x ;s L s represents the distance between the target point and the left eye center of the target binocular camera. R Let x0, y0, and z0 represent the distance between the target point and the right optical center of the target stereo camera, respectively. Let d', y0, and z0 represent the three-dimensional coordinates of the target point in the left eye camera coordinate system. Let d' represent the distance between the optical axes (left and right optical centers) of the target stereo camera, and d represent the baseline distance of the target stereo camera. x Represents the physical size of a pixel in the x-direction, u L0 This represents the coordinates of the principal point of the left-eye camera on the u-axis of the left-eye pixel coordinate system, u R0 This indicates the coordinates of the principal point of the right camera on the u-axis of the left camera pixel coordinate system.

[0087] The distance s between the target point and the left eye center of the target binocular camera is determined using the method described above. L And the distance s between the target point and the right eye center of the target binocular camera. R .

[0088] Optionally, if the distance is not within the current depth of field of the target camera, then based on the joint adjustment model, the current of the binocular liquid lens is adjusted so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera, and the intrinsic parameters of the target binocular camera are adjusted according to the adjusted current, and the line of sight estimation is performed again, including:

[0089] Based on the distance-current relationship model, and the first distance and the second distance, the adjusted current of the binocular liquid lens is determined; the adjusted current includes the adjusted current of the left eye lens of the target binocular camera and the adjusted current of the right eye lens of the target binocular camera.

[0090] Based on the current camera intrinsic parameter relationship model and the current after adjustment of the binocular liquid lens, the adjusted intrinsic parameters of the target binocular camera are determined.

[0091] Based on the camera depth-of-field model and the adjusted intrinsic parameters of the target binocular camera, the adjusted depth-of-field range of the target binocular camera is determined.

[0092] The joint adjustment model includes the distance-current relationship model, the current-camera intrinsic parameter relationship model, and the camera depth-of-field model.

[0093] Specifically, the aforementioned joint depth-of-field adjustment model includes a distance-current relationship model, a current-camera intrinsic parameter relationship model, and a camera depth-of-field model.

[0094] The distance between the target point and the left eye center of the target binocular camera (first distance), and the distance between the target point and the right eye center of the target binocular camera (second distance) are input into the distance-current relationship model, and the corresponding currents after adjustment of the left eye lens and the right eye lens in the binocular liquid lens are output respectively.

[0095] Changes in the binocular lens current will also lead to corresponding changes in the intrinsic parameters of the binocular camera. Specifically, the adjusted intrinsic parameters of the binocular camera are determined based on the current-camera intrinsic parameter relationship model. This model represents the correspondence between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera; that is, it represents the relationship between the intrinsic parameters of the left eye camera and the current of the left eye lens, and the relationship between the intrinsic parameters of the right eye camera and the current of the right eye lens. The relevant parameters in this relationship are determined through calibration.

[0096] Thus, given the current after the binocular lens is adjusted, the intrinsic parameters of the target binocular camera corresponding to the current after the binocular liquid lens is adjusted can be determined by linear interpolation, i.e., the intrinsic parameters of the target binocular camera after adjustment are obtained.

[0097] Then, based on the camera depth-of-field model and the adjusted intrinsic parameters of the target binocular camera, the adjusted depth-of-field range of the target binocular camera is determined. The distance between the target point and the left eye center of the target binocular camera and the distance between the target point and the right eye center of the target binocular camera must be within the adjusted depth-of-field range.

[0098] The formula corresponding to the above camera depth-of-field model can be expressed as:

[0099]

[0100] Where, Δs L Δs R Let S represent the distance between the target point and the left eye center of the target binocular camera. L The distance between the target point and the right eye center of the target binocular camera is S. R In this case, the depth of field of the left camera and the depth of field of the right camera, Δs L1 Δs R1 Indicates a deep foreground, Δs L2 Δs R2 F indicates depth of field. L F R The aperture value is represented by δ, which represents the allowable circle of confusion diameter, and S... L S R These represent the distances between the target point and the left eye center of the target binocular camera, and the distances between the target point and the right eye center of the target binocular camera, respectively, when the binocular liquid lens is at its current current (before adjusting the binocular liquid lens current). If s L ∈(S L -Δs L1 ,S L +Δs L2 ), indicating that the target point is within the depth of field of the left eye; similarly, if s R ∈(S R -Δs R1 ,S R +Δs R2 This indicates that the target point is within the depth of field of the right camera. Only when the target point is simultaneously within the depth of field of both cameras is it unnecessary to adjust the binocular lens current; otherwise, the current of the binocular lenses needs to be adjusted simultaneously based on the distance-current relationship model. Determining whether the target point is within the depth of field is primarily to avoid frequent changes in lens current and ensure the speed of line-of-sight estimation.

[0101] Optionally, determining the adjusted current of the binocular liquid lens based on the distance-current relationship model, and the first distance and the second distance, includes:

[0102] Based on the first correlation in the distance-current relationship model, the current of the left eye lens of the binocular liquid lens after adjustment is determined;

[0103] Based on the second correlation in the second distance and the distance-current relationship model, the current of the right eye lens of the binocular liquid lens after adjustment is determined;

[0104] The formula corresponding to the first association relationship is:

[0105] The formula corresponding to the second association relationship is:

[0106] Among them, I L I represents the current of the left eye lens. R The current s represents the current of the right eye lens. L The distance s represents the distance between the target point and the left eye center of the stereo camera. R This represents the distance between the target point and the right eye center of the stereo camera, where a1, a2, a3, and a4 represent s. L to I L The relationship coefficients, b1, b2, b3, b4 represent s R to I R The relationship coefficient.

[0107] Specifically, the distance-current relationship model is used to represent the relationship between the distance (first distance) between the target point and the left eye center of the target binocular camera, and the current of the left eye liquid lens, as well as the relationship between the distance (second distance) between the target point and the right eye center of the target binocular camera, and the current of the right eye liquid lens.

[0108] The relationship between the first distance and the current of the left eye liquid lens, also known as the first correlation, and the relationship between the second distance and the current of the right eye liquid lens, also known as the second correlation, can be expressed as follows:

[0109]

[0110] Among them, I L I represents the current of the left eye lens. R The current s represents the current of the right eye lens. L s represents the distance between the target and the left eye center of the target's binocular camera. R This represents the distance between the target and the right eye center of the target's binocular camera; a1 / a2 / a3 / a4 represent s. L to I L The relationship coefficients, b1 / b2 / b3 / b4, represent s R to I R The relationship coefficient.

[0111] The relationship coefficients in the first and second association relationships mentioned above are determined through a calibration method, specifically including:

[0112] Based on the imaging results of the target binocular camera, obtain the working range of the target binocular camera's line of sight, for example, the minimum working distance is L. min The maximum working distance is L maxThe minimum working distance represents the closest distance required to ensure complete facial image imaging, while the maximum working distance represents a pupil diameter of at least 12 pixels in the image.

[0113] Get the preset number of calibration positions n s-i The calibration step size is determined to be w. s-i And satisfy n s-i Take the integer part;

[0114] First, place the calibration object at the initial calibration position set with the minimum working distance. Manually adjust the left and right lens currents to make the target clear, and record the current value of the first set of currents as I. L1 and I R1 Simultaneously, the coordinates of the calibration object in the physical coordinate system of the left eye image are obtained. Based on the optical center distance formula, the distances between the calibration object and the left optical center, and between the calibration object and the right optical center, were calculated, and the first set of distance values ​​was recorded as s. L1 and s R1 The calibration object is moved according to the step size w s-i Move along the z-axis of the left eye camera coordinate system and repeat the above operation to obtain the correspondence between different current value groups and distance groups; use these calibration values ​​to determine the values ​​of relationship coefficients a1, a2, a3, a4 and b1, b2, b3, b4.

[0115] After determining the relationship coefficients in the first and second correlation relationships using the above calibration method, the adjusted currents corresponding to the left and right eye lenses in the binocular liquid lens can be determined based on the distance between any point and the center of the left eye in the target binocular camera, and the distance between that point and the center of the right eye in the target binocular camera.

[0116] Optionally, the method for determining the intrinsic parameter relationship model of the current camera includes:

[0117] A third correlation relationship is established between the intrinsic parameter matrix of the left eye camera in the target binocular camera, the current of the left eye lens, the horizontal coordinate of the principal point of the left eye camera during calibration, the vertical offset of the principal point of the left eye camera, and the first constant coefficient. A fourth correlation relationship is established between the intrinsic parameter matrix of the right eye camera in the target binocular camera, the current of the right eye lens, the horizontal coordinate of the principal point of the right eye camera during calibration, the vertical offset of the principal point of the right eye camera, and the second constant coefficient. This relationship serves as the current camera intrinsic parameter relationship model.

[0118] By calibration, a first correspondence between the principal point abscissa of the left eye camera and the calibration current of the left eye camera, a second correspondence between the principal point abscissa of the right eye camera and the calibration current of the right eye camera, as well as the first constant coefficient and the second constant coefficient are determined.

[0119] Specifically, in order to realize the corresponding change of the intrinsic parameters of the target binocular camera based on the change of the liquid lens current, the current camera intrinsic parameter relationship model proposed in this invention is established, which can be expressed as:

[0120]

[0121]

[0122] Among them, A L A R f represents the intrinsic parameter matrices of the left and right cameras. Lx f Ly This represents the normalized focal length along the u-axis and v-axis in the physical coordinate system of the left eye image. L0 v L0 This represents the u-axis and v-axis coordinates of the left principal point of the target stereo camera in the image pixel coordinate system, f. Rx f Ry This represents the normalized focal length along the u-axis and the normalized focal length along the v-axis in the physical coordinate system of the right eye image. R0 v R0 This represents the u-axis and v-axis coordinates of the right principal point of the target stereo camera in the image pixel coordinate system. L I R U represents the current of the left eye lens and the current of the right eye lens. L0i+1 u L0i Let u represent the x-coordinates of the principal point of the left eye camera during the (i+1)th and i-th calibrations, respectively. L0 value, u R0i+1 u R0i Let u represent the x-coordinates of the principal point of the right eye camera during the (i+1)th and i-th calibrations, respectively. R0 The value of I Li+1 I Li Let I represent the current I of the left eye camera during the (i+1)th and i-th calibrations, respectively. L The value of I Ri+1 I Ri Let I represent the current I of the right eye camera during the (i+1)th and i-th calibrations, respectively. R The value of α L1 α L2 α L3 β L1 β L2 β L3 α represents the first constant coefficient. R1 α R2 α R3 β R1 β R2 β R3 Denotes the second constant coefficient, Δy L ΔyR This indicates the vertical offset of the principal point of the left camera and the vertical offset of the principal point of the right camera.

[0123] The vertical offset of the principal point of the left eye camera is determined based on the input current of the left eye lens and the angle between the left eye lens and the vertical direction; the vertical offset of the principal point of the right eye camera is determined based on the input current of the right eye lens and the angle between the right eye lens and the numerical direction, and the corresponding formula is expressed as follows:

[0124]

[0125] Where, Δy L Δy R C represents the vertical offset of the principal point of the left eye camera and the vertical offset of the principal point of the right eye camera in a stereo camera system, respectively. L1 C L2 C L3 C represents the unknown constant coefficient of the left eye camera in a stereo camera setup. R1 C R2 C R3 θ represents the unknown constant coefficient of the right camera in a stereo camera setup. L θ represents the angle between the left eye liquid lens and the vertical direction. R θ represents the angle between the right eye liquid lens and the vertical direction; for binocular liquid lenses, θ L =θ R =θ, where θ represents the angle between the optical axis of the binocular camera and the horizontal direction; I L I R This indicates the current of the left eye liquid lens and the current of the right eye liquid lens.

[0126] First, the vertical offset of the principal point of the left eye camera and the vertical offset of the principal point of the right eye camera in a binocular camera system are calibrated. Specific methods include:

[0127] 1) Fix the target binocular camera on the electric flipping platform so that the optical axis of the target binocular camera is parallel to the platform plane, and fix a reference object in front of the binocular liquid lens, such that the plane of the reference object is perpendicular to the optical axis of the target binocular camera and the reference object is within the field of view of the target binocular camera.

[0128] When the optical axis of the target binocular camera is parallel to the platform plane, the angle θ between the optical axis of the target binocular camera and the horizontal direction is 0°.

[0129] 2) Assume the number of times the current of the left and right liquid lenses is adjusted is n. y When θ = 0°, the liquid lens current of the left eye changes from I. L1 Adjust to Step length Right eye liquid lens current from I R1 Adjust to Step length Calculate the pixel coordinates of the reference object on the image at each current adjustment, record the corresponding current, the vertical offset of the left and right eye camera principal points, the angle between the optical axis of the binocular camera and the horizontal direction, and other relevant data; here, the adjustment range of the left eye liquid lens current is... and the adjustment range of the liquid lens current for the right eye. When calibrating with the above distance-current relationship model, the current group (I) determined at the minimum working distance. L1 ,I R1 ), and the current group determined at the maximum working distance. correspond.

[0130] 3) Preset the angle step size wθ. By adjusting the electric flipping platform, the angle θ between the optical axis of the target binocular camera and the horizontal direction increases by one step each time. Repeat step 2 until θ = 90°.

[0131] 4) By fitting the calibration data of the left and right eye liquid lenses, the unknown constant coefficients C of the left and right eye cameras in the binocular camera can be calculated. L1 C L2 C L3 C R1 C R2 C R3 The value;

[0132] After calibrating the offset using the above method, it is also necessary to calibrate the first and second constant coefficients in the current camera intrinsic parameter relationship model. Specific methods include:

[0133] 1) Prepare several checkerboard calibration plates with different unit lengths. Adjust the current of the left and right eye liquid lenses to 0mA (initial position). Manually adjust the focus ring to make the target binocular camera focus clearly at the same position.

[0134] 2) Using the same current step size w Li =w Ri Synchronously adjust the left and right liquid lenses, and acquire n samples in each state. c (10≤n c ) Valid calibration image, minimum calibration current not greater than min(I) L1 ,I R1 The maximum rated current is not less than This means that the current range calibrated by the current camera's intrinsic parameters must include the minimum and maximum values ​​of the calibrated current during the offset calibration mentioned above. A valid calibration image is one that is fully displayed and clear, occupying more than 1 / 5 of the entire display field of view.

[0135] 3) Solve the camera intrinsic parameters for each set of images acquired by the left and right cameras under different current conditions of the binocular liquid lens using the Zhang Zhengyou method, and record the calibration data. The calibration data here includes the calibration current of the left eye liquid lens, the coordinates of the principal point of the left eye camera on the u-axis in the image pixel coordinate system, the calibration current of the right eye liquid lens, and the coordinates of the principal point of the right eye camera on the u-axis in the image pixel coordinate system.

[0136] 4) Use the least squares method to fit and solve the coefficients of each term in the intrinsic parameter expression; that is, determine the first constant coefficient α. L1 α L2 α L3 β L1 β L2 β L3 and the second constant coefficient α R1 α R2 α R3 β R1 β R2 β R3 .

[0137] During the calibration process of the current camera intrinsic parameter relationship model described above, the abscissa u of the principal point of the left eye camera will also be determined. L0i and the calibration current I of the left eye camera Li The first correspondence between them, the x-coordinate of the principal point of the right eye camera, u R0i And the calibration current I of the right eye camera Ri The second correspondence between them.

[0138] Optionally, determining the adjusted intrinsic parameters of the target binocular camera based on the current camera intrinsic parameter relationship model and the current adjusted by the binocular liquid lens includes:

[0139] Based on linear interpolation and the correspondence between the principal point abscissa and the calibration current, the first calibration current corresponding to the current of the left eye lens in the current after the binocular liquid lens is adjusted, and the second calibration current corresponding to the current of the right eye lens in the current after the binocular liquid lens is adjusted.

[0140] Based on the first calibration current, the current of the left eye lens in the current after adjustment of the binocular liquid lens, and the intrinsic parameter relationship model of the current camera, the intrinsic parameters of the left eye camera after adjustment are determined.

[0141] Based on the second calibration current, the current of the right eye lens in the current after adjustment of the binocular liquid lens, and the intrinsic parameter relationship model of the current camera, the intrinsic parameters of the right eye camera after adjustment are determined.

[0142] The correspondence between the principal point abscissa and the calibration current includes a first correspondence between the principal point abscissa of the left eye camera and the calibration current of the left eye camera, and a second correspondence between the principal point abscissa of the right eye camera and the calibration current of the right eye camera.

[0143] Specifically, based on the correspondence between the principal point's abscissa and the calibration current, a linear interpolation method is used to determine the first calibration current corresponding to the current of the left eye lens in the adjusted binocular liquid lens current, and the second calibration current corresponding to the current of the right eye lens in the adjusted binocular liquid lens current. This can be understood as determining the current I of the left eye lens. L Which I corresponds to? Li+1 and I Li Within this range, that is, determining the first calibration current, and then, based on the first correspondence between the principal point abscissa of the left eye camera and the calibration current of the left eye camera, determining the principal point abscissa of the left eye camera corresponding to the first calibration current, that is, determining u. L0i+1 and u L0i Then, using the current camera intrinsic parameter relationship model A proposed in this invention... L This allows us to determine the internal parameters f after the left eye camera is adjusted. Lx f Ly u L0 v L0 Similarly, the adjusted intrinsic parameters of the right eye camera are determined using the same steps as those used to determine the adjusted intrinsic parameters of the left eye camera.

[0144] After determining the intrinsic parameters of the target binocular camera using the above method, the focal length f of the left eye camera is... L The internal parameters f can be adjusted based on the left eye camera as described above. Lx f Ly Determined, for example, the method of weighted summation. And a + b = 1. Similarly, the focal length f of the right eye camera... R The internal parameters f can be adjusted based on the left eye camera as described above. Rx f Ry It can be determined that the specific method used can also be weighted summation.

[0145] Optionally, the generalized depth estimation model is further used to determine the z-axis coordinate of any point in the left-eye camera coordinate system based on the coordinates of any point in the right-eye image pixel coordinate system and the intrinsic parameters of the target binocular camera. The corresponding determination method includes:

[0146] Based on the rotation matrix R and translation matrix T of the right camera coordinate system relative to the left camera coordinate system, the coordinates of any point in the right camera coordinate system, and the transformation formula, the coordinates of the point in the left camera coordinate system are determined; the coordinates of the point in the right camera coordinate system are determined based on the coordinates of the point in the right image pixel coordinate system.

[0147] The conversion formula is C' = R -1 ·(CT), where C represents the coordinates of any point in the right eye camera coordinate system; C' represents the coordinates of the point in the left eye camera coordinate system; R represents the rotation matrix of the right eye camera coordinate system relative to the left eye camera coordinate system; and T represents the translation matrix of the right eye camera coordinate system relative to the left eye camera coordinate system.

[0148] Specifically, the aforementioned gaze estimation method based on binocular liquid lenses primarily involves determining the coordinates of image points in the acquired effective facial images within the left-eye camera coordinate system. However, the coordinates of image points in the right-eye camera coordinate system may not be directly used for gaze estimation. Therefore, it is necessary to transform the coordinates of image points in the right-eye camera coordinate system to the left-eye camera coordinate system. This can be achieved by using the rotation matrix R and translation matrix T of the right-eye camera coordinate system relative to the left-eye camera coordinate system, the coordinates of any point in the right-eye image physical coordinate system, and the transformation formula to determine the coordinates of any point in the left-eye image coordinate system. The coordinates of the image point in the right-eye camera coordinate system are then determined based on the coordinates of that image point in the right-eye image pixel coordinate system.

[0149] The conversion formula is C' = R -1 ·(CT), where C represents the coordinates of any point in the right eye camera coordinate system; C' represents the coordinates of the point in the left eye camera coordinate system; R represents the rotation matrix of the right eye camera coordinate system relative to the left eye camera coordinate system; and T represents the translation matrix of the right eye camera coordinate system relative to the left eye camera coordinate system.

[0150] The formulas for the rotation matrix R and translation matrix T of the right camera coordinate system relative to the left camera coordinate system are as follows:

[0151]

[0152] Where R represents the rotation matrix of the right camera coordinate system relative to the left camera coordinate system, R i T represents the rotation matrix of the right camera coordinate system relative to the left camera coordinate system obtained during the i-th calibration, m represents the number of calibrations, and T represents the rotation matrix of the right camera coordinate system relative to the left camera coordinate system. X T Y T Z T represents the translation of the right camera coordinate system relative to the left camera coordinate system along the X, Y, and Z axes, respectively. Xi T Yi T Zi Let d' represent the translation of the right camera coordinate system relative to the left camera coordinate system along the X, Y, and Z axes, respectively, obtained during the i-th calibration. Let v' represent the distance between the optical axes of the binocular cameras. L0 v R0Let d represent the v-axis coordinates of the left and right camera principal points in the image pixel coordinate system, respectively. y I represents the physical size of a pixel in the y-direction. Li I represents the calibration current of the left eye lens in the binocular liquid lens during the i-th calibration. L This represents the current in the left eye lens of a binocular liquid lens.

[0153] Furthermore, the current camera intrinsic parameter relationship model proposed in this invention can also adjust the distortion coefficient of the target binocular camera.

[0154] The relationship between camera distortion coefficients and image point offset can be expressed as:

[0155]

[0156] Camera distortion coefficient:

[0157]

[0158] In the formula, ε U ε V This represents the offset of the image point along the u-axis and v-axis in the image pixel coordinate system. U and V represent the coordinates of the center points of both eyes under ideal conditions. k1 and k2 represent the radial distortion coefficients, and p1 and p2 represent the tangential distortion coefficients. L1 k L2 p represents the radial distortion coefficient of the left camera. L1 p L2 k represents the tangential distortion coefficient of the left camera. R1 k R2 p represents the radial distortion coefficient of the right camera. L1 p L2 G represents the tangential distortion coefficient of the right camera. L1 ~g L12 g represents the constant coefficient of the left camera. R1 ~g R12 f represents the constant coefficient of the right camera. Lm f Rm Indicates the normalized focal length of the left and right cameras;

[0159] When calibrating the intrinsic parameters of the current camera using the current camera intrinsic parameter relationship model, the constant coefficient g of the aforementioned left camera can also be determined. L1 ~g L12 The constant coefficient g of the right camera R1 ~g R12 .

[0160] This invention provides a gaze estimation method based on a binocular liquid lens. Through an established joint adjustment model, it determines the adjusted current of the binocular liquid lens based on the distance from the target to the binocular eye's center. Then, using the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera, it determines the intrinsic parameters of the target binocular camera and re-estimates the gaze, determining the adjusted depth of field. In other words, by establishing a joint adjustment model, the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera are dynamically adjusted together based on the distance from the target to the binocular eye's center, ensuring that the target point is within the adjusted depth of field and guaranteeing the accuracy of the target binocular camera's gaze estimation. This enables full depth-of-field eye tracking, promoting the application and development of liquid lenses in the field of gaze estimation.

[0161] To more clearly illustrate the line-of-sight estimation method based on binocular liquid lenses provided in the embodiments of the present invention, specific examples are given below.

[0162] Figure 2 This is a schematic diagram illustrating the statistical results of the relationship between the binocular liquid lens and the focus position calibration provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the black dots represent the calibration data, which includes the focusing coordinates and the current corresponding to the binocular liquid lens. The curve represents the fitting result based on the calibration data. When the distance from the center point of both eyes to the center of the left and right eyes of the target binocular camera gradually increases from 0.65m to 1.7m, the focusing current also gradually decreases from 20.7mA to -47.9mA, and the decreasing trend, i.e., the gradient, becomes increasingly smaller, showing a very clear trend relationship. In practical applications, the distance change is continuous, so the current change should also be continuous rather than quantitative. If interpolation is used, there will be a certain deviation between the current control and the theoretical value. Therefore, curve fitting using the calibration data can make the liquid lens current focusing control more accurate at different distances. According to the calculation results, for the calibration method provided in this embodiment of the invention, the accuracy of the trinomial fitting can reach 99.8%, with coefficients from high order to low order being -23.06561, 132.25490, -268.77715, and 145.82714. Although using higher-order polynomials can slightly increase the fitting accuracy, the improvement is minimal and will significantly increase the complexity of the model.

[0163] Based on spherical geometry and the lens focal length formula, the expression for the focal length of a binocular liquid lens is as follows:

[0164]

[0165] Among them, f opto R represents the focal length of a binocular liquid lens. e The radius of curvature of the flexible thin film in the binocular liquid lens is represented by n, and n0 represents the air refractive index. optoR represents the refractive index of the liquid optical medium in a binocular liquid lens. h h represents the light transmission radius of the flexible thin-film optical cavity. max C1 and C2 represent constants, where C1 and C2 represent the height of the center of the flexible thin film surface of the binocular liquid lens.

[0166] Therefore, for an optical focusing system based on a binocular liquid lens, applying the combined focal length formula, the total focal length of the system is as follows:

[0167]

[0168] In the formula, f fix Indicates a fixed focal length for the lens, d of C3 and C4 represent the physical distance from the plane of the binocular liquid lens to the optical center of the lens, respectively. The total focal length of the lens system is inversely proportional to the input circuitry.

[0169] Figure 3 This is a schematic diagram illustrating the statistical results of experimental data on the relationship between the current and total focal length of a binocular liquid lens provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the focal length of a standard lens is 8mm. During calibration, the initial position (i.e., the distance from the center of the sharp target to the lens when the current is 0) is determined to be approximately 24cm. The current is increased every 2mA from 0mA, or decreased every 2mA from 0mA, and a certain number of calibration board images are acquired after each adjustment (considering the possibility of images of poor quality, the initial number of images acquired is greater than or equal to 30). The left and right eye liquid lenses are adjusted synchronously with the same current. The calibration current ranges from -22mA to 12mA, for a total of 18 groups. The higher the current value, the closer the focusing distance; conversely, the lower the current value, the farther the focusing distance. When the current value is positive, it corresponds to the binocular liquid lens convex outward; the larger the value, the greater the curvature of the convexity. Conversely, when the current value is negative, it corresponds to the binocular liquid lens concave inward; the larger the absolute value, the greater the curvature of the concavity. Due to the field of view, various calibration plates can be used. The appropriate calibration plate size is selected based on the distance between the imaging point and the target binocular camera. For example, a small calibration plate with a size of 12*9mm and a checkerboard size of 3*3mm can be used for close-range imaging; a large calibration plate with a size of 12*9mm and a checkerboard size of 15*15mm can be used for long-range imaging. If the imaging point is at an even greater distance, a larger calibration plate can be used.

[0170] from Figure 3 It can be seen that the rectangular and circular points represent the focal lengths of the left optical system in the x and y directions when the current of the left eye lens is adjusted, respectively; the equilateral and inverted triangles represent the focal lengths of the right optical system in the x and y directions when the current of the right eye lens is adjusted, respectively. Adjusting the lens current will affect the focal length of the binocular camera. When the current is adjusted from small to large, the focal length shows a clear increasing trend and the change is relatively consistent. Therefore, the relationship can be represented by a model. Figure 3Points with large offsets or abrupt changes may be due to the limited number of calibration images collected. Such abnormal data is common in the general calibration process, but it will not affect the final model fitting. The focal length of the right target is larger than that of the left target, which may be due to manual adjustment error of the initial focus position of a normal lens.

[0171] Although binocular liquid lenses are mounted parallel to the front of the lens during use, different mounting positions may result in corresponding pitch angles. Furthermore, due to gravity, the curved surface of the binocular liquid lens will shift downwards to some extent, causing drift of the principal point of the optical system. Considering the influence of pitch angle on the binocular liquid lens can improve the estimation accuracy of the corneal curvature center. Therefore, current adjustment has a significant impact on the v-axis pixel coordinates of the system's principal point. Thus, after the offset model is calibrated, a first-order polynomial is designed to improve the robustness of this coordinate direction to current.

[0172] According to the imaging principle, adjusting the current of the binocular liquid lens has virtually no impact on the coordinates of the principal point along the u-axis in the image pixel coordinate system. However, according to test statistics, the u-axis coordinates of the principal point still fluctuate irregularly under different currents. Since the intrinsic parameters of the binocular camera are unknown, calibration is generally the only reliable standard. Therefore, to calibrate the impact of this fluctuation error on the line-of-sight estimation calculation, the u-axis coordinates of the principal point in the image pixel coordinate system are calculated using interpolation. Since the camera is generally used in indoor environments, the influence of ambient temperature on the input current is not considered, i.e., temperature calibration of the current is not performed.

[0173] Based on the imaging principle and calibration test results, when the current of the binocular liquid lens changes, the parameters in the rotation matrix R of the right eye camera coordinate system relative to the left eye camera coordinate system remain essentially unchanged. Therefore, R is the average of m calibration data. When the current of the binocular liquid lens is adjusted, the parameters in the offset matrix T... X The value of T is determined by the mean of the calibration data and the mean of the distance d' between the optical axes of the binocular cameras. This reflects the dynamic change of the value with current adjustment while avoiding large-scale adjustments. Y The value is determined by the mean of the calibration data and the mean of the difference between the left and right optical centers in the y-direction; T Z The coordinates are calculated using linear interpolation based on the calibration data. Compared to using fixed data, this method allows for a more accurate transformation of points from the right camera coordinate system to the left camera coordinate system.

[0174] Figure 4 This is a schematic diagram of the apparatus for implementing the binocular liquid lens-based line-of-sight estimation method provided in an embodiment of the present invention, as shown below. Figure 4As shown, the entire device is a line-of-sight estimation device. 1 is a binocular camera with a binocular liquid lens mounted at the center of the device, and 2 are infrared light sources symmetrically mounted at both ends of the device. The binocular liquid lens is generally fixed to the front end of the lens. This is a typical embodiment of the present invention, but it is also applicable to binocular single-light source devices.

[0175] Assume the current in the left eye lens is I. L0 At that time, after calibration, the 3D coordinates of the left and right light sources relative to the left eye coordinate system are (x... Lg ,y Lg ,z Lg ), (x Rg ,y Rg ,z Rg When the current of the left eye lens changes, the coordinates of the light source also change due to the drift of the optical center coordinates of the binocular camera. When the current of the left eye lens is adjusted to I... L At that time, the offset of the optical center coordinates (Δx) Lop ,Δy Lop ,Δz Lop As shown in the following formula:

[0176]

[0177] At this point, the coordinates of the left and right light sources should be:

[0178]

[0179] The above formula is the calculation method for the coordinates of the left and right light sources when the current of the binocular liquid lens is adjusted, (x' Lg ,y' Lg ,z' Lg (x') represents the new coordinates of the left light source. Rg ,y' Rg ,z' Rg The new coordinates of the right light source are shown in the figure. By considering the effect of lens current adjustment on the initial calibration coordinates of the light source, the optical axis estimation can be made more accurate.

[0180] Figure 5 This is a schematic diagram of the gaze estimation principle based on a 3D eyeball model provided in this embodiment of the invention. Point O represents the center of the eyeball, point c represents the center of corneal curvature, point p represents the center of the pupil, L1 and L2 represent the left and right light sources, O1 and O2 represent the optical centers of the left and right liquid lens optical focusing systems, and q represents the center of the eyeball. 11 q 12 q 21 q 22 p1 and p2 represent the reflection points of the left and right light sources on the surface of the pupil, respectively, and the image points on the left and right camera target surfaces, respectively, where p1 and p2 are the refraction points of the pupil center through the corneal surface. 11 u12 u 21 u 22 These represent the imaging points on the left and right camera target surfaces, indicating the reflection point of the light source. The coordinates of these points are in a binocular coordinate system. According to the law of reflection, the light source, camera optical center, reflection point, and corneal curvature center can be determined to lie on the same plane, which can be defined as follows:

[0181]

[0182] The 3D coordinates of the corneal curvature center c can be obtained by optimizing the above formula. Based on the fact that the incident ray, refracted ray, and normal are coplanar when light refracts, the following formula can be obtained:

[0183]

[0184] The above formula can be used to calculate The vector is the optical axis. However, the coordinates of all points involved in the simultaneous calculations of the two sets of formulas above must be unified to the left-eye coordinate system. In this case, O1 is the origin of the coordinate system, with coordinates (0,0,0). If O2, p2, u 21 u 22 The 3D coordinates in the right eye coordinate system are respectively The 3D coordinates in the left-eye coordinate system are respectively but:

[0185]

[0186] The aforementioned optical axis vectors are in the stereo coordinate system and need to be transformed to the screen's world coordinate system. Assuming that, after calibration, the rotation matrix of the stereo camera coordinate system relative to the world coordinate system is R... S The translation matrix is ​​T S Then the optical axis vector in the world coordinate system In actual calculations, vectors need to be converted into corresponding matrix forms. Let the horizontal and vertical angles of the eye's optical axis be φ. he φ ve Its value can be calculated from the unit vector of the optical axis.

[0187] The line of sight and the optical axis have a kappa angle. After calibration, let its horizontal and vertical components be respectively... and The unit vector of the view axis can then be expressed as: The point where the visual axis intersects the screen is the fixation point.

[0188] Figure 6 This is a schematic diagram of the line-of-sight estimation device based on a binocular liquid lens provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the device includes:

[0189] The determining module 601 is used to determine whether the first distance and the second distance are within the current depth of field range of the target binocular camera; the first distance is the distance between the target point and the center of gravity of the left eye of the target binocular camera; the second distance is the distance between the target point and the center of gravity of the right eye of the target binocular camera; the target point is the center point of both eyes determined based on the facial image acquired by the target binocular camera;

[0190] The adjustment module 602 is used to adjust the current of the binocular liquid lens so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera if they are not within the current depth of field range of the target camera, based on a joint adjustment model, and adjust the intrinsic parameters of the target binocular camera according to the adjusted current, and re-estimate the line of sight; the binocular liquid lens is disposed at the front end of the lens of the target binocular camera.

[0191] The joint adjustment model is used to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, as well as the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera.

[0192] The line-of-sight estimation device based on a binocular liquid lens provided in this embodiment of the invention can execute the technical solution of the line-of-sight estimation method based on a binocular liquid lens in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the line-of-sight estimation method based on a binocular liquid lens. Please refer to the implementation principle and beneficial effects of the line-of-sight estimation method based on a binocular liquid lens. It will not be repeated here.

[0193] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a gaze estimation method based on a binocular liquid lens, the method including:

[0194] Determine whether the first distance and the second distance are within the current depth of field of the target binocular camera; the first distance is the distance between the target point and the center of gravity of the left eye of the target binocular camera; the second distance is the distance between the target point and the center of gravity of the right eye of the target binocular camera; the target point is the binocular center point determined based on the facial image acquired by the target binocular camera;

[0195] If the distance is not within the current depth of field of the target camera, the current of the binocular liquid lens is adjusted based on the joint adjustment model so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera. The intrinsic parameters of the target binocular camera are adjusted according to the adjusted current, and the line of sight is re-estimated. The binocular liquid lens is disposed at the front end of the lens of the target binocular camera.

[0196] The joint adjustment model is used to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, as well as the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera.

[0197] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the gaze estimation method based on binocular liquid lenses provided by the above methods, the method comprising:

[0199] Determine whether the first distance and the second distance are within the current depth of field of the target binocular camera; the first distance is the distance between the target point and the center of gravity of the left eye of the target binocular camera; the second distance is the distance between the target point and the center of gravity of the right eye of the target binocular camera; the target point is the binocular center point determined based on the facial image acquired by the target binocular camera;

[0200] If the distance is not within the current depth of field of the target camera, the current of the binocular liquid lens is adjusted based on the joint adjustment model so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera. The intrinsic parameters of the target binocular camera are adjusted according to the adjusted current, and the line of sight is re-estimated. The binocular liquid lens is disposed at the front end of the lens of the target binocular camera.

[0201] The joint adjustment model is used to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, as well as the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera.

[0202] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the line-of-sight estimation method based on binocular liquid lenses provided by the methods described above, the method comprising:

[0203] Determine whether the first distance and the second distance are within the current depth of field of the target binocular camera; the first distance is the distance between the target point and the center of gravity of the left eye of the target binocular camera; the second distance is the distance between the target point and the center of gravity of the right eye of the target binocular camera; the target point is the binocular center point determined based on the facial image acquired by the target binocular camera;

[0204] If the distance is not within the current depth of field of the target camera, the current of the binocular liquid lens is adjusted based on the joint adjustment model so that the first distance and the second distance are within the adjusted depth of field of the target binocular camera. The intrinsic parameters of the target binocular camera are adjusted according to the adjusted current, and the line of sight is re-estimated. The binocular liquid lens is disposed at the front end of the lens of the target binocular camera.

[0205] The joint adjustment model is used to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, as well as the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera.

[0206] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0207] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for line of sight estimation based on dual-element liquid lens, characterized in that, The method comprises the following steps: determining whether the first distance and the second distance are within the current depth of field range of the target binocular camera; the first distance is the distance between the target point and the left eye optical center of the target binocular camera; the second distance is the distance between the target point and the right eye optical center of the target binocular camera; the target point is the center of the eyes determined based on the face image obtained by the target binocular camera; if the target camera is not within the current depth of field range, adjusting the current of the binocular liquid lens based on a joint adjustment model so that the first distance and the second distance are within the adjusted depth of field range of the target binocular camera, and adjusting the intrinsic parameters of the target binocular camera according to the adjusted current to re-estimate the line of sight; the binocular liquid lens is arranged at the front end of the lens of the target binocular camera; the joint adjustment model is used to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, and the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera; if the target camera is not within the current depth of field range, adjusting the current of the binocular liquid lens based on a joint adjustment model so that the first distance and the second distance are within the adjusted depth of field range of the target binocular camera, and adjusting the intrinsic parameters of the target binocular camera according to the adjusted current to re-estimate the line of sight, comprising: determining the adjusted current of the binocular liquid lens based on a distance-current relationship model and the first distance and the second distance; the adjusted current includes the adjusted current of the left eye lens of the target binocular camera and the adjusted current of the right eye lens of the target binocular camera; determining the adjusted intrinsic parameters of the target binocular camera based on a current-intrinsic parameter relationship model and the adjusted current of the binocular liquid lens; determining the adjusted depth of field range of the target binocular camera based on a camera depth of field model and the adjusted intrinsic parameters of the target binocular camera; the joint adjustment model comprises the distance-current relationship model, the current-intrinsic parameter relationship model and the camera depth of field model; the current-intrinsic parameter relationship model is used to represent the corresponding relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera; the determination of the adjusted current of the binocular liquid lens based on the distance-current relationship model and the first distance and the second distance comprises: determining the adjusted current of the left eye lens of the binocular liquid lens based on the first distance and the first correlation relationship in the distance-current relationship model; determining the adjusted current of the right eye lens of the binocular liquid lens based on the second distance and the second correlation relationship in the distance-current relationship model; The formula corresponding to the first association relationship is: The formula corresponding to the second association relationship is: wherein I L represents the current of the left eye lens, I R represents the current of the right eye lens, s L represents the distance between the target point and the left eye optical center of the target binocular camera, s R represents the distance between the target point and the right eye optical center of the target binocular camera, a1, a2, a3, a4 represent s L the relationship coefficient of I L , b1, b2, b3, b4 represent s R the relationship coefficient of I R . 2.The binocular liquid lens based line of sight estimation method of claim 1, wherein, before the determination of whether the first distance and the second distance are within the current depth of field range of the target binocular camera, comprising: determining the z-axis coordinate of the center of the eyes in the left eye camera coordinate system as the z-axis coordinate of the target point based on a depth estimation generalized model and the eye key points of the face image; determine the three-dimensional coordinates of the target point in the left camera coordinate system based on the conversion relationship of the three-dimensional coordinates in the left camera coordinate system and the z-axis coordinate of the target point; determine the distance between the target point and the left optical center of the target binocular camera and the distance between the target point and the right optical center of the target binocular camera as the first distance and the second distance respectively based on the three-dimensional coordinates of the target point in the left camera coordinate system, the distance between the optical axes of the target binocular camera and the optical center distance formula; the optical center distance formula is: wherein s L represents the distance between the target point and the left eye optical center of the target binocular camera, s R represents the distance between the target point and the right eye optical center of the target binocular camera, d' represents the distance between the optical axes of the target binocular camera, x0, y0, z0 represent the three-dimensional coordinates of the target point in the left eye camera coordinate system; the depth estimation generalized model is used to determine the z-axis coordinate of any point in the left camera coordinate system based on the coordinates of the point in the left image pixel coordinate system and the intrinsic parameters of the target binocular camera. 3.The binocular liquid lens based line of sight estimation method of claim 1, wherein, The method for determining the current camera intrinsic parameter relationship model comprises: establish a third correlation relationship between the intrinsic parameter matrix of the left camera in the target binocular camera, the current of the left lens, the horizontal coordinate of the principal point of the left camera during calibration, the offset of the principal point of the left camera along the vertical direction and the first constant coefficient, and a fourth correlation relationship between the intrinsic parameter matrix of the right camera in the target binocular camera, the current of the right lens, the horizontal coordinate of the principal point of the right camera during calibration, the offset of the principal point of the right camera along the vertical direction and the second constant coefficient, as the current camera intrinsic parameter relationship model; determine the first correspondence relationship between the horizontal coordinate of the principal point of the left camera and the calibration current of the left camera, the second correspondence relationship between the horizontal coordinate of the principal point of the right camera and the calibration current of the right camera, and the first constant coefficient and the second constant coefficient through calibration.

4. The vergence estimation method based on dual-element liquid lens according to claim 3, wherein, The determination of the adjusted intrinsic parameters of the target binocular camera based on the current camera intrinsic parameter relationship model and the adjusted current of the binocular liquid lens comprises: determine the first calibration current corresponding to the current of the left lens in the adjusted current of the binocular liquid lens and the second calibration current corresponding to the current of the right lens in the adjusted current of the binocular liquid lens based on the linear interpolation method and the correspondence relationship between the horizontal coordinate of the principal point and the calibration current; determine the adjusted intrinsic parameters of the left camera based on the first calibration current, the current of the left lens in the adjusted current of the binocular liquid lens and the current camera intrinsic parameter relationship model; determine the adjusted intrinsic parameters of the right camera based on the second calibration current, the current of the right lens in the adjusted current of the binocular liquid lens and the current camera intrinsic parameter relationship model; the correspondence relationship between the horizontal coordinate of the principal point and the calibration current comprises the first correspondence relationship between the horizontal coordinate of the principal point of the left camera and the calibration current of the left camera and the second correspondence relationship between the horizontal coordinate of the principal point of the right camera and the calibration current of the right camera. 5.The line-of-sight estimation method based on binocular liquid lens according to claim 3, wherein, The offset of the principal point of the left camera along the vertical direction is determined based on the input current of the left lens and the included angle between the left lens and the vertical direction; and the offset of the principal point of the right camera along the vertical direction is determined based on the input current of the right lens and the included angle between the right lens and the vertical direction. 6.The line-of-sight estimation method based on binocular liquid lens according to claim 2, wherein, The depth estimation generalized model is also used to determine the z-axis coordinate of the arbitrary point in the left camera coordinate system based on the coordinates of the arbitrary point in the right image pixel coordinate system and the intrinsic parameters of the target binocular camera, and a corresponding determination method comprises the following steps: determining the coordinates of the arbitrary point in the left camera coordinate system based on the rotation matrix R, the translation matrix T, the coordinates of the arbitrary point in the right camera coordinate system and the conversion formula; the coordinates of the arbitrary point in the right camera coordinate system are determined based on the coordinates of the arbitrary point in the right image pixel coordinate system; The conversion formula is C' = R -1 ·(C-T), where C represents the coordinates of any point in the right-eye camera coordinate system; C' represents the coordinates of the any point in the left-eye camera coordinate system; R represents the rotation matrix of the right-eye camera coordinate system relative to the left-eye camera coordinate system; and T represents the translation matrix of the right-eye camera coordinate system relative to the left-eye camera coordinate system.

7. A line of sight estimation device based on a dual-element liquid lens, characterized in that, comprise: The determination module is configured to determine whether the first distance and the second distance are within the current depth of field range of the target binocular camera. The first distance is the distance between the target point and the optical center of the left eye of the target binocular camera; the second distance is the distance between the target point and the optical center of the right eye of the target binocular camera; the target point is the center of the eyes determined based on the face image obtained by the target binocular camera; The adjustment module is configured to, if the first distance and the second distance are not within the current depth of field range of the target camera, adjust the current of the binocular liquid lens based on a joint adjustment model so that the first distance and the second distance are within the adjusted depth of field range of the target binocular camera, and adjust the intrinsic parameters of the target binocular camera according to the adjusted current to re-perform the line-of-sight estimation; the binocular liquid lens is arranged at the front end of the lens of the target binocular camera. The joint adjustment model is configured to represent the relationship between the current of the binocular liquid lens and the first distance and the second distance, and the relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera. The adjustment module is specifically configured to: determine the adjusted current of the binocular liquid lens based on a distance-current relationship model and the first distance and the second distance; the adjusted current comprises the adjusted current of the left lens of the target binocular camera and the adjusted current of the right lens of the target binocular camera; determine the adjusted intrinsic parameters of the target binocular camera based on a current-intrinsic parameter relationship model and the adjusted current of the binocular liquid lens; determine the adjusted depth of field range of the target binocular camera based on a camera depth of field model and the adjusted intrinsic parameters of the target binocular camera; The joint adjustment model comprises the distance-current relationship model, the current-intrinsic parameter relationship model and the camera depth of field model. The current-intrinsic parameter relationship model is configured to represent the corresponding relationship between the current of the binocular liquid lens and the intrinsic parameters of the target binocular camera; and the adjustment module is specifically configured to: determine the adjusted current of the left lens of the binocular liquid lens based on the first distance and a first association relationship in the distance-current relationship model; determine the adjusted current of the right lens of the binocular liquid lens based on the second distance and a second association relationship in the distance-current relationship model. The formula corresponding to the first association relationship is: The formula corresponding to the second association relationship is: wherein I L represents the current of the left eye lens, I R represents the current of the right eye lens, s L represents the distance between the target point and the left eye optical center of the target binocular camera, s R represents the distance between the target point and the right eye optical center of the target binocular camera, a1, a2, a3, a4 represent s L the relationship coefficient of I L , b1, b2, b3, b4 represent s R the relationship coefficient of I R .

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the binocular liquid lens-based line-of-sight estimation method of any one of claims 1 to 6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the line-of-sight estimation method based on a binocular liquid lens as claimed in any one of claims 1 to 6.

Images