Improved photographing module of gaze direction measurement device

Abstract

The application discloses an improved photographing module of a visual light direction measuring device, which comprises an adjusting assembly and an image collecting assembly installed on the adjusting assembly, and the image collecting assembly at least comprises the photographing module, wherein the adjusting assembly is used for adjusting the spatial position of the photographing module, the photographing module is provided with a first camera in the center, and a measuring unit for emitting light to the outside and receiving light is arranged on both sides of the first camera, no matter whether the optical axis of the first camera in the photographing module intersects with the pupil center of the user's eyes, the two measuring units installed on the photographing module can judge a point on the surface of a high light surface object, and the positional relationship between the first measuring unit simplified model and the second measuring unit simplified model is similar to the positional relationship between the pupil center of the user's eyes and the first measuring unit and the second measuring unit when the user's eyes gaze at the first camera.

Description

Technical Field

[0001] This invention relates to the field of eye-tracking technology, and more specifically to an improved photographic module for a gaze pointing measurement device. Background Technology

[0002] Based on patent 202110609915.2 (Eye Pointing Measurement Device and Eye Pointing Model Establishment Method), an eye pointing measurement device is described, including an adjustment component and an image acquisition component mounted on the adjustment component. The image acquisition component includes at least a photographing module. The adjustment component is used to adjust the spatial position of the photographing module. Figure 1 , Figure 2 The camera module is used to collect information about the user's eyes and generate a light pointing model. The eye pointing model is used in the field of eye tracking.

[0003] The usage, principle, and function of the aforementioned camera module are roughly as follows:

[0004] The user controls their eyeballs to look directly at the first camera in the imaging module. Simultaneously, the eye-pointing measurement device adjusts the spatial position and angle of the imaging module so that the optical axis of the first camera intersects with the center of the pupil of one of the user's eyes. An infrared point light source then emits infrared light outwards. At the same time, a second camera in the imaging module captures the reflected light from all the emitted light that reaches the user's cornea, obtaining the highlight points of the reflected light in the captured image. Based on the obtained image, the relative positional relationship between the imaging module and the user's cornea (i.e., the center point of the pupil) is calculated. This method is used to generate a light-pointing model, thereby achieving a facial and eye model for eye tracking purposes.

[0005] This patent has the following shortcomings:

[0006] 1. When the optical axis of the first camera does not intersect with the center of the pupil due to large equipment error, the straight line of the path of the light reflected by the center of the pupil and into the second camera is calculated based on the point position calculation method to form the light angle. However, the straight line may not intersect with the optical axis of the first camera lens. Therefore, it is impossible to form a right triangle, and thus it is impossible to determine the relative positional relationship between the center of the pupil and the camera module.

[0007] When the optical axis of the first camera is small due to equipment error and approximately intersects the center of the pupil, the straight line along the path of the light reflected from the center of the pupil and into the second camera, calculated using the point position calculation method, approximately intersects the optical axis of the first camera lens. Therefore, it can only approximately form a right triangle, but not a true triangle. The distance from the center of the pupil to the focal point of the first camera obtained from the approximate triangle is not accurate enough, meaning the positional relationship between the pupil midpoint and the imaging module is not precise enough.

[0008] Specifically, the error value of the calculated distance from the pupil center point to the focal point of the first camera will be amplified exponentially by the distance between the optical axis of the first camera and the center of the pupil after the position is adjusted, i.e., the intersection error value. This ultimately leads to errors in the gaze pointing model, and consequently, errors in eye tracking based on the gaze pointing model.

[0009] 2. The shape of the operator's cornea cannot be measured using this gaze-pointing measurement device and gaze-pointing model establishment method. Summary of the Invention

[0010] The purpose of this invention is to provide an improved photo module for a gaze pointing measurement device. Regardless of whether the optical axis of the first camera in the photo module intersects with the center of the user's pupil, the two measurement units installed on the photo module can determine a point on the surface of a high-gloss object (eyeball). The positional relationship between this point and the simplified models of the first and second measurement units is approximately the same as the positional relationship between the center of the user's pupil (the intersection of the visual axis and the outer layer of the cornea) and the first and second measurement units (i.e., the photo module) when the user's eyeball is focused on the first camera.

[0011] The objective of this invention can be achieved through the following technical solutions:

[0012] An improved photographing module for a gaze-direction measuring device includes an adjustment component and an image acquisition component mounted on the adjustment component. The image acquisition component includes at least a photographing module. The adjustment component is used to adjust the spatial position of the photographing module. A first camera is disposed at the center of the photographing module, and measuring units for emitting light to the outside and receiving light are respectively disposed on both sides of the first camera.

[0013] As a further aspect of the present invention: the camera module has a nine-square grid structure, and a measurement unit is provided at each inflection point of the nine-square grid of the camera module.

[0014] As a further aspect of the present invention: the measuring unit includes a light-emitting element and a camera unit, and the number of the light-emitting element and the camera unit corresponds one-to-one with the number of measuring units;

[0015] The ratio of camera units to light-emitting elements within the same measurement unit is 1:M, where M≥1.

[0016] As a further aspect of the present invention: the light emitted by the light-emitting element is diffused, so that the light can be emitted at various angles in space.

[0017] As a further aspect of the present invention: the light-emitting element is a circular LED lamp.

[0018] As a further aspect of the present invention: the camera unit is a color camera.

[0019] The beneficial effects of this invention are:

[0020] (1) In this invention, two measurement units are used to replace the first infrared point light source and the second camera in the camera module respectively: regardless of whether the optical axis of the first camera in the camera module intersects with the center of the user's pupil, the two measurement units installed on the camera module determine a point on the surface of the high-gloss object (eyeball) according to the method mentioned above. The positional relationship between the simplified model of the first measurement unit and the simplified model of the second measurement unit is approximately the same as the positional relationship between the center of the user's pupil (the intersection of the visual axis and the outer layer of the cornea) and the first and second measurement units when the user's eyeball is looking at the first camera. This avoids measurement errors or the inability to measure the position of the center of the operator's pupil because the optical axis of the first camera is not completely aligned with the center of the operator's pupil.

[0021] (2) The present invention installs multiple measurement units consisting of circular LED lights and color cameras on the camera module. That is, the measurement unit replaces the color camera on the traditional camera module. The shape of the user's eyeball is measured by the calculation of the measurement unit, so as to improve the detail of the user's face and eyeball model, thereby improving the accuracy of the final eye tracking. Attached Figure Description

[0022] The invention will now be further described with reference to the accompanying drawings.

[0023] Figure 1 This is a schematic diagram of the existing technology structure;

[0024] Figure 2 This is a schematic diagram of the operation of an existing camera module;

[0025] Figure 3 This is a schematic diagram of the structure of the camera module of the present invention;

[0026] Figure 4 This is a schematic diagram of the eye-pointing structure of the camera module of the present invention;

[0027] Figure 5This is a schematic diagram of the structure of the photographing module of the present invention imported into three-dimensional space. Figure 1 ;

[0028] Figure 6 This is a schematic diagram of the structure of the photographing module of the present invention imported into three-dimensional space. Figure 2 ;

[0029] Figure 7 This is a schematic diagram of the structure of the photographing module of the present invention imported into three-dimensional space. Figure 3 ;

[0030] Figure 8 This is a schematic diagram simulating the structure of the camera module of the present invention;

[0031] Figure 9 This is a schematic diagram of the structure of the present invention, which modifies a color camera into a measurement unit. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1

[0034] See Figures 1-3 As shown, the present invention is an improved photographing module of a gaze pointing measurement device, including an adjustment component and an image acquisition component mounted on the adjustment component. The image acquisition component includes at least a photographing module. The adjustment component is used to adjust the spatial position of the photographing module. A first camera is arranged at the center of the photographing module, and measuring units for emitting light to the outside and receiving light are respectively arranged on both sides of the first camera.

[0035] The measurement unit includes a light-emitting element and a camera unit. The number of light-emitting elements and camera units corresponds one-to-one with the number of measurement units. The ratio of the number of camera units to the number of light-emitting elements in the same measurement unit is 1:M, where M≥1. The light emitted by the light-emitting element is diffused, allowing the light to be emitted at various angles in space. The light-emitting element is a ring-shaped LED light, and the camera unit is a color camera.

[0036] like Figure 4As shown, the first measuring unit 131b2 and the second measuring unit 131b3 are disposed on both sides of the optical axis T171 of the first camera 131b1; the line T172 connecting the camera focal point in the first measuring unit 131b2 and the camera focal point in the second measuring unit 131b3 intersects the optical axis T171 of the first camera 131b1, and the line T172 is perpendicular to the optical axis T171 of the first camera 131b1; the distance between the camera focal point in the second measuring unit 131b3 and the optical axis T172 of the first camera 131b1 is equal to the distance between the camera focal point in the first measuring unit 131b2 and the optical axis of the first camera 131b1.

[0037] The distance is 20mm;

[0038] The specific process is as follows:

[0039] like Figure 5 As shown, the three-dimensional model of the shooting module is imported into the simulated three-dimensional space, and the point where the focus of the first camera lens is located is named point A, the focus of the camera in the first measurement unit is named point B, and the focus of the camera in the second measurement unit is named point C.

[0040] like Figure 6 As shown, the eye model is imported into the same simulated three-dimensional space. The eye model is set to consist of a sphere with a diameter of 25mm and a ray named the visual axis. The starting point of the ray where the visual axis is located intersects the center of the sphere. The only intersection point of the ray and the sphere is named the pupil center point.

[0041] Arbitrarily set the spatial position of the camera module and the spatial position of the center of the sphere where the eyeball model is located, and ensure that the visual axis in the eyeball model always intersects the focal point of the first camera in the camera module (and the position of the optical axis of the first camera in the camera module is also arbitrary, and does not need to intersect the center point of the pupil in the eyeball model).

[0042] like Figure 7 As shown, a point on the spherical surface of the eyeball model is taken and designated as point F. Point F is used to simulate the unique point on the cornea of ​​the user's eye that is reflected by the light scattered by the LED light in the first measurement unit and illuminating all points on the user's cornea. According to the law of reflection and the principle of reversibility of light, this point on the cornea is also the point on the cornea of ​​the user's eye that is reflected by the light scattered by the LED light in the second measurement unit and illuminating all points on the user's cornea. In other words, point F is equivalent to the location of the pupil center point calculated based on the two measurement units in reality.

[0043] Finding point F on the sphere containing the simulated eyeball model requires that point F's position on the sphere satisfy the following:

[0044] 1: Point F is in the same plane as the line connecting it to the focal point B of the camera in the first measurement unit, the focal point C of the camera in the second measurement unit, the normal line of the sphere passing through the point (the line passing through the point and perpendicular to the tangent on the sphere), and these two lines.

[0045] 2: The lines connecting point F to the focal point B of the camera in the first measurement unit and the focal point C of the camera in the second measurement unit are respectively on both sides of the spherical normal passing through that point;

[0046] 3: The angles between the lines connecting point F to the focal point B of the camera in the first measurement unit and the focal point C of the camera in the second measurement unit, and the normal to the sphere passing through that point, are equal; (equivalent to the law of reflection of light)

[0047] The relative positional relationship between six units—point F on the sphere of the simulated eye model, focus A of the first camera on the 3D model of the improved camera module, focus B of the camera in the first measurement unit, focus C of the camera in the second measurement unit, the line connecting point F and focus B of the camera in the first measurement unit, and the line connecting point F and focus C of the camera in the second measurement unit—is equivalent to the relative positional relationship between six substances in reality: a point on the cornea of ​​the eye, focus A of the first camera on the improved camera module, focus B of the camera in the first measurement unit, focus C of the camera in the second measurement unit, a ray of light emitted by the light-emitting element in the second measurement unit that shines on a point on the surface of the cornea and is reflected by the mirror surface of that point and received by the camera in the first measurement unit, and a ray of light emitted by the light-emitting element in the first measurement unit that shines on the same point on the surface of the cornea and is reflected by the mirror surface of that point and received by the camera in the second measurement unit.

[0048] The location of the pupil center point is calculated according to the method described in patent application number 202111123670.9 (A device and method for measuring the shape of a smooth surface), specifically as follows:

[0049] Step 1: The first light-emitting element in the first measurement unit emits light. One of the emitted rays illuminates a point on the smooth surface of the object and is reflected by the mirror at that point. The reflected ray is received by the second camera in the second measurement unit, and the ray forms a light path in space.

[0050] Step 2: At the same time, the second light-emitting element in the second measuring unit emits light. One of the emitted rays illuminates the same point on the smooth surface of the object along the light path in Step 1, is reflected by the mirror at this point, and travels along the light path to the first camera in the first measuring unit, where it is received by the first camera.

[0051] Step 3: Based on the data obtained from the two light rays in the optical path in Step 1 and Step 2 that are incident on the first camera and the second camera and received by the first camera and the second camera, calculate the positional relationship of the point on the smooth object surface relative to the second camera and the first camera, and the direction of the tangent of the point on the smooth object surface.

[0052] That is, the first measurement unit, the second measurement unit, and so on up to the Nth measurement unit can measure the surface of a smooth object. The direction of the normal to the point and the tangent of the smooth surface of the object containing the point.

[0053] The coordinates of point F on the sphere of the eyeball model are obtained in the simulation space. Let L be the distance between the pupil center point E of the eyeball model and the measured point F on the eyeball model. This represents the distance between the correct pupil center point E of the sphere model in the simulation space and the pupil center point F calculated by the two measurement units mentioned in this paper. The error value between the calculated pupil center point and the correct pupil center point is also calculated.

[0054] Figure 8 The results of the three simulations A, B, and C are shown in Table (1) below:

[0055]

[0056] Table (1)

[0057] In the data from the above simulation test, the optical axis T171 of the first camera 131b1 never intersected with the center point of the operator's pupil.

[0058] The above data illustrates that even without adjusting the position of the camera module so that the optical axis of the first camera intersects with the center of the pupil of the eye model, the points obtained by the two measurement units installed on the camera module when measuring the eye model are approximately the center of the pupil when the eye model is looking directly at the first camera in the camera module.

[0059] Example 2

[0060] This embodiment is basically the same as the implementation of embodiment 1, except that a measurement unit is set at each inflection point of the nine-grid of the camera module.

[0061] like Figure 9As shown, the measuring unit is the measuring unit described in patent application number 202111123670.9 (a device and method for measuring the shape of a smooth surface), which realizes the measurement of the shape of a smooth surface, thereby realizing the measurement of the shape of the user's cornea;

[0062] While measuring the position of the center point of the user's pupil, multiple measuring units work together according to the method described in patent application number 202111123670.9 (a device and method for measuring the shape of a smooth surface) to add the information of the user's eyeball shape to the eye pointing model described in patent application number 202110609915.2 (eye pointing measurement device and eye pointing model establishment method), which can further improve the information of the eye pointing model. Using the above information as known conditions helps to measure the user's eye pointing more accurately.

[0063] The light-emitting element in the measurement unit is a ring-shaped LED light, and the ring-shaped LED light can be limited to emit infrared light of a specific wavelength. Since the infrared light cannot be seen by the user's eyes, it will not distract the user's attention, allowing the user to concentrate, control their eyeballs, and look at the center of the lens of the first camera.

[0064] The camera unit is a color camera. The wavelength range of light that the photosensitive element of the color camera can receive and image is set and adjusted. It can not only receive the wavelength of visible light without losing its original function, but also serve as a measurement unit to receive the wavelength of specific infrared light emitted by the ring LED light and reflected by the cornea of ​​the eye. The shape of the user's cornea is measured in accordance with the measurement method of patent application number 202111123670.9 (a device and method for measuring the shape of a smooth surface).

[0065] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. An improved photographing module of a gaze direction measurement device, comprising an adjusting assembly and an image capturing assembly mounted on the adjusting assembly, the image capturing assembly at least comprising a photographing module, wherein the adjusting assembly is used to adjust the spatial position of the photographing module, characterized in that, The camera module has a first camera at its center, and measurement units for emitting light to the outside and receiving light are respectively located on both sides of the first camera. A first measuring unit and a second measuring unit are provided on both sides of the optical axis of the first camera, and the center of the first camera is located on the perpendicular bisector of the line connecting the camera focus in the first measuring unit and the camera focus in the second measuring unit. The measurement unit includes a light-emitting element and a camera unit. The number of light-emitting elements and camera units corresponds one-to-one with the number of measurement units, or the ratio of the number of camera units to the number of light-emitting elements in the same measurement unit is 1:M, where M≥1. The light emitted by the light-emitting element is diffused, allowing the light to be emitted at various angles in space; The two measurement units installed on the camera module can determine a point on the surface of a glossy object. The positional relationship between this point and the first and second measurement unit models is equivalent to the positional relationship between the center of the user's pupil and the first and second measurement units when the user's eye is focused on the first camera. The location of the pupil's center point is calculated using the following method: Step 1: The first light-emitting element in the first measurement unit emits light. One of the emitted rays illuminates a point on the smooth surface of the object and is reflected by the mirror at that point. The reflected ray is received by the second camera in the second measurement unit, and the ray forms a light path in space. Step 2: At the same time, the second light-emitting element in the second measuring unit emits light. One of the emitted rays illuminates the same point on the smooth surface of the object along the light path in Step 1, is reflected by the mirror at this point, and travels along the light path to the first camera in the first measuring unit, where it is received by the first camera. Step 3: Based on the data obtained from the two light rays in the optical path in Step 1 and Step 2 that are incident on the first and second cameras and received by the first and second cameras, calculate the positional relationship of the point on the smooth object surface relative to the second and first cameras, and the direction of the tangent plane on the smooth object surface where the point is located.

2. The improved photographing module of a visual direction measurement device according to claim 1, wherein, The camera module has a nine-square grid structure, and a measurement unit is set at each inflection point of the nine-square grid.

3. The improved photographing module of a visual direction measurement device according to claim 1, wherein, The light-emitting element is a ring-shaped LED.

4. The improved photographing module of a visual direction measurement device according to claim 1, wherein, The camera unit is a camera.

Images