Vehicle-mounted optical path hardware system based on eye pupil tracking

Abstract

The invention provides a vehicle-mounted light path hardware system based on eye pupil tracking, and relates to the technical field of optical design, the system comprises a reflection light array which comprises a windshield glass, a first free-form surface mirror, a second free-form surface mirror, a beam combiner and a diffuser screen and is used for receiving an infrared receiving light path and an imaging display light path and displaying the light path on the windshield glass, an infrared receiving and transmitting light path reflected by the human face is transmitted to the infrared receiving and transmitting module; the projection module comprises a projection lens and a display chip and is used for generating an imaging display light path by utilizing the display chip and projecting the imaging display light path by utilizing the projection lens; the infrared receiving and transmitting module comprises an infrared lighting source LED and an infrared lens and is used for generating an infrared receiving and transmitting light path and receiving an infrared receiving and transmitting light path of the reflected light array; and adjusting the position and the angle of the reflected light array based on the infrared receiving and transmitting light path to finish eye pupil tracking. The problem that it is difficult for a driver to accurately interact with a vehicle-mounted light path system is solved.

Description

Technical Field

[0001] The present invention relates to the field of optical design technology, and in particular to a vehicle-mounted optical path hardware system based on pupil tracking. Background Art

[0002] Pupil tracking is widely used in the study of human cognition. It was first used in the early 20th century using a contact lens with a pointer. Thirty years later, the technology was optimized by using light beams and recording their reflections on film. Modern methods for studying eye movements were developed in the 1960s and have been further refined since then. The exact methods for monitoring eye movements have changed a lot in the past few years. Today, video-based systems using computer vision techniques dominate. Due to the development of small, high-quality cameras in devices such as smartphones, it is now possible to have pupil tracking systems that are lightweight and convenient, and can even be installed in VR headsets or portable glasses. These technologies can monitor eye movements quickly and accurately, providing a large amount of eye movement data. Due to the close relationship between eye movements and cognition, eye tracking has received increasing attention in a variety of experimental settings. With the advancement of technology and the increase in the number of studies in the field of eye tracking, it has now developed to a point where it can work well in a wide variety of settings to study human cognitive processes.

[0003] In the context of the growing development of modern science and technology, the demand for using pupil tracking technology to dynamically sense the user's gaze intensity, thereby intelligently adjusting the HUD brightness and ensuring visual comfort is becoming more and more prominent. With people's continuous pursuit of the user experience of technological products, the application of pupil tracking technology will bring users a more intelligent and comfortable visual experience. As an emerging technical means, pupil tracking technology can accurately perceive the user's gaze focus and gaze intensity through real-time monitoring and analysis of the user's eye movement behavior. When applied to HUD (head-up display) devices, pupil tracking technology can monitor the user's gaze on the HUD screen in real time, and then adjust the screen brightness and contrast, so that the user can still maintain a comfortable visual experience during long-term use. Eye movement interaction joins the automotive field in a new way of human-computer interaction, allowing the driver to control the screen without leaving the steering wheel, and actively avoid the screen when the HUD screen blocks the field of vision; attention monitoring analyzes the trajectory and stay of the gaze point to accurately judge the driver's fatigue level, rather than judging fatigue by the length of time the eyes are closed. Because when the eyes are closed, the driver is already fatigued and should not drive; in addition, by utilizing the physiological characteristics of the human eye, the user does not need to "open his mouth" or "blink" to achieve liveness detection of face recognition and iris recognition, thereby improving the anti-counterfeiting and ease of use of identity authentication. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a vehicle-mounted optical path hardware system based on pupil tracking to solve the problem that it is difficult for the driver to accurately interact with the vehicle-mounted optical path system.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a vehicle-mounted optical path hardware system based on pupil tracking, including a reflected light array, an infrared transceiver module and a projection module:

[0006] The reflected light array includes a windshield, a first free-form surface mirror, a second free-form surface mirror, a beam combining mirror and a scattering screen, and is used to receive the infrared light receiving and transmitting path and the imaging display light path and display them on the windshield, and transmit the infrared light receiving and transmitting path reflected by the face to the infrared transceiver module;

[0007] A projection module, comprising a projection lens and a display chip, for generating the imaging display light path using the display chip, and projecting the imaging display light path using the projection lens;

[0008] The infrared transceiver module includes an infrared illumination light source LED and an infrared lens, which is used to generate an infrared light-receiving and light-emitting circuit using the infrared illumination light source LED, and to receive the infrared light-receiving and light-emitting circuit of the reflected light array using the infrared lens; based on the infrared light-receiving and light-emitting circuit, the reflection angle of the beam combiner is adjusted to complete pupil tracking.

[0009] Furthermore, the reflected light array comprises:

[0010] A scattering screen, used for scattering the imaging display light path to obtain a scattered imaging display light path;

[0011] A beam combiner is used to merge the infrared light receiving and emitting path and the imaging display light path to obtain a merged transmission light path; and transmit the infrared light receiving and emitting path reflected by the face to the infrared transceiver module;

[0012] The first free-form surface mirror and the second free-form surface mirror are used to collimate the fused transmission light path and project it onto the windshield; and reflect the infrared light receiving and transmitting path reflected by the face to the beam combining mirror;

[0013] The windshield is used to horizontally project the fused transmission light path to obtain a display result; and receive the infrared light-receiving and light-receiving path reflected by the face.

[0014] Furthermore, the beam combining mirror comprises a beam splitting film, which is located on the surface of the beam combining mirror and is used to separate the infrared light receiving and emitting path and the imaging display light path, and transmit the separated infrared light receiving and emitting path to the infrared transceiver module.

[0015] Furthermore, the infrared transceiver module includes:

[0016] Infrared lighting source LED, used to generate infrared light-receiving and light-emitting circuit;

[0017] The infrared lens is used to receive the infrared light-receiving and light-emitting path reflected by the human face; and based on the infrared light-receiving and light-emitting path reflected by the human face, detect the position of the illumination light source transmitted on the human face, adjust the position and angle of the reflected light array, and complete the pupil tracking of the vehicle-mounted optical path system.

[0018] The beneficial effects of the present invention are as follows: a vehicle-mounted optical path hardware system based on pupil tracking integrates the optical hardware of pupil tracking with the optical path system of the vehicle-mounted HUD. When the eye rotates, the position of the pupil center on the camera sensor will change, but the position of the corneal reflection is relatively fixed (when the head is stable), so that the human eye can observe the HUD display image while also tracking the pupil, thereby achieving accurate interaction between the driver and the vehicle-mounted optical path system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0020] Figure 1 is a module schematic diagram of a vehicle-mounted optical path hardware system based on pupil tracking according to some embodiments of this specification;

[0021] Figure 2 is a structural schematic diagram of a vehicle-mounted optical path hardware system based on pupil tracking according to some embodiments of this specification;

[0022] Figure 3 is an exemplary schematic diagram of a collimated and fused transmission optical path according to some embodiments of this specification;

[0023] Figure 4 is an exemplary schematic diagram of pupil tracking according to some embodiments of this specification;

[0024] Figure 5 This is an exemplary flow chart of a vehicle-mounted optical path hardware control method based on pupil tracking according to some embodiments of this specification. DETAILED DESCRIPTION

[0025] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0026] Example

[0027] Figure 1 It is a module schematic diagram of a vehicle-mounted optical path hardware system based on pupil tracking according to some embodiments of this specification.

[0028] In some embodiments, the vehicle-mounted optical path hardware system based on pupil tracking may include a reflected light array, an infrared transceiver module and a projection module.

[0029] The reflected light array is used to receive the infrared light-receiving and light-emitting path and the imaging display light path and display them on the windshield, and transmit the infrared light-receiving and light-emitting path reflected by the human face to the infrared transceiver module.

[0030] The infrared light-transmitting and -receiving path is the light path used to locate the pupil.

[0031] The imaging display optical path is the optical path of the vehicle condition information in the vehicle optical path system (AR-HUD). For example, the imaging display optical path may include vehicle information, navigation information, etc.

[0032] In some embodiments, Figure 2 As shown, the reflected light array may include a windshield, a first free-form surface mirror, a second free-form surface mirror, a beam combining mirror and a scattering screen.

[0033] The scattering screen is used to scatter the imaging display light path to obtain a scattered imaging display light path.

[0034] The diffuser is a functional film with a surface micro-nano structure. For example, the diffuser may include a holographic diffuser.

[0035] In some embodiments, the embossed structure on the surface of the scattering screen is generated by utilizing the speckle principle. When coherent parallel light is incident on the random phase plane, a speckle distribution will be generated on the observation plane. The average size of the speckle depends on the random phase plane diameter and the diffraction distance. For example, the scattering screen can diverge the beam of the projected image incident on the scattering screen to achieve the required display viewing angle and ensure uniform brightness distribution within the viewing angle range.

[0036] In some embodiments, the parameters of the scattering screen such as the scattering angle, spatial brightness non-uniformity, angular brightness attenuation, illumination aperture, diffraction distance, exposure and development time, etching depth, exposure spot interval, etc. can be adjusted to obtain the optimal scattered light distribution. For example, the scattering angle can be -40° to +40° in the horizontal direction and -15° to +15° in the vertical direction, the spatial brightness non-uniformity is ≤20%, the angular brightness attenuation is within the required display time, and the brightness at any observation angle relative to the maximum brightness attenuation is ≤40%, etc.

[0037] The beam combiner is used to fuse the infrared light receiving and emitting path and the imaging display light path to obtain a fused transmission light path; and transmit the infrared light receiving and emitting path reflected by the face to the infrared transceiver module.

[0038] The fused transmission optical path is a transmission optical path that has the function of positioning and displaying vehicle condition information.

[0039] A beam combiner is an optical element used to combine two or more beams of light with different wavelengths into one beam of light.

[0040] In some embodiments, the beam combiner can merge the infrared light emitted from the infrared LED with the optical path of the projection lens, and then project it through the rear-end reflection optical path to illuminate the pupil, which then returns in reverse to form an image in the infrared lens.

[0041] In some embodiments, the infrared reflectivity of the beam combiner is greater than 50%, the visible light transmittance is greater than 90%, the material is K9 glass, and the outer dimensions are 80×35 mm.

[0042] In some embodiments, the beam combiner may further include a beam splitter film, wherein the beam splitter film is located on the surface of the beam combiner and is used to separate the infrared light receiving and transmitting path and the imaging display light path, and transmit the separated infrared light receiving and transmitting path to the infrared transceiver module.

[0043] A beam splitter is a multilayer film structure used to separate light waves of different refractive indices. For example, a beam splitter may include a combination of materials such as magnesium fluoride and zinc sulfide.

[0044] In some embodiments, the prismatic film can utilize the difference in refractive index between different materials in the infrared and visible light bands to achieve a target prismatic effect.

[0045] The first free-form surface mirror and the second free-form surface mirror are used to collimate the fused transmission light path and project it onto the windshield; and reflect the infrared light receiving and transmitting path reflected by the face to the beam combining mirror.

[0046] In some embodiments, the first free-form surface mirror is a convex mirror with a projection area of ​​131×68 mm, and the free-form surface processing coefficient of its reflective surface is shown in Table 1.

[0047] Table 1

[0048]

[0049] In some embodiments, the second free-form surface mirror is a concave mirror with a projection area of ​​310×152 mm, and the free-form surface processing coefficient of its reflective surface is shown in Table 2.

[0050] Table 2

[0051]

[0052] In some embodiments, the first free-form surface mirror and the second free-form surface mirror form a collimated light path, and collimate each point on the received fusion transmission light path to form parallel light. Figure 3 As shown, different points correspond to parallel light of different angles. The field of view of AR-HUD is 10°×4°. The angle of the light emitted from point A' after collimation is -5°, the angle corresponding to A is +5°, B' corresponds to +2°, and B corresponds to -2°.

[0053] The windshield is used to horizontally project the fused transmission light path to obtain a display result; and receive the infrared light-receiving and light-receiving path reflected by the face.

[0054] The display result includes vehicle condition information and corresponding projection position.

[0055] In some embodiments, the windshield may be a GAC ​​Trumpchi GS5 windshield.

[0056] In some embodiments, the driver can obtain the display results through the windshield and interact with the vehicle-mounted optical path system in real time.

[0057] Projection module, such as Figure 2 As shown, it includes a projection lens and a display chip, which are used to generate the imaging display light path using the display chip and project the imaging display light path using the projection lens.

[0058] In some embodiments, specific parameters of the projection lens may include (1) magnification: 15.7; (2) object-side NA: 0.3; (3) prism thickness: 15.5 mm, material HLAK7A_CDGM; (4) back focal length: ≥200 mm; (5) focusing range: ±5 mm; (6) object-side area: 10.368 mm×2.675 mm; (7) wavelength: 451 nm to 639 nm; (8) cut-off frequency: 6 lp / mm; (9) optical system eccentricity: 3.9 mm.

[0059] The display chip is a mainstream digital display chip for projectors. For example, the display chip may include DMD, LCOS, etc.

[0060] In some embodiments, the display chip can use FPGA as the main chip of the control circuit, take advantage of its flexible peripheral interface design capabilities and combine it with the NiosII embedded microprocessor soft core to complete the system's external communication functions through the asynchronous serial interface UART and the dedicated RS485 communication chip, including the reception and response of host commands; the dedicated DVI receiving and decoding chip and the interface implemented inside the FPGA complete the parsing of the serial DVI video signal into an image signal and processing, such as gamma control, mirror display, image shift, special character display and other functions; the processed image signal is then encoded into a serial video signal by a dedicated DVI encoding chip and output to the LCOS module for display; the LED driver part adjusts the PWM duty cycle according to the brightness level formula of the host command to complete the system brightness control.

[0061] The NiosII embedded microprocessor soft core is a user-configurable general-purpose 32-bit RISC microprocessor based on FPGA architecture. Its specific parameters are shown in Table 3.

[0062] Table 3

[0063] Serial number project parameter 1 Resolution SXGA (1280x1024 pixels) 2 Fill factor >96％ 3 Refresh Rate(Max) 85Hz at SXGA 24bit colour 4 Communications Interface DVI / USB 2.0 / RS-232 5 Operating Temperature Range -10℃ to+65℃ 6 Storage Temperature Range -40℃ to+80℃ 7 Power +12V 8 Weight 69g

[0064] The infrared transceiver module includes an infrared illumination light source LED and an infrared lens, which is used to generate an infrared light-receiving and light-emitting circuit using the infrared illumination light source LED, and to receive the infrared light-receiving and light-emitting circuit of the reflected light array using the infrared lens; based on the infrared light-receiving and light-emitting circuit, the reflection angle of the beam combiner is adjusted to complete pupil tracking.

[0065] In some embodiments, Figure 2 As shown, the infrared transceiver module may include an infrared illumination light source LED and an infrared lens.

[0066] Infrared lighting source LED, used to generate infrared light-receiving circuit.

[0067] In some embodiments, the specific parameters of the infrared lighting light source LED may include: wavelength: 850NM, 940NM, 1000NM, 1350NM and other wavelengths corresponding to the infrared receiving lens; power: 1W; current: 700-1000MA; luminous efficiency: more than 30%; luminous angle: 90-120 degrees; forward voltage: red and yellow 1.9-2.2V, white, blue and purple 3.0-3.2V.

[0068] The infrared lens is used to receive the infrared light-receiving and light-emitting path reflected by the human face; and based on the infrared light-receiving and light-emitting path reflected by the human face, detect the position of the illumination light source transmitted on the human face, adjust the position and angle of the reflected light array, and complete the pupil tracking of the vehicle-mounted optical path system.

[0069] Pupil tracking is a function that adjusts the image position in real time by monitoring the pupil position. For example, pupil tracking can include bright pupil tracking and dark pupil tracking.

[0070] Bright pupil tracking is when the light source is coaxial with the line of sight, the retina will reflect the light, and the pupil will become relatively brighter, thus resulting in a bright pupil effect. The on-board optical path system can perform eye tracking by identifying the white oval area as the pupil.

[0071] In this way, the effect of eyelashes on tracking is reduced. However, pupil size has a great influence on the effect of bright pupil tracking. The pupil is affected by several external factors, such as age, emotional response of the subject, and light conditions of the experimental environment.

[0072] Dark pupil tracking is when the light source is not coaxial with the line of sight, that is, the infrared light source is projected far away from the camera, making the pupil appear dark and the iris brighter, thus forming a dark pupil effect. At this time, the brightness of the pupil is lower than that of the iris. Under this tracking method, the on-board optical system can identify the dark part of the iris as the pupil for pupil tracking.

[0073] In some embodiments, Figure 4 As shown, the infrared illumination light source LED is located at point A. After passing through free-form mirror 1 and free-form mirror 2, the angle between the light projected onto the face and the normal of the human eye is -5°. The light reflected from the face will pass through the first free-form mirror and the second free-form mirror and return to the beam combiner. By adjusting the position and angle of the reflected light array, the angle between the imaging display light path and the infrared light receiving and receiving path is made ≥5°, thereby realizing dark pupil tracking.

[0074] This improves tracking in bright environments and outdoor natural light conditions, and improves tracking of dark pupils.

[0075] In some embodiments, the infrared lens can obtain the infrared light-receiving and light-emitting path generated by the infrared illumination light source LED, and obtain the energy returned to the camera:

[0076]

[0077] Where ΔW2 represents the energy returned to the camera, W represents the infrared LED power, and R windshild Represents the reflectivity of the windshield, S eye represents the equivalent scattering area of ​​the human eye, Ω represents the angle of infrared light scattered by the human eye, L represents the distance from the human eye to the first free-form surface reflector, S freeform represents the diameter of the circular area of ​​the second free-form surface mirror, D lens Indicates the aperture size of the projection lens. Indicates the area of ​​the diffuser screen.

[0078] In some embodiments, an R of a vehicle-mounted optical path hardware system based on pupil tracking windshild =0.10, L=100cm, S freeform =10D lens That is, the reflectivity of the windshield can be 0.1, the distance from the human eye to the first free-form surface reflector can be 100 cm, the product of the aperture size of the projection lens and the area of ​​the scattering screen can be 0.5, and the diameter of the equal circle area of ​​the second free-form surface mirror can be ten times the aperture size of the projection lens. The energy returned to the camera is:

[0079]

[0080] In some embodiments, a vehicle-mounted optical path hardware system based on pupil tracking can improve the energy emitted by the acquired infrared illumination light source LED, thereby improving the energy utilization of the LED and the accuracy of pupil tracking.

[0081] In some embodiments of the present specification, a vehicle-mounted optical path hardware system based on pupil tracking integrates the optical hardware of pupil tracking with the optical path system of the vehicle-mounted HUD. When the eye rotates, the position of the pupil center on the camera sensor will change, but the position of the corneal reflection is relatively fixed (when the head is stable), so that the human eye can observe the HUD display image while also tracking the pupil, thereby achieving accurate interaction between the driver and the vehicle-mounted optical path system.

[0082] Embodiment 2

[0083] Figure 5 is an exemplary flow chart of a vehicle-mounted optical path hardware control method based on pupil tracking according to some embodiments of this specification. Figure 5 As shown, the process includes the following steps.

[0084] S1: using a display chip to generate the imaging display light path, and using a projection lens to project the imaging display light path.

[0085] S2: Use infrared lighting source LED to generate infrared light transmission and receiving circuit.

[0086] S3: Using a reflected light array, receiving the infrared light-receiving and light-emitting path and the imaging display light path.

[0087] S4: Infrared light transmission and receiving path using an infrared lens to receive the reflected light array.

[0088] S5: Based on the infrared light-receiving and light-emitting circuit, the reflection angle of the beam combiner is adjusted to complete pupil tracking.

Claims

1. A vehicle-mounted optical path hardware system based on pupil tracking, characterized in that: Including reflected light array, infrared transceiver module and projection module: The reflected light array includes a windshield, a first free-form surface mirror, a second free-form surface mirror, a beam combining mirror and a scattering screen, and is used to receive the infrared light receiving and transmitting path and the imaging display light path and display them on the windshield, and transmit the infrared light receiving and transmitting path reflected by the face to the infrared transceiver module; A projection module, comprising a projection lens and a display chip, for generating the imaging display light path using the display chip, and projecting the imaging display light path using the projection lens; The infrared transceiver module includes an infrared illumination light source LED and an infrared lens, and is used to generate an infrared light-receiving and light-emitting path using the infrared illumination light source LED, and to receive the infrared light-receiving and light-emitting path of the reflected light array using the infrared lens; Based on the infrared light-receiving and light-emitting circuit, the reflection angle of the beam combiner is adjusted to complete pupil tracking.

2. The vehicle-mounted optical path hardware system based on pupil tracking according to claim 1, characterized in that: The reflected light array comprises: A scattering screen, used for scattering the imaging display light path to obtain a scattered imaging display light path; A beam combiner is used to merge the infrared light receiving and emitting path and the imaging display light path to obtain a merged transmission light path; and transmit the infrared light receiving and emitting path reflected by the face to the infrared transceiver module; The first free-form surface mirror and the second free-form surface mirror are used to collimate the fused transmission light path and project it onto the windshield; and reflect the infrared light receiving and transmitting path reflected by the face to the beam combining mirror; The windshield is used to horizontally project the fused transmission light path to obtain a display result; and receive the infrared light-receiving and light-receiving path reflected by the face.

3. The vehicle-mounted optical path hardware system based on pupil tracking according to claim 2, characterized in that: The beam combining mirror comprises a beam splitting film, which is located on the surface of the beam combining mirror and is used to separate the infrared light receiving and emitting path and the imaging display light path, and transmit the separated infrared light receiving and emitting path to the infrared transceiver module.

4. The vehicle-mounted optical path hardware system based on pupil tracking according to claim 1, characterized in that: The infrared transceiver module comprises: Infrared lighting source LED, used to generate infrared light-receiving and light-emitting circuit; The infrared lens is used to receive the infrared light-receiving and light-emitting path reflected by the human face; and based on the infrared light-receiving and light-emitting path reflected by the human face, detect the position of the illumination light source transmitted on the human face, adjust the position and angle of the reflected light array, and complete the pupil tracking of the vehicle-mounted optical path system.

Images