Transparent HUD interaction system based on event-frame-electro-oculogram three-source fusion

Abstract

The invention discloses a transparent HUD interaction system based on event-frame-electro-oculogram three-source fusion. The transparent HUD interaction system comprises an event camera, an electro-oculogram sensor and a frame type camera. The voice analysis module is used for collecting a voice instruction, extracting a specified interface element in the voice instruction as a voice interaction target and generating a voice confidence coefficient; the three-source fusion module is used for fusing the eye movement event flow, the eye electric signal and the eye image frame to obtain a fixation vector; the prediction module is used for predicting a gaze vector and gaze confidence at the next moment according to the gaze vector sequence in the recent preset time period; the interaction target confirmation module is used for acquiring a sight line interaction target, and confirming the interaction target when the sight line interaction target is consistent with the voice interaction target and the product of the voice confidence coefficient and the gaze confidence coefficient is greater than or equal to a preset gating threshold value; and the transparent display module is used for projecting the interactive interface on a vehicle-mounted transparent interface and displaying the real-time interactive information. The method is higher in recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to a HUD interaction system, in particular to a transparent HUD interaction system based on event-frame-EOG three-source fusion. BACKGROUND

[0002] With the development of vehicle intelligent systems and virtual reality technology, users have higher requirements for intuitive and efficient information acquisition and interaction when driving and touring. However, the current HUD (Head-Up Display) system in the intelligent cockpit environment still relies on traditional touch or voice methods in the interaction level, and lacks natural and intuitive eye movement control support. Existing eye movement tracking technology is mostly based on infrared binocular camera scheme, which has the phenomenon of inaccurate recognition accuracy. At the same time, pure voice interaction also has the phenomenon of limited recognition accuracy, unclear reference, and frequent false triggering in the driving scene, which affects user experience and driving safety. SUMMARY

[0003] In view of the problems existing in the prior art, the purpose of the present application is to provide a transparent HUD interaction system based on event-frame-EOG three-source fusion with higher recognition accuracy.

[0004] In order to achieve the above-mentioned purpose of the application, the present application provides the following technical scheme:

[0005] A transparent HUD interaction system based on event-frame-EOG three-source fusion, comprising:

[0006] An event camera for capturing edge brightness change events caused by user gaze changes and generating an eye movement event stream;

[0007] An EOG sensor arranged in the user contact area for collecting EOG signals generated during user eye rotation;

[0008] A frame camera for shooting eye image frames of the user;

[0009] A voice analysis module for collecting voice instructions of the user, extracting specified interface elements in the voice instructions as voice interaction targets through voice recognition and natural language processing, and generating a voice confidence level;

[0010] A three-source fusion module for fusing the eye movement event stream, the EOG signal and the eye image frame to obtain a gaze vector and its covariance;

[0011] A prediction module for predicting the gaze vector and its covariance matrix at the next moment according to the gaze vector sequence in the recent preset time period, and obtaining a gaze confidence level according to the inverse mapping of the covariance matrix;

[0012] An interaction target confirmation module is used to obtain the interface element on the interactive interface where the predicted gaze vector falls as the gaze interaction target. When the gaze interaction target is consistent with the voice interaction target and the product of the voice confidence and the gaze confidence is greater than or equal to a preset gating threshold, the gaze interaction target or the voice interaction target is confirmed as the interaction target;

[0013] The transparent display module is used to project the interactive interface onto the vehicle-mounted transparent interface and display real-time interactive information.

[0014] Furthermore, the system further comprises:

[0015] The interaction module is used to trigger a highlight prompt of the interactive target when the user's gaze on the interactive target exceeds a first gaze retention threshold T1, and to trigger automatic confirmation when the user's gaze on the interactive target exceeds a second gaze retention threshold T2; wherein T2 = T1 + f(v), f(v) is a monotonically decreasing function of the vehicle speed v.

[0016] Furthermore, the system further comprises:

[0017] The edge inference module is used to perform differential privacy perturbation based on Laplace noise injection on the predicted gaze vector and its corresponding gaze confidence, as well as the voice interaction target and its voice confidence, and send the perturbed data to the interaction target confirmation module.

[0018] Furthermore, the system further comprises:

[0019] The dot matrix lighting module includes multiple infrared laser point source units, which adopt an event density drive control strategy to turn on short-term pulse lighting when the eye movement event flow is lower than a set threshold or the ambient light is insufficient.

[0020] Furthermore, in the fusion judgment module, when the sight interaction target and the voice interaction target are inconsistent, a selection is made or a secondary confirmation by the user is initiated according to the corresponding confidence level.

[0021] Furthermore, the transparent display module specifically includes a HUD projection unit and a brightness adaptive adjuster, wherein the HUD projection unit is used to project the interactive interface and real-time interactive information onto the vehicle-mounted transparent interface, and the brightness adjuster is used to automatically adjust the projection intensity according to the ambient light L so that the contrast under strong light is ≥5:1.

[0022] Furthermore, the system further comprises:

[0023] The self-calibration module is used to automatically update the frame-based camera extrinsic parameters in a period of no more than 10 seconds based on the stable image features of lane lines or reference objects.

[0024] Furthermore, the system further comprises:

[0025] The model update module is used to synchronize system parameters using federated learning.

[0026] Furthermore, the prediction module is specifically Tiny-Transformer.

[0027] Furthermore, when the eye movement event stream is interrupted or the user wears an occluder, resulting in loss of eye image frames, the three-source fusion module only uses the electrooculogram signal to obtain the gaze vector and its covariance.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. By introducing a three-source fusion perception structure consisting of an event camera, a frame-based camera, and an electrooculogram sensor, the present invention can stably obtain user gaze information under different lighting conditions, line of sight obstruction states, or dynamic driving scenarios. This provides greater robustness and fault tolerance, and improves recognition accuracy.

[0030] 2. The timing prediction algorithm can perform forward compensation prediction of the gaze vector, effectively reducing the problem of false touch caused by dynamic head movement or system delay, improving the overall interactive stability and predictive ability of the system, and further improving recognition accuracy;

[0031] 3. Through a gating mechanism based on the product of voice and gaze confidence, the system integrates the user's voice commands and gaze position to accurately identify the interaction target, avoiding execution errors caused by semantic ambiguity or gaze misjudgment in traditional interactions.

[0032] 4. The high-frequency event stream driven by the event camera, combined with frame images to compensate for low-activity scenes, improves system response speed while significantly reducing energy consumption. The event gating mechanism of the dot matrix lighting enables infrared light to be activated only when necessary, reducing average power consumption by approximately 40% compared to traditional continuous lighting solutions.

[0033] 5. By combining electrooculogram (EOG) signals as an auxiliary path, the system can maintain continuous perception of the user's gaze behavior even when eye image acquisition fails or is blocked, greatly improving interaction continuity in complex scenarios.

[0034] 6. A dual-threshold gaze determination strategy is implemented to ensure a safe and efficient intention confirmation process under various driving conditions.

[0035] 7. The system adopts an edge inference architecture, with all computing tasks completed on the device side. A differential privacy mechanism is used to protect the user's sight and voice data from being uploaded in its original form. This complies with GDPR and related automotive privacy regulations and is suitable for large-scale deployment.

[0036] 8. The introduction of a self-calibration module enables dynamic calibration of multi-camera systems without relying on dedicated targets, improving the long-term stability and maintenance convenience of the system;

[0037] 9. The system supports model updates under the federated learning architecture, synchronizing only parameter increments without involving the transmission of original user data, effectively ensuring the sustainable evolution of the model and privacy security.

[0038] In summary, the present invention significantly improves the overall performance of the HUD system from multiple dimensions, including recognition accuracy, perception accuracy, interaction accuracy, power consumption control, response efficiency, and privacy compliance, and has extremely high practical application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a structural diagram of the first embodiment of the transparent HUD interactive system based on event-frame-electrooculography triple-source fusion provided by the present invention;

[0040] Figure 2 This is a structural diagram of a second embodiment of a transparent HUD interactive system based on event-frame-electrooculography triple-source fusion provided by the present invention;

[0041] Figure 3 This is a structural diagram of the third embodiment of the transparent HUD interactive system based on event-frame-electrooculography triple-source fusion provided by the present invention;

[0042] Figure 4 This is a structural diagram of a fourth embodiment of a transparent HUD interactive system based on event-frame-electrooculography triple-source fusion provided by the present invention;

[0043] Figure 5 This is a structural diagram of the fifth embodiment of the transparent HUD interactive system based on event-frame-electrooculography triple-source fusion provided by the present invention;

[0044] Figure 6 This is a structural diagram of a sixth embodiment of a transparent HUD interactive system based on event-frame-electrooculography triple-source fusion provided by the present invention;

[0045] Figure 7 This is a structural diagram of the seventh embodiment of the transparent HUD interactive system based on event-frame-electrooculography triple-source fusion provided by the present invention. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0047] The embodiment of the present invention provides a transparent HUD interactive system based on event-frame-electrooculography three-source fusion, such as Figure 1 Shown, including:

[0048] An event camera is used to capture edge brightness change events caused by changes in the user's gaze and generate an eye movement event stream. The event camera outputs the eye movement event stream at a frequency of no less than 1kHz, with ultra-low latency time resolution.

[0049] Electrooculogram (EOG) sensors are placed in user contact areas (such as steering wheels and eyeglass nose pads) to collect EOG signals generated by eye rotation, improving adaptability to occlusion or image distortion.

[0050] A frame camera, specifically a near-infrared frame camera, is used to capture eye image frames at a low frame rate (≤30 Hz) and automatically enable frame image compensation when the event density is continuously lower than a preset threshold or the EOG signal variance exceeds a set threshold to improve perception stability in low-frequency gaze states;

[0051] The speech analysis module, which includes a directional microphone and a semantic parser, is used to collect user voice commands, extract the specified interface elements in the voice commands as voice interaction targets through speech recognition and natural language processing, and generate the voice confidence C_voice, which is obtained by ASR confidence mapping;

[0052] The three-source fusion module is used to fuse the eye movement event stream, the electrooculogram signal, and the eye image frame using a Kalman filter to obtain the gaze vector g(t) and its covariance Σ(t). Furthermore, when the eye movement event stream is interrupted or the eye image frame is lost due to an occlusion, only the electrooculogram signal is used to obtain the gaze vector g(t) and its covariance Σ(t).

[0053] The prediction module is used to predict the gaze vector and its covariance matrix at the next moment based on the gaze vector sequence within the most recent preset time period (for example, the next 150ms), and obtain the gaze confidence C_gaze based on the inverse mapping of the covariance matrix. The inverse mapping specifically means that the gaze confidence C_gaze is obtained by mapping the trace value trace(Σ) of the covariance matrix Σ in an exponential decay form. The specific calculation formula is: C_gaze = exp(-α*trace(Σ)), where α is an empirical calibration coefficient used to adjust the sensitivity of the confidence to the size of the covariance. This mapping rule ensures that the smaller Σ is (indicating a more reliable estimate), the closer C_gaze is to 1; the larger Σ is, the closer C_gaze is to 0. The prediction module is specifically Tiny-Transformer, which is a lightweight neural network consisting of no more than 6 layers and a total parameter amount of no more than 0.6 million. It is used to complete gaze vector prediction within 1 millisecond;

[0054] The interaction target confirmation module is used to obtain the interface element on the interactive interface where the predicted gaze vector falls as the gaze interaction target. When the gaze interaction target is consistent with the voice interaction target and the product of the voice confidence C_voice and the gaze confidence C_gaze is greater than or equal to the preset gating threshold β, the gaze interaction target or the voice interaction target is confirmed as the interaction target. When the gaze interaction target is inconsistent with the voice interaction target, a selection is made or a secondary confirmation by the user is initiated based on the corresponding confidence.

[0055] The transparent display module is used to project the interactive interface onto the vehicle-mounted transparent interface (windshield or other transparent display interface) and display real-time interactive information; it specifically includes a HUD projection unit and a brightness adaptive adjuster, wherein the HUD projection unit is used to project the interactive interface and real-time interactive information onto the vehicle-mounted transparent interface, and the brightness adjuster is used to automatically adjust the projection intensity according to the ambient light L so that the contrast ratio under strong light is ≥5:1, thereby ensuring that the user can clearly identify the virtual and real superimposed information under different lighting conditions.

[0056] like Figure 2 As shown, in other embodiments, the transparent HUD interaction system may further include:

[0057] The edge inference module, specifically the PEIL (Privacy-Enhanced Edge Inference Layer) edge inference module, is used to perform differential privacy perturbations on the predicted gaze vector t) and its corresponding gaze confidence C_gaze, the voice interaction target and its voice confidence C_voice, and send the perturbed data to the interaction target confirmation module; the differential privacy perturbation adopts the differential privacy protection mechanism of the Laplace distribution, and the privacy budget ε is not higher than 1.0.

[0058] like Figure 3 As shown, in other embodiments, the transparent HUD interaction system may further include:

[0059] The dot matrix lighting module includes multiple infrared laser point source units, which adopt an event flow density drive control strategy to turn on short-term pulse lighting when the eye movement event flow is lower than the set threshold or the ambient light is insufficient. The average duty cycle is ≤10%, and the single pulse width is no more than 10 microseconds, effectively reducing energy consumption while enhancing pupil feature contrast.

[0060] like Figure 4 As shown, in other embodiments, the transparent HUD interaction system may further include:

[0061] The interaction module is used to trigger a highlight prompt of the interactive target when the user's gaze on the interactive target exceeds a first gaze retention threshold T1, and to trigger automatic confirmation when the user's gaze on the interactive target exceeds a second gaze retention threshold T2; wherein T2 = T1 + f(v), f(v) is a monotonically decreasing function of the vehicle speed v, so as to reduce operation waiting time and improve response efficiency in high-speed driving conditions.

[0062] like Figure 5 As shown, in other embodiments, the transparent HUD interaction system may further include:

[0063] The self-calibration module is used to automatically update the frame camera's extrinsic parameters in a period of no more than 10 seconds based on the stable image features of lane markings or reference objects. Self-calibration uses Auto-BEV to perform reverse bird's-eye view feature extraction based on image elements such as lane markings and reference objects, achieving dynamic calibration of extrinsic parameters between on-board cameras. The update period is no more than 10 seconds and the accuracy is better than 0.5°.

[0064] like Figure 6 As shown, in other embodiments, the transparent HUD interaction system may further include:

[0065] The model update module is used to synchronize system parameters using federated learning. Specifically, it uses an OTA model update mechanism to remotely update the Tiny-Transformer model in the system, transmitting only encrypted increments without uploading any image, voice, or raw sensor data. This enhances algorithm adaptability while protecting user privacy.

[0066] like Figure 7 As shown, in other embodiments, the transparent HUD interaction system may further include:

[0067] The mode conversion module is used to cause the event camera, frame camera, electrooculogram sensor, dot matrix lighting module and speech analysis module to enter 5Hz low-frequency sniffing mode to reduce power consumption when the vehicle speed v is lower than 5km / h and the system determines that the covariance matrix Σ of the gaze vector exceeds the set upper limit.

[0068] It can be understood that all of the above modules can be combined to exist in one embodiment, or one or more modules can be freely combined to exist in one embodiment.

[0069] It is worth noting that in the above embodiments, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0070] The above-described embodiments are merely illustrative for the present application, wherein the modules illustrated as separate components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, i.e., can be located in one place or can be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purposes of the present embodiments. It is clear to those skilled in the art that the various embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized only by hardware, as long as the functions or effects can be realized.

[0071] It should be understood that the above embodiments and descriptions in the specification are only the principles, main features and advantages of the present application, and various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the protection scope of the present application.

Claims

1. A transparent HUD interactive system based on event-frame-electrooculography three-source fusion, characterized by: include: An event camera is used to capture edge brightness change events caused by changes in user gaze and generate an eye movement event stream; Electrooculogram sensors are placed in the user contact area to collect electrooculogram signals generated during the user's eye rotation; A frame camera, used to capture eye image frames of the user; The speech analysis module is used to collect user voice commands, and through speech recognition and natural language processing, extract the specified interface elements in the voice commands as voice interaction targets and generate speech confidence; The three-source fusion module is used to fuse the eye movement event stream, electrooculogram signal and eye image frame to obtain the gaze vector and its covariance; A prediction module is used to predict the gaze vector and its covariance matrix at the next moment based on the gaze vector sequence in the most recent preset time period, and obtain the gaze confidence according to the inverse mapping of the covariance matrix; An interaction target confirmation module is used to obtain the interface element on the interactive interface where the predicted gaze vector falls as the gaze interaction target. When the gaze interaction target is consistent with the voice interaction target and the product of the voice confidence and the gaze confidence is greater than or equal to a preset gating threshold, the gaze interaction target or the voice interaction target is confirmed as the interaction target; The transparent display module is used to project the interactive interface onto the vehicle-mounted transparent interface and display real-time interactive information.

2. The transparent HUD interactive system based on event-frame-electrooculography three-source fusion according to claim 1 is characterized in that: The system further comprises: The interaction module is used to trigger a highlight prompt of the interactive target when the user's gaze on the interactive target exceeds a first gaze retention threshold T1, and to trigger automatic confirmation when the user's gaze on the interactive target exceeds a second gaze retention threshold T2; wherein T2 = T1 + f(v), f(v) is a monotonically decreasing function of the vehicle speed v.

3. The transparent HUD interactive system based on event-frame-electrooculography three-source fusion according to claim 1 is characterized in that: The system further comprises: The edge inference module is used to perform differential privacy perturbation based on Laplace noise injection on the predicted gaze vector and its corresponding gaze confidence, as well as the voice interaction target and its voice confidence, and send the perturbed data to the interaction target confirmation module.

4. The transparent HUD interactive system based on event-frame-electrooculography triple-source fusion according to claim 1 is characterized in that: The system further comprises: The dot matrix lighting module includes multiple infrared laser point source units, which adopt an event density drive control strategy to turn on short-term pulse lighting when the eye movement event flow is lower than a set threshold or the ambient light is insufficient.

5. The transparent HUD interactive system based on event-frame-electrooculography three-source fusion according to claim 1 is characterized in that: In the fusion judgment module, when the sight interaction target and the voice interaction target are inconsistent, a selection is made according to the corresponding confidence level or a secondary confirmation by the user is initiated.

6. The transparent HUD interactive system based on event-frame-electrooculography three-source fusion according to claim 1 is characterized in that: The transparent display module specifically includes a HUD projection unit and a brightness adaptive adjuster, wherein the HUD projection unit is used to project the interactive interface and real-time interactive information onto the vehicle-mounted transparent interface, and the brightness adjuster is used to automatically adjust the projection intensity according to the ambient light L so that the contrast ratio under strong light is ≥5:

1.

7. The transparent HUD interactive system based on event-frame-electrooculography three-source fusion according to claim 1 is characterized in that: The system further comprises: The self-calibration module is used to automatically update the frame-based camera extrinsic parameters in a period of no more than 10 seconds based on the stable image features of lane lines or reference objects.

8. The transparent HUD interactive system based on event-frame-electrooculography three-source fusion according to claim 1 is characterized in that: The system further comprises: The model update module is used to synchronize system parameters using federated learning.

9. The transparent HUD interactive system based on event-frame-electrooculography three-source fusion according to claim 1 is characterized in that: The prediction module is specifically Tiny-Transformer.

10. The transparent HUD interactive system based on event-frame-electrooculography three-source fusion according to claim 1 is characterized in that: When the eye movement event stream is interrupted or the user wears an occluder, resulting in the loss of eye image frames, the three-source fusion module only uses the electrooculogram signal to obtain the gaze vector and its covariance.

Images