Prejudgment type interaction system and method based on multimode perception

Abstract

The invention discloses a pre-judgment type interaction system and method based on multi-mode perception, and relates to the technical field of intelligent cabins and man-machine interaction, and the system comprises a multi-source data collection module, an intention pre-judgment module, an interaction area dynamic adjustment module, a multi-mode feedback decision module and a self-adaptive strategy module. The multi-source data acquisition module acquires sight lines, biological characteristics and environmental parameters through a hardware synchronization mechanism; the intention pre-judgment module generates an intention probability vector by using a space-time alignment algorithm and a pre-judgment model; the interaction area dynamic adjustment module adjusts the high-probability target control to a sight focus area and optimizes the display attribute; the multi-mode feedback decision module selects an optimal feedback channel according to the environment and the user state; the adaptive strategy module intervenes in the system based on fatigue or emergency conditions. According to the invention, through active pre-judgment and interface reconstruction, the interaction load is reduced, and the interaction efficiency and safety in a complex driving environment are improved in combination with multi-modal feedback.

Description

Technical Field

[0001] This application relates to the field of intelligent cockpit and human-computer interaction technology, and in particular to a predictive interaction system and method based on multimodal perception. Background Technology

[0002] As vehicles become increasingly intelligent, the functions of in-vehicle human-machine interaction systems are becoming more complex, and the amount of information drivers need to process while driving is increasing. To improve the interactive experience, modern smart cockpits are beginning to incorporate various sensor technologies such as eye tracking, biometric monitoring, and environmental perception. However, existing multimodal interaction systems still face many technical bottlenecks in practical applications, affecting the efficiency and safety of the interaction.

[0003] Existing multimodal data processing architectures often struggle to address the spatiotemporal consistency issues of multi-source heterogeneous data. Cameras, radars, and biosensors integrated within vehicles typically have different sampling frequencies and communication delays, and their physical installation locations vary. Without high-precision hardware synchronization and strict spatial coordinate alignment, the gaze data, environmental parameters, and biometric features collected by the system often exhibit temporal deviations and spatial misalignments. This prevents backend algorithms from accurately associating information across different dimensions, thereby reducing the accuracy and stability of user intent recognition.

[0004] Traditional in-vehicle interfaces typically employ a fixed-layout graphical user interface design. Regardless of changes in the driving scenario or the driver's intentions, the position, size, and display attributes of controls on the interface remain unchanged. This static interaction mode ignores the driver's real-time focus and operational needs, forcing the driver to perform long-distance eye shifts and visual searches to locate target controls. This not only increases the driver's cognitive load and prolongs operation time but also increases the risk of the driver's gaze drifting off the road.

[0005] Furthermore, most existing interactive feedback mechanisms employ single or fixed preset modes, lacking the ability to adapt to the environment and user state. Systems often fail to automatically switch between appropriate feedback channels, such as tactile or visual feedback, based on drastic changes in in-vehicle noise levels or light intensity, causing feedback information to be masked or interfered with in specific environments. Simultaneously, existing systems lack high-priority safety arbitration mechanisms, allowing complex non-driving tasks even when the driver is deeply fatigued or the vehicle faces an emergency collision risk, failing to place driving safety at the core of the interaction process control. Summary of the Invention

[0006] The purpose of this invention is to provide a predictive interaction system and method based on multimodal perception, which at least solves the problems in the prior art, such as low accuracy of intent recognition due to spatiotemporal deviation of multi-source heterogeneous data, increased cognitive load and operational difficulty for drivers due to fixed interactive interface layout, and lack of adaptability of feedback mechanism to environment and user state, thus affecting interaction efficiency and safety.

[0007] The first aspect of the present invention provides a predictive interaction system based on multimodal perception, the system comprising: a multi-source data acquisition module, an intent prediction module, an interaction area dynamic adjustment module, a multimodal feedback decision module, and an adaptive strategy module;

[0008] The multi-source data acquisition module is used to acquire user eye tracking data, biometric data, and environmental parameter data of the vehicle's environment in real time, and to format and add timestamps to the acquired data.

[0009] The intent prediction module is connected to the multi-source data acquisition module to receive and fuse data, perform calculations through the prediction model, and generate probabilistic prediction results of user interaction intent.

[0010] The interactive area dynamic adjustment module is connected to the intent prediction module, which is used to reconstruct the layout of the user interface in real time based on high-probability intents.

[0011] The multimodal feedback decision module generates a combination of feedback strategies based on probabilistic prediction results and combined with environmental parameter data and user status.

[0012] The adaptive strategy module stores preset rules for a scenario and intervenes in or arbitrates the behavior of other modules when a trigger condition is detected.

[0013] In one specific implementation, to ensure strict alignment of multi-source heterogeneous data in the time dimension, the multi-source data acquisition module includes a gaze tracking unit, an environmental sensing unit, a biosensing unit, and a time synchronization unit. The time synchronization unit periodically sends synchronization trigger pulse signals to each of the aforementioned sensing units. Upon receiving the synchronization trigger pulse signal, the multi-source data acquisition module initiates data acquisition simultaneously and appends a uniform timestamp to each acquired frame of data, thereby eliminating data asynchrony errors caused by differences in sensor sampling rates or transmission delays.

[0014] In one specific implementation, the intent prediction module includes a spatiotemporal alignment submodule to address the inconsistency between the spatial references of the line-of-sight data and the environment data. This submodule employs an iterative nearest-point algorithm to align the line-of-sight coordinate system with the environment coordinate system. Its specific calculation logic is as follows: An error function is constructed based on the sum of squared distances between a 3D point in the line-of-sight coordinate system and its corresponding nearest point in the environment coordinate system. By minimizing this error function, the optimal rotation matrix and translation vector are obtained, thereby achieving coordinate system unification.

[0015] In one specific implementation, the intent prediction module employs a Long Short-Term Memory (LSTM) network structure as the prediction model to handle temporal features. This network structure receives a fused feature vector composed of multi-dimensional data, and outputs a hidden state through calculations involving a forget gate, input gate, cell state update, and output gate. This hidden state is further processed by a fully connected layer and transformed using a Softmax function to generate an intent probability vector. This probability vector represents the numerical likelihood of the user performing each potential interactive task.

[0016] In one specific implementation, the interactive area dynamic adjustment module uses the K-means clustering algorithm to process continuous gaze-focus data. By calculating the centroid of the cluster containing the most data points, it determines the center of the core gaze focus area. Based on this spatial positioning, when the probability value of a certain task in the intent probability vector exceeds a trigger threshold, the system adjusts the position of the target control associated with that task to the center of the core gaze focus area. Furthermore, the system performs visual enhancement processing on control attributes: enlarging the size of the target control according to a preset scaling factor; controlling the glow intensity of the target control to flash according to a periodic function that changes over time; and reducing the transparency of non-target controls according to a preset attenuation factor, thereby guiding user attention and reducing visual search load.

[0017] In one specific implementation, the multimodal feedback decision module dynamically selects the feedback channel based on the environment and user status. When the ambient noise exceeds a preset noise threshold, the system selects tactile feedback, sending a pulse width modulation signal to the linear resonant actuator to provide vibration cues, and adjusting the vibration intensity by changing the duty cycle of the pulse width modulation signal. When the ambient light is lower than a preset light threshold, the system prioritizes visual enhancement feedback to adapt to the visual perception needs in low-light environments.

[0018] In one specific implementation, the adaptive strategy module integrates a fatigue handling strategy. The triggering logic for this strategy is as follows: the user's heart rate variability is lower than a preset heart rate variability threshold, and the variance of grip strength within a preset time window is greater than a preset grip strength variance threshold. Once triggered, the system dynamically adjusts the interface display of the command interaction area module to simplify the interface and instructs the multimodal feedback decision module to generate vibration feedback simulating a heartbeat rhythm for non-visual disturbance-based status reminders.

[0019] In one specific implementation, the adaptive strategy module integrates an emergency freeze strategy. The triggering logic for this strategy is as follows: if the collision time calculated based on the relative distance and speed between the vehicle and the obstacle ahead is less than a preset safety threshold, the system immediately issues an interrupt signal to terminate the current interaction process and forcibly displays a warning message, ensuring that driving safety is the highest priority.

[0020] A second aspect of the present invention provides a predictive interaction method based on multimodal perception, comprising the following steps:

[0021] S1. Start the multi-source data acquisition module, and collect the user's gaze trajectory, biometrics and environmental parameters in parallel through the hardware synchronization mechanism, and perform time alignment and data fusion to form a time-series feature vector;

[0022] S2. The intent prediction module receives the temporal feature vector, uses the prediction model to perform inference calculations on the temporal feature vector, and outputs an intent probability vector representing the likelihood of the user performing different interactive operations.

[0023] S3. Based on the intent probability vector, the interactive area dynamic adjustment module reconstructs the control layout of the user interface according to the high-probability intent, while the multimodal feedback decision module generates a combination of feedback strategies based on the current environmental parameters and user status.

[0024] S4. The system executes the reconstructed interface display and the determined feedback strategy;

[0025] The adaptive strategy module monitors the system status in real time and intervenes in the module behavior in the above steps when it detects that the triggering conditions of the fatigue handling strategy or the emergency freeze strategy are met.

[0026] The above solution achieves the following beneficial technical effects:

[0027] This application addresses the temporal and spatial discrepancies in multi-source heterogeneous sensor data by sending trigger pulses through a hardware-level time synchronization unit and combining iterative nearest-point algorithm for spatiotemporal alignment. This mechanism ensures accurate fusion of gaze tracking, biometrics, and environmental parameters under the same spatiotemporal reference, providing high-quality input data for the intent prediction model and thus improving the system's accuracy in recognizing user interaction intents in complex dynamic scenarios such as vehicle movement.

[0028] This application achieves dynamic reconstruction of the interactive interface based on intent prediction results and gaze focus clustering analysis. The system can proactively move high-probability target controls to the user's core gaze focus area and automatically adjust the control size and glow effect, while reducing the salience of irrelevant elements. This design shortens the driver's gaze shift path and visual search time, reducing cognitive load and operational difficulty during driving.

[0029] This application integrates multimodal feedback decision-making and adaptive safety strategies, enabling dynamic adjustment of feedback channels and interaction logic based on ambient noise, light intensity, and the driver's physiological state. For example, it automatically switches to haptic feedback in high-noise environments, or forcibly interrupts non-core interaction processes when collision risk or fatigue characteristics are detected. This mechanism ensures the effectiveness of information transmission in different environments and guarantees that driving safety is given the highest priority in emergency or special situations. Attached Figure Description

[0030] Figure 1 This is a block diagram of the overall architecture of the adaptive interactive system of the present invention.

[0031] Figure 2 This is the overall flowchart of the adaptive interaction method of the present invention.

[0032] The module includes: a multi-source data acquisition module 100, an intent prediction module 200, an interactive area dynamic adjustment module 300, a multimodal feedback decision module 400, and an adaptive strategy module 500. Detailed Implementation

[0033] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] See attached document Figure 1The present invention provides a predictive interaction system based on multimodal perception. The predictive interaction system based on multimodal perception includes: a multi-source data acquisition module 100, an intent prediction module 200, an interaction area dynamic adjustment module 300, a multimodal feedback decision module 400, and an adaptive strategy module 500.

[0035] The multi-source data acquisition module 100 is used to acquire user gaze tracking data, biometric data, and environmental parameter data of the vehicle's surroundings in real time. This module is the data input source for the entire system; the various types of data it collects are formatted and timestamped to provide a basis for subsequent processing.

[0036] The data output of the multi-source data acquisition module 100 is connected to the intent prediction module 200. The intent prediction module 200 receives and integrates multi-dimensional data from the multi-source data acquisition module 100, performs calculations using a built-in prediction model, and generates probabilistic prediction results for the user's current and next interaction intent.

[0037] The output of the intent prediction module 200 is sent to the interactive area dynamic adjustment module 300 and the multimodal feedback decision module 400, respectively. This parallel processing mechanism ensures that the interface response and feedback strategy can be generated collaboratively.

[0038] The interactive area dynamic adjustment module 300 reconstructs the layout of the user interface in real time based on the high-probability intent given by the intent prediction module 200. This includes adjusting the position, size, priority, and visibility of controls to make the interactive elements that the user is interested in more easily accessible.

[0039] The multimodal feedback decision module 400, based on the prediction results of the intent prediction module 200 and combined with the environmental conditions (such as ambient noise and light intensity) and user conditions (such as fatigue level) directly obtained from the multi-source data acquisition module 100, generates the optimal combination of feedback strategies. This strategy determines whether the system should use one or more modalities—visual, tactile, or auditory—to convey information to the user.

[0040] The adaptive strategy module 500 manages and adjusts the overall operating status of the system. This module stores a series of preset rules for different scenarios, such as fatigue driving response strategies or emergency event response strategies. When a trigger condition is detected, the adaptive strategy module 500 intervenes in or arbitrates the behavior of other modules to ensure the safety and reliability of the interaction.

[0041] See attached document Figure 2 This invention provides a predictive interaction method based on multimodal perception, comprising the following steps:

[0042] In step S1, the system starts the multi-source data acquisition module 100, which, through a hardware synchronization mechanism, collects heterogeneous data from multiple sources in parallel, including user gaze trajectory, biometrics, environmental parameters, and vehicle dynamics. The collected raw data stream undergoes preliminary time alignment and data fusion to form a unified temporal feature vector containing multi-dimensional information.

[0043] In step S2, the intent prediction module 200 receives the temporal feature vector generated in step S1. The prediction model (e.g., Long Short-Term Memory Network LSTM) performs inference calculations on the temporal data and outputs a quantified user intent probability vector, which represents the likelihood of the user performing different interactive operations.

[0044] In step S3, the system executes two operations in parallel based on the intent probability vector output in step S2. On one hand, the interactive area dynamic adjustment module 300 reconstructs the controls of the user interface based on the high-probability intent and gaze heatmap analysis results, such as moving or enlarging high-probability target controls. On the other hand, the multimodal feedback decision module 400 selects or generates an optimal feedback combination strategy based on the current environment and user state, such as selecting haptic feedback in a high-noise environment.

[0045] In step S4, the system executes the interactive behavior determined in step S3. For example, haptic feedback is generated by driving actuators within the steering wheel, or visual enhancements to controls are achieved through a graphics rendering engine. After the interaction is complete, the system records all data from the entire interaction process, including the accuracy of intent prediction, the user's operation time, and whether the operation was successful, forming closed-loop data. This recorded data can be used for subsequent offline or online optimization and iteration of the prediction model.

[0046] In one specific embodiment, the implementation of the multi-source data acquisition module 100 of the present invention will be described in detail.

[0047] The function of the multi-source data acquisition module 100 is to provide the necessary, time-synchronized, multi-dimensional data for subsequent intent prediction and strategy generation. The multi-source data acquisition module 100 integrates multiple sets of different types of sensors and adopts a unified timing control mechanism to ensure the synchronization and consistency of data acquisition.

[0048] Specifically, the hardware configuration of the multi-source data acquisition module 100 includes a gaze tracking unit, an environmental sensing unit, and a biosensing unit.

[0049] The gaze tracking unit is used to acquire the user's three-dimensional head pose, gaze direction, and pupil physiological parameters. The gaze tracking unit can consist of an 850nm infrared camera and a 640×480 resolution time-of-flight (ToF) depth sensor. The infrared camera captures images of the user's eyes, while the ToF depth sensor acquires depth information of the scene; the two are combined to calculate the precise direction of the user's gaze in three-dimensional space.

[0050] An environmental sensing unit is used to monitor key environmental parameters inside and outside the vehicle. The environmental sensing unit may include a microphone array with a signal-to-noise ratio of at least 65 dB for acquiring the noise level of the vehicle's interior; a measurement range covering 1 to 10. 5 Lux's light sensor measures the light intensity inside the cockpit, while a millimeter-wave radar with a detection range of 150 meters is used to sense traffic conditions and potential risks ahead of the vehicle.

[0051] The biosensing unit is used to monitor biosignatures that reflect the user's physiological and psychological state. The biosensing unit may include a capacitive grip force sensor with an accuracy of ±5N mounted on the steering wheel to measure the magnitude and variation of the driver's grip force on the steering wheel, and a photoplethysmography (PPG) heart rate sensor with a sampling rate of 100Hz to monitor the user's heart rate and heart rate variability (HRV) in real time.

[0052] To address the issue of inconsistent sampling rates and data delays among different sensors, the multi-source data acquisition module 100 in this embodiment employs a hardware-triggered data synchronization mechanism. An independent time synchronization unit (e.g., a field-programmable gate array (FPGA)) can be configured within the system. This unit periodically sends synchronization trigger pulse signals to each sensor within the eye-tracking unit, environmental sensing unit, and biosensing unit. Each sensor initiates its current data acquisition at the same moment it receives the pulse signal.

[0053] After data acquisition is complete, each frame or data point is appended with a unified, high-precision timestamp provided by the time synchronization unit. This ensures that even with varying processing delays from different sensors, the data remains precisely aligned in time, guaranteeing the effectiveness of subsequent data fusion processing. The multi-source data acquisition module 100 ultimately outputs these synchronized and formatted data streams to the intent prediction module 200.

[0054] In one specific embodiment, the implementation of the intent prediction module 200 of the present invention will be described in detail.

[0055] The intent prediction module 200 receives and processes synchronous data streams from the multi-source data acquisition module 100, and predicts user interaction intents through data fusion and model inference. The intent prediction module 200 mainly consists of two core components: a spatiotemporal alignment submodule and a prediction model.

[0056] The spatiotemporal alignment submodule aims to unify the coordinate reference systems of data from different sensors. This embodiment employs the Iterative Closest Point (ICP) algorithm to align the line-of-sight coordinate system with the environment coordinate system. The goal of this algorithm is to find an optimal rotation matrix. Translation vector This makes the error function between the two point cloud sets... Minimize. The error function can be expressed as:

[0057] ;

[0058] in, It is the first in the line-of-sight coordinate system A three-dimensional point; It is the nearest point corresponding to it in the environmental coordinate system; It represents the number of points. By minimizing this function, the transformation relationship can be obtained, thereby achieving consistency in spatial data.

[0059] The prediction model in this embodiment employs a Long Short-Term Memory (LSTM) network structure. The model's input is a set of parameters at each time step. Fusion feature vector composed of multidimensional data The internal calculation process of this LSTM unit is as follows:

[0060] Forget Gate: Decides what information to discard from the cell state.

[0061] ;

[0062] Input gate: determines what new information is stored in the cell state.

[0063] ;

[0064] ;

[0065] Cell state update: The cell state is updated by combining the results of the forget gate and the input gate.

[0066] ;

[0067] Output gate: determines what value is output.

[0068] ;

[0069] ;

[0070] in, and These are the hidden state and the cell state from the previous moment, respectively. It is the input feature vector at the current moment. and These represent the weight matrix and bias vector for different gates, respectively. It is the Sigmoid activation function. It is the hyperbolic tangent activation function. It represents the Hadamardi (or Hadama) stack.

[0071] After processing by the LSTM network and subsequent fully connected layers, an output vector is obtained. This vector is transformed into the final intent probability vector using the Softmax function. :

[0072] ;

[0073] in, Representing the user to execute the first The probability value of each interactive task. This represents the total number of preset interactive tasks.

[0074] In one specific embodiment, the implementation of the interactive area dynamic adjustment module 300 of the present invention will be described in detail.

[0075] The interactive area dynamic adjustment module 300 first processes the continuous gaze point data using the K-means clustering algorithm. This is to determine the user's core visual focus area. The goal of the algorithm is to find... Clusters Minimize the sum of squares within clusters (WCSS):

[0076] ;

[0077] in, It is clustering The centroid of the cluster containing the most data points. It was identified as the center of the core visual focus area.

[0078] When the probability value of a certain task in the intent probability vector Exceeding the trigger threshold At this time, the UI controls associated with the task will be adjusted. Let the original position of the target control be... The size is Its adjusted new position and new size The calculation is as follows:

[0079] ;

[0080] ;

[0081] in, It is a preset scaling factor, such as 1.5.

[0082] For other non-target controls in the interface, their opacity (alpha channel value) will be adjusted:

[0083] ;

[0084] in, It is an attenuation coefficient, for example, 0.3.

[0085] For the dynamic glow effect of the target control, its glow intensity It can be generated by a time Controlled by changing periodic functions, such as the absolute value of a sine function:

[0086] ;

[0087] in, It is the maximum glow intensity; It is the flicker frequency, for example, 10Hz.

[0088] In one specific embodiment, the implementation of the multimodal feedback decision module 400 of the present invention will be described in detail.

[0089] The decision logic of this module is based on a predefined decision matrix, and its rules can be formalized as a set of logical judgments:

[0090] ;

[0091] ;

[0092] in, This is the noise threshold (e.g., 65dB). It is the light threshold (e.g., 300 lux).

[0093] The haptic feedback is implemented by sending a pulse-width modulation (PWM) signal to a linear resonant actuator (LRA). One cycle is... Duty cycle is PWM signal It can be represented as:

[0094] ;

[0095] By changing the duty cycle It can adjust the average intensity of vibration.

[0096] In one specific embodiment, the implementation of the adaptive strategy module 500 of the present invention will be described in detail.

[0097] The triggering conditions for fatigue management strategies can be formalized as follows:

[0098] ;

[0099] in, It is heart rate variability. It is a time window Variance of internal grip strength and These are the corresponding preset thresholds.

[0100] The emergency freeze strategy is determined by the collision time ( )trigger. The calculation formula is:

[0101] ;

[0102] in, It is the relative distance between this vehicle and the obstacle in front. It is the relative velocity between the two. When calculated... Less than the safety threshold The policy is activated at a time (e.g., 2.0 seconds):

[0103] ;

[0104] The technical solution of the present invention will be described below through a specific embodiment to illustrate its working process in a high-speed driving scenario during heavy rain.

[0105] Assume the vehicle is traveling normally on a highway, and the environment changes from sunny to rainy. The environmental sensing unit of the multi-source data acquisition module 100 detects rainfall information, while the ambient noise inside the vehicle, collected by the microphone array, increases to 70dB due to raindrop impact, and the light sensor measures a decrease in cabin illumination intensity. The eye-tracking unit records that the driver's gaze is primarily focused on the road ahead, but there are several brief glances towards the steering wheel or center console area.

[0106] The intent prediction module 200 receives and integrates the aforementioned time-series data. Its internal prediction model, based on rainfall signals in the environment and the user's visual search behavior towards the control area, calculates a generated intent probability vector. Within this vector, the probability value corresponding to the windshield wiper adjustment task is... The value increased to 0.85, which is much higher than the probability of other tasks such as adjusting the air conditioner or changing music.

[0107] When the probability value When the preset trigger threshold (e.g., 0.7) is exceeded, the interactive area dynamic adjustment module 300 is activated. This module enlarges the display size of the virtual control corresponding to the windshield wiper speed adjustment function by 1.5 times on the vehicle's head-up display (HUD) or instrument panel, and adds a blue glowing border with a frequency of 10Hz. Simultaneously, the display brightness of other unrelated controls in the interface, such as media playback information, is reduced to 30% of its original value.

[0108] Meanwhile, the multimodal feedback decision module 400 formulates a feedback strategy based on the collected environmental parameters. Since the ambient noise exceeds the 65dB threshold, its internal decision matrix determines that auditory feedback is not appropriate at this time. Therefore, module 400 generates a tactile feedback command, ready to confirm the user's action via the tactile channel.

[0109] When the user confirms the interaction intention through eye contact (e.g., staring at the highlighted wiper control for more than a preset 0.8 seconds) or by pressing a physical button on the steering wheel, the system executes the command to adjust the wiper speed. At the instant the command is successfully executed, linear resonant actuators mounted at the 3 o'clock and 9 o'clock positions on the steering wheel simultaneously generate two short vibrations, each lasting 100 milliseconds, as confirmation feedback of successful operation.

[0110] In the scenario of this embodiment, if the millimeter-wave radar in front of the vehicle detects a vehicle braking suddenly ahead, resulting in a calculated collision time ( If the time exceeds 2.0 seconds, the emergency freeze strategy of the adaptive strategy module 500 will be triggered. This module will immediately issue an interrupt signal, terminate the current wiper adjustment interaction process, and force the collision warning information to be displayed with the highest priority on all display interfaces to ensure that the driver's attention is fully focused on avoiding danger.

[0111] The technical solution of the present invention will be described below through a specific embodiment to illustrate its working process in the light and dark switching scenario at the tunnel entrance and exit.

[0112] In this embodiment, the system utilizes predictive information in conjunction with physiological characteristics to address the interference caused by drastic changes in lighting conditions on the driver's vision and interaction. When the vehicle is traveling on a highway and is about to enter a tunnel, the system uses onboard GPS data and route planning information to predict the presence of a tunnel ahead. Simultaneously, the forward-facing camera in the environmental sensing unit detects the tunnel entrance features ahead using image recognition algorithms.

[0113] The anticipated information triggers the preset tunnel mode strategy within the adaptive strategy module 500. This module sends a command to the interactive area dynamic adjustment module 300 to pre-adjust the display scheme of the user interface. Specifically, before the vehicle enters the tunnel, the interactive area dynamic adjustment module 300 switches the color mode of all display interfaces (such as the instrument panel and HUD) from daytime mode (such as black text on a white background) to nighttime mode (such as white text on a black background) and increases their overall brightness to the preset tunnel mode brightness level.

[0114] The moment the vehicle enters the tunnel, the gaze tracking unit of the multi-source data acquisition module 100 detects that the driver's pupil diameter dilates dramatically within a short period (e.g., within 0.5 seconds), with the rate of change exceeding a preset physiological adaptation threshold. This pupil dilation rate data is used as a feedback signal to confirm the effectiveness of the pre-loaded interface scheme.

[0115] If the pupil takes too long to return to its normal diameter after dilation, the system can further enhance the contrast or brightness of key information to assist the driver in acquiring information during dark adaptation.

[0116] As the vehicle approaches the tunnel exit, the system executes the reverse process. GPS data and the forward-facing camera pre-identify the tunnel exit, the adaptive strategy module 500 reactivates the tunnel mode strategy, and the command interaction area dynamic adjustment module 300 pre-emptively restores the interface to daytime mode adapted to the high-light environment. This pre-adjustment operation prevents glare interference to the driver caused by excessive brightness of the internal display interface when the vehicle exits the tunnel.

[0117] The technical solution of the present invention will be described below through a specific embodiment to illustrate its working process in urban congestion and fatigue driving scenarios.

[0118] In this embodiment, the vehicle travels at low speeds for an extended period in congested urban areas. The multi-source data acquisition module 100 continuously monitors the driver's physiological indicators through its integrated biosensor unit. Within a continuous 5-minute time window, the system detects that the driver's heart rate variability (HRV) remains below 50 milliseconds, while the grip force sensor on the steering wheel displays a grip force fluctuation exceeding 20%.

[0119] The aforementioned combination of physiological indicators reached the fatigue driving judgment threshold preset within the adaptive strategy module 500. Therefore, the adaptive strategy module 500 activated its fatigue driving response strategy.

[0120] After the strategy is activated, the adaptive strategy module 500 first sends an instruction to the interactive area dynamic adjustment module 300, requesting simplification of the user interface. The interactive area dynamic adjustment module 300 executes this instruction, hiding or setting all non-essential elements (such as media playback information and detailed navigation maps) on the display interface, except for core driving information (such as vehicle speed, gear position, and distance to the vehicle in front). The purpose of this operation is to reduce the driver's visual information processing load.

[0121] Simultaneously, the adaptive strategy module 500 sends a command to the multimodal feedback decision module 400, requesting the execution of a fatigue warning. Based on the command, the multimodal feedback decision module 400 sends a PWM drive signal to the linear resonant actuator installed in the steering wheel.

[0122] The drive signal is designed to cause the actuator to generate a low-frequency pulsed vibration that mimics a heartbeat rhythm. This non-intrusive biorhythmic feedback aims to alert the driver to their physiological state through tactile feedback, guiding them to take necessary measures such as rest to ensure driving safety.

[0123] It should be understood that the above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0124] The core technical solution of this invention lies in constructing a closed-loop adaptive interaction system. This system improves the efficiency and safety of human-computer interaction in a dynamically changing driving environment through the synergistic effect of proactive prediction, interface reconstruction, and multimodal feedback.

[0125] Regarding the implementation of proactive prediction, the above embodiments employ a Long Short-Term Memory (LSTM) network as the prediction model. However, the present invention is not limited to this implementation. Any machine learning model capable of processing time-series data and outputting probabilistic predictions of future events, such as a Gated Recurrent Unit (GRU), a Transformer network, or a Dynamic Bayesian network, can be used to replace the above prediction model to achieve prediction of user interaction intentions.

[0126] The core of dynamically adjusting the interactive area lies in changing the interactivity and prominence of interface elements based on the predicted results and the user's visual focus. The above embodiments achieve this through methods such as moving, scaling, and changing transparency, but this is not the only approach. Any other graphics methods that can highlight the target control and weaken non-target areas, such as applying Gaussian blur to the background, changing the color saturation of the target control, or adding dynamic textures, as long as they can guide the user's attention, fall within the scope of this invention.

[0127] The key to implementing multimodal feedback lies in selecting the optimal feedback channel based on the environment and user state. The above embodiment employs a rule-based decision matrix and a linear resonant actuator (LRA). In other embodiments, this decision-making mechanism can also be implemented by a trained classification model. Furthermore, the actuator for haptic feedback is not limited to an LRA; other types of vibration elements such as an eccentric rotor motor (ERM) and piezoelectric ceramic actuators can also be used. Its installation location is not limited to the steering wheel and can be extended to seats, pedals, and other locations that come into contact with the user's body.

[0128] Regarding the implementation of the security mechanism, its essence is to establish an arbitration layer that transcends conventional interaction logic to handle high-risk events. The above embodiments illustrate fatigue detection based on heart rate variability (HRV) and fatigue detection based on collision time (HRV). Emergency event detection is used. However, the physiological and environmental indicators used to determine fatigue status or emergency level are not limited to this. For example, eye movement parameters (such as blink frequency and saccade speed) can also serve as supplementary evidence for fatigue assessment; and the determination of emergency events can also comprehensively consider more dimensions of information such as vehicle lateral acceleration and road curvature. The triggered response strategies are not limited to extending the interaction time or freezing the interface, but may also include actively activating driver assistance functions or sending alerts to a remote monitoring center.

[0129] For ease of description, the above system is described by dividing it into various units and modules based on their functions. Of course, in implementing this application, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0130] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0131] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0132] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pre-emptive interaction system based on multi-modal awareness, characterized in that, The system comprises a multi-source data acquisition module, an intention prediction module, an interactive area dynamic adjustment module, a multi-modal feedback decision module, and an adaptive strategy module. The multi-source data acquisition module is configured to acquire user's gaze tracking data, biological feature data, and environmental parameter data of the environment in which the vehicle is located in real time, and format and add time stamps to the acquired data. The intention prediction module is connected to the multi-source data acquisition module and configured to receive and fuse data from the multi-source data acquisition module, calculate by a prediction model, and generate a probabilistic prediction result of user's interactive intention. The interactive area dynamic adjustment module is connected to the intention prediction module and configured to reconstruct the layout of the user interactive interface in real time according to the high-probability intention output by the intention prediction module. The multi-modal feedback decision module is connected to the intention prediction module and the multi-source data acquisition module and configured to generate a feedback strategy combination according to the probabilistic prediction result and in combination with the environmental parameter data and user state. The adaptive strategy module is configured to store preset rules for scenarios and intervene or arbitrate the behaviors of other modules when a trigger condition is detected. The multi-source data acquisition module comprises a gaze tracking unit, an environmental sensing unit, a biological sensing unit, and a time synchronization unit.

2. A multimodal awareness based anticipatory interaction system as claimed in claim 1, wherein, The time synchronization unit is configured to periodically send a synchronization trigger pulse signal to the gaze tracking unit, the environmental sensing unit, and the biological sensing unit; and the multi-source data acquisition module starts data acquisition at the same time when the synchronization trigger pulse signal is received and adds a uniform time stamp to each frame of data collected. The intention prediction module comprises a space-time alignment sub-module.

3. A multimodal awareness based anticipatory interaction system as claimed in claim 1, wherein, The space-time alignment sub-module aligns the gaze coordinate system and the environmental coordinate system by using an iterative closest point algorithm to obtain an optimal rotation matrix and translation vector by minimizing an error function. The error function is constructed based on the square sum of distances between three-dimensional points in the gaze coordinate system and corresponding closest points in the environmental coordinate system. The prediction model in the intention prediction module adopts a long short-term memory network structure.

4. A multimodal awareness based anticipatory interaction system as claimed in claim 1, wherein, The long short-term memory network structure receives a fusion feature vector composed of multi-dimensional data, and outputs a hidden state through the calculation process of a forgetting gate, an input gate, a cell state update, and an output gate. The hidden state is processed by a full connection layer and converted by a Softmax function to generate the probabilistic prediction result, which is an intention probability vector. The interactive area dynamic adjustment module processes continuous gaze landing point data by a K-means clustering algorithm, and the center of the cluster containing the most data points is determined as the center of the core gaze focal point area.

5. A multimodal awareness based anticipatory interaction system as claimed in claim 1, wherein, When the probability value of a task in the intention probability vector exceeds a trigger threshold, the interactive area dynamic adjustment module adjusts the position of a target control associated with the task to the center of the core gaze focal point area. The interactive area dynamic adjustment module adjusts the target control and non-target control as follows:

6. A multimodal awareness based anticipatory interaction system as claimed in claim 5, wherein, The size of the target control is enlarged by a preset scaling factor. ​ The luminous intensity of the target control is controlled to flash according to a periodic function varying with time; The transparency of the non-target control is reduced according to a preset attenuation coefficient.

7. A multimodal awareness based anticipatory interaction system as claimed in claim 1, wherein, The multi-modal feedback decision module generates a feedback strategy based on a predefined decision matrix: When the ambient noise exceeds a noise threshold, a haptic feedback is selected, which is realized by sending a pulse width modulation signal to a linear resonant actuator and adjusting the vibration intensity by changing the duty cycle of the pulse width modulation signal; When the ambient light is lower than a light threshold, a visual enhancement feedback is selected.

8. A multimodal awareness based anticipatory interaction system as claimed in claim 1, wherein, The adaptive strategy module stores a fatigue processing strategy; The trigger condition of the fatigue processing strategy is that the heart rate variability of the user is lower than a preset heart rate variability threshold, and the variance of the grip force in a preset time window is greater than a preset grip force variance threshold; When the fatigue processing strategy is triggered, the adaptive strategy module sends an instruction to the interactive area dynamic adjustment module to simplify the interface, and sends an instruction to the multi-modal feedback decision module to generate a vibration feedback simulating the rhythm of a heartbeat.

9. A multimodal awareness based anticipatory interaction system as claimed in claim 1, wherein, The adaptive strategy module stores an emergency freezing strategy; The trigger condition of the emergency freezing strategy is that the collision time calculated according to the relative distance and relative speed between the vehicle and the front obstacle is less than a preset safety threshold; When the emergency freezing strategy is triggered, the adaptive strategy module sends an interrupt signal to terminate the current interaction process and forcibly displays a warning information.

10. A pre-emptive interaction method based on multi-modal awareness, characterized in that, The method comprises the following steps: S1, starting a multi-source data acquisition module, collecting the user's gaze trajectory, biological characteristics and environmental parameters in parallel through a hardware synchronization mechanism, and performing time alignment and data fusion to form a time sequence feature vector; S2, the intention prediction module receives the time sequence feature vector, and uses a prediction model to infer and calculate the time sequence feature vector, outputting an intention probability vector representing the possibility of the user performing different interaction operations; S3, based on the intention probability vector, the interactive area dynamic adjustment module reconstructs the control layout of the user interface according to the high-probability intention, and the multi-modal feedback decision module generates a feedback strategy combination according to the current environmental parameters and user state; S4, the system executes the reconstructed interface display and the determined feedback strategy; The adaptive strategy module monitors the system state in real time, and intervenes in the module behavior in the above steps when the trigger conditions of the fatigue processing strategy or the emergency freezing strategy are met.

Images