A multi-modal human-computer intelligent interaction system based on eye tracking

Abstract

This invention discloses a multimodal human-computer intelligent interaction system based on eye tracking, belonging to the field of human-computer intelligent interaction technology. It includes an eye-tracking region modeling module, an interaction trigger execution module, and a trigger region update module. By dynamically modeling the eye-tracking trigger region and combining vehicle driving parameters, user visual characteristics, and task priorities, the system adapts the position, size, and response threshold of the trigger region in real time. This reduces the risk of the user's gaze deviating from the road and avoids accidental touches, balancing interaction efficiency and driving focus. Simultaneously, a multimodal mechanism is employed, matching head movements, short voice prompts, and other confirmation methods to different tasks, combined with visual and tactile dual feedback. This solves the problem of misoperation in single eye-tracking interaction while maintaining ease of operation and accuracy. Furthermore, the dynamic updating and smooth transition design of the trigger region intelligently adjusts the update frequency according to scene changes and avoids discomfort from abrupt changes in the region through linear transitions, ensuring real-time adaptation to complex scenes.

Description

Technical Field

[0001] This invention relates to the field of human-computer intelligent interaction technology, specifically to a multimodal human-computer intelligent interaction system based on eye tracking. Background Technology

[0002] With the rapid iteration of intelligent driving technology, in-vehicle human-machine interaction systems have become one of the core components for improving driving experience and driving safety. Traditional in-vehicle interaction methods (such as touch screens and physical buttons) have the drawback of distracting the driver's attention: in complex scenarios such as high speeds and curves, the moment the user's eyes deviate from the road, a safety risk may be triggered. Moreover, existing interaction systems mostly use fixed areas and single trigger logic, which are difficult to adapt to dynamically changing driving environments.

[0003] Currently, eye-tracking technology, due to its "non-contact and low-interference" characteristics, has gradually become a research hotspot in in-vehicle interaction. However, existing eye-tracking interaction solutions still have significant limitations: First, the design of trigger areas is mostly static, without dynamic adjustment based on driving scenario parameters (such as vehicle speed and road curvature). When the vehicle is at high speed or on a sharp bend, the trigger logic of a fixed small area increases the difficulty for users to focus their gaze, while a fixed large area is prone to accidental touches. Second, single eye-tracking triggers lack multimodal confirmation mechanisms. Interaction methods that rely solely on gaze placement are prone to misoperation when the user's gaze naturally drifts, making it difficult to match the needs of interaction tasks with different priorities. Third, the update of trigger areas lacks a smooth transition when the scene changes. Abruptly changing interaction areas can interfere with users' visual habits and even exacerbate the risk of driver distraction.

[0004] Meanwhile, users' visual attention resources fluctuate dynamically during driving: as vehicle speed increases and road curvature increases, users need to allocate more visual resources to the road environment, significantly reducing the proportion of visual resources available for interaction. However, existing systems lack a mapping model between visual attention and interaction areas, making it impossible to dynamically optimize interaction triggering conditions based on real-time visual resources. Furthermore, different interaction tasks (such as emergency navigation adjustments and music switching) have significantly different requirements for response efficiency and error tolerance. Existing solutions do not match differentiated confirmation modalities to task priorities, resulting in delayed responses for high-priority tasks or a high error rate for low-priority tasks. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal human-computer intelligent interaction system based on eye tracking, which solves the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal human-computer intelligent interaction system based on eye tracking, comprising:

[0007] The eye-tracking region modeling module is used to collect basic driving scenario parameters, including a set of vehicle driving parameters, a set of user visual feature parameters, and interaction task type parameters. Then, based on the basic driving scenario parameters, the visual attention weight is determined. Next, based on the visual attention weight and the set of user visual feature parameters, the center coordinates of the trigger area are determined, and then the area of ​​the trigger area and the response threshold are determined. The interaction trigger execution module first determines the validity of the eye movement behavior when the user's gaze enters the trigger area after dynamic modeling, and then matches the corresponding preset confirmation modality according to the interaction task type parameter. The trigger area update module is used to update the trigger area in real time when the driving scenario parameters change; As a further aspect of the present invention: the set of vehicle driving parameters is labeled as S={s1,s2,s3}; where s1 represents the current vehicle speed; s2 represents the current road curvature, and for a straight road, s2=0; s3 represents the current ambient light intensity; simultaneously, the set of user visual feature parameters is labeled as V={v1,v2}; where v1 represents the angle between the user's line of sight focus and the road in front of the vehicle, and when the user's line of sight is directly in front, v1=0; v2 represents the frequency of the user's line of sight movement per unit time; and the interaction task type parameter is labeled as T, representing the priority of the interaction task to be executed; where the interaction task type parameter takes the value T∈{1,2,3}, where 1 is high priority, 2 is medium priority, and 3 is low priority.

[0008] As a further aspect of the present invention, the visual attention weight is calculated as follows: pass: Calculate the user's visual attention weight W in the current scene; Among them, the visual attention weight W is used to characterize the proportion of visual resources that the user can allocate to the interactive interface; In the formula, This refers to normalizing the driving speed, with a weighting of 0.6. 120 refers to the highway speed limit of 120 km / h. This refers to normalizing the road curvature, with a weighting of 0.3; 0.1 refers to the upper limit of the sharp curvature rate being 0.1 rad / m. This refers to normalizing the frequency of eye movement, with a weighting of 0.1, and 5 refers to the upper limit of high-frequency movement being 5 times / second.

[0009] As a further aspect of the present invention, the determination method of the center coordinates, area, and response threshold of the triggering region is as follows; pass: Calculate the center coordinates (x0, y0) of the trigger area. In the formula, (x f ,yf ) refers to the projected coordinates of the user's current gaze focus on the interactive screen, with the center of the screen as the origin; W×5 refers to the distance between the trigger area and the gaze focus; cos(v1) and sin(v1) represent the direction of the trigger area relative to the gaze focus, which is consistent with the user's gaze direction; pass To determine the validity of the eye movement behavior; In the formula, E is the effectiveness index, and t s This refers to the actual time the user's gaze lingers within the trigger area; t refers to the response threshold in the current scenario; v0 is the gaze direction corresponding to the trigger area. Refers to the normalized value of the deviation in line of sight; When E ≥ 0.8, the eye movement behavior is considered valid; Otherwise, it will be considered a mistaken touch and no action will be triggered.

[0010] As a further aspect of the present invention, the matching method for modal confirmation is as follows: Based on the interaction task type T, select the corresponding preset confirmation modality, according to the following rules: When T=1, i.e. for high-priority interactive tasks: head movement confirmation is used, and the head movement angle parameter H={h1,h2} is defined, where h1 is the left and right deflection angle and h2 is the up and down deflection angle; The confirmation condition is that |h1| ≤ 5° and |h2| ≤ 3°; that is, if it is a nod or a shake of the head, then the confirmation is complete. When T=2, for medium-priority interactive tasks: short voice confirmation is used. The confirmation condition is that the voice input duration is between 0.3 and 0.8 seconds, and the content is "yes" if executed, and "no" if cancelled. (If the duration exceeds the limit or the content is inconsistent, no response will be given.) When T=3, i.e. for low-priority interactive tasks, eye-tracking secondary focusing confirmation is used. The confirmation condition is the second dwell time t of the gaze within the trigger area. s2 ≥0.5×t.

[0011] As a further aspect of the present invention: after completing a valid trigger and confirmation, the interaction trigger execution module executes the corresponding interaction task and informs the user through visual feedback and tactile feedback, with the following feedback parameters: Visual feedback: Display the completion animation within the triggered area. Animation duration: ; Haptic feedback: Achieved through steering wheel vibration; vibration frequency: Vibration duration: .

[0012] As a further aspect of the present invention, the process for triggering a region update is as follows: Firstly, through: The change in parameters ΔS={Δs1,Δs2,Δs3} is calculated to determine whether the scene has changed. Among them, s 1，新 This refers to the vehicle speed collected at the current moment, i.e., the latest vehicle speed data, s 1，旧 This refers to the vehicle speed collected at the previous moment, i.e., the previously recorded vehicle speed data; s 2，新 This refers to the road curvature measured at the current moment, s 2，旧 This refers to the road curvature collected at the previous moment; s 3，新 This refers to the ambient light intensity collected at the current moment, s 3，旧 This refers to the ambient light intensity collected at the previous moment; Next, based on the type of scene change, that is, through: Determine the update frequency F of the trigger region; Among them, 20 is the preset judgment criterion for changes in vehicle speed; 0.05 is the preset judgment criterion for changes in road curvature; and 500 is the preset judgment criterion for changes in illumination. Then a linear transition algorithm is used, that is: Calculate the center coordinates (x, y) of the trigger area during the transition process. t ,y t ); Among them, t t The total transition time is a predefined, fixed value; t p This represents the time elapsed during the transition process, and 0 ≤ t. p ≤t t ; At the same time, the size A of the trigger area t and response threshold t t The same linear transition algorithm logic is used for updating.

[0013] As a further aspect of the present invention, the method for determining whether the scene has changed is as follows: When Δs1≥20km / h, the speed scenario changes; When Δs2 ≥ 0.05 rad / m, the road curvature scenario is determined to have changed; When Δs3≥500lux, it is determined that the lighting scene has changed.

[0014] (III) Beneficial Effects This invention provides a multimodal human-computer intelligent interaction system based on eye tracking. Compared with existing technologies, it has the following advantages: This invention dynamically models the eye-tracking trigger area and adjusts its position, size, and response threshold in real time based on vehicle driving parameters, user visual characteristics, and task priorities. For example, in high-speed or sharp-curve scenarios, the trigger area is expanded and the response threshold is extended, reducing the risk of the user's gaze deviating from the road and preventing accidental touches. This adaptation logic, centered on visual attention weighting, ensures that the interface always meets the safety requirements of driving scenarios, effectively balancing interaction efficiency and driving focus, and reducing potential dangers caused by distracted vision.

[0015] The multimodal interaction mechanism significantly improves the reliability and convenience of human-computer interaction. The system adopts a combination of "eye-tracking triggering + corresponding modal confirmation," matching head movements, short voice messages, and secondary eye-tracking confirmation methods for high, medium, and low priority tasks. This not only solves the problem of misoperation in single eye-tracking interaction but also enhances the interactive perception through visual and tactile dual feedback. For example, high-priority tasks use head-tracking confirmation, which is both fast and requires no additional eye shift; low-priority tasks use secondary eye-tracking confirmation, balancing operational simplification and accuracy, allowing different types of driving interactions to adapt to the pace of the scene.

[0016] Dynamic updates and smooth transitions optimize the system's usability and user experience. The trigger area update module determines the update timing based on scene parameter change thresholds and uses a linear transition algorithm to ensure smooth area adjustments, avoiding visual discomfort caused by abrupt changes in the trigger area. Simultaneously, the update frequency dynamically adjusts according to the degree of scene change, ensuring real-time adaptation in complex scenarios while reducing power consumption in stable scenarios, balancing system performance and resource consumption. This adaptive dynamic adjustment logic allows the interactive system to continuously adapt to the dynamic changes during driving, improving long-term comfort and stability. Attached Figure Description

[0017] Figure 1 This is a system block diagram of a multimodal human-computer intelligent interaction system based on eye tracking according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 As shown, the embodiments of the present invention provide the following technical solutions: As an embodiment of the present invention: This invention relates to a multimodal human-computer intelligent interaction system based on eye tracking, comprising: The eye-tracking region modeling module is used to collect basic driving scenario parameters, including a set of vehicle driving parameters, a set of user visual feature parameters, and interaction task type parameters. Then, based on the basic driving scenario parameters, the visual attention weight is determined. Next, based on the visual attention weight and the set of user visual feature parameters, the center coordinates of the trigger region are determined, and then the area of ​​the trigger region and the response threshold are determined. The core of this module is to build a mapping model between visual attention distribution and eye-tracking trigger areas in driving scenarios. By collecting driving state parameters in real time, the position, size and response threshold of the trigger areas are dynamically adjusted to prevent the user's gaze from deviating from the road. The specific method is as follows: Step 1.1: Real-time acquisition of driving scenario parameters: First, collect parameters for three basic driving scenarios, defined as follows: The set of vehicle driving parameters is S = {s1, s2, s3}. Where s1 represents the current vehicle speed; s2 represents the current road curvature, and for a straight road, s2=0; s3 represents the current ambient light intensity. User visual feature parameter set V={v1,v2}; Where v1 refers to the angle between the user's line of sight and the road in front of the vehicle. When the user's line of sight is directly in front, v1=0; v2 refers to the frequency of line of sight movement per unit time specified by the user. The interaction task type parameter T indicates the priority of the interaction task to be executed. In this embodiment, the interaction task type parameter takes the value T∈{1,2,3}, where 1 is high priority, such as emergency navigation adjustment; 2 is medium priority, such as answering or hanging up incoming calls; and 3 is low priority, such as music switching. Step 1.2, Calculation of visual attention weights: pass: Calculate the user's visual attention weight W in the current scene; Among them, the visual attention weight W is used to characterize the proportion of visual resources that the user can allocate to the interactive interface; In the formula, This refers to normalizing driving speed; the higher the speed, the less visual resources are available, with a weighting of 0.6. 120 refers to the upper limit of 120 km / h for highways. This refers to normalizing the road curvature; the greater the curvature, the lower the proportion of visual resources, with a weight of 0.3. 0.1 means that the upper limit of sharp curvature is 0.1 rad / m. This refers to normalizing the frequency of eye movement. The higher the frequency, the lower the proportion of visual resources. The weight is 0.1. 5 refers to the upper limit of 5 times / second for high-frequency movement. The final value range of W is [0.2, 0.9]. In this embodiment, during the above calculation process, 120, 0.1, and 5 are all preset reference values ​​used to normalize different types of scene parameters so that they participate in the calculation on the same order of magnitude. These values ​​can be adjusted according to actual application scenarios, vehicle performance, road conditions or user habits, and are not limited to the specific values ​​given in this implementation method. Specifically: 120 represents the upper limit of the vehicle's speed reference, in km / h, and is used to normalize the actual vehicle speed to between 0 and 1. In this embodiment, it is assumed that the maximum speed limit of the highway is 120 km / h, so it is used as the benchmark value for speed normalization; In practical applications, this value can be replaced according to the road speed limit standards of different countries or regions, for example, it can be set to 100, 110 or 130.

[0020] 0.1 represents the upper limit of the road curvature reference, in rad / m, and is used to normalize the road curvature; In this embodiment, it is assumed that the maximum curvature of the sharp bend is about 0.1 rad / m, so it is used as the benchmark value for curvature normalization. The greater the curvature, the more obvious the road bend, and the more visual attention the user needs to pay to observe the road. Therefore, the proportion of visual resources available for interaction is lower. This value can also be adjusted according to the actual road design standards or the typical road environment in which the vehicle travels. For example, it can be set to 0.08, 0.12, etc.

[0021] 5 represents the upper limit of the reference range for the frequency of user eye movement, in units of times per second, used to normalize the frequency of eye movement. In this embodiment, it is assumed that the user's gaze moves rapidly at a frequency of approximately 5 times per second during driving, and this is used as the baseline value for normalizing the gaze movement frequency. The higher the gaze movement frequency, the more dispersed the user's current visual attention is, and the less visual resources are available for interaction. This value can also be adjusted according to the characteristics of the user group or experimental statistical results, for example, it can be set to 4, 6, etc.

[0022] By introducing these adjustable reference values, the system can adapt to different driving environments and user needs, improving the model's flexibility and versatility.

[0023] Furthermore, 0.6, 0.3, and 0.1 are pre-set parameter weight percentages used to distinguish the degree of influence of different scenario parameters on the allocation of user visual resources; These percentages can be adjusted based on actual driving safety needs and user behavior data statistics; they are not fixed values.

[0024] Specifically: A percentage of 0.6 corresponds to the driving speed parameter, indicating that in driving scenarios, vehicle speed has the highest impact on user visual attention; a percentage of 0.3 corresponds to the road curvature parameter, indicating that the degree of road curvature has the second highest impact on visual attention; and a percentage of 0.1 corresponds to the gaze movement frequency parameter, indicating that its impact on visual attention is relatively low. In practical applications, these percentages can be adjusted to other reasonable values, such as 0.5, 0.35, or 0.15, depending on the characteristics of different vehicle models and driving scenarios, to adapt to the visual attention distribution patterns in different scenarios.

[0025] Step 1.3: Dynamically adjust the position of the trigger area: Based on the visual attention weight W and the user's current gaze focus v1, the center coordinates (x0, y0) of the trigger area are calculated using the following formula: ; In the formula, (x f ,y f W×5 refers to the projected coordinates of the user's current gaze focus on the interactive screen, with the screen center as the origin; W×5 refers to the distance between the trigger area and the gaze focus. The 5 in W×5 is a preset distance reference value in centimeters. This value is a configurable parameter that can be adjusted according to screen size, interactive interface layout, user habits, or vehicle type. For example, it can be set to 4, 6, or other suitable values ​​to control the maximum possible distance between the trigger area and the user's gaze focus. The larger W is, the more visual resources there are, and the distance can be increased appropriately; cos(v1) and sin(v1) represent the direction of the trigger area relative to the gaze focus, which is consistent with the user's gaze direction to avoid large deviations in gaze. Step 1.4: Adjust the size of the trigger area and the response threshold: The size of the triggering region and the response threshold are adjusted based on the visual attention weight W. The formula is: ; In the formula, A represents the size of the trigger region, i.e., the area A of the trigger region, and t represents the response threshold of the trigger region, i.e., the eye movement dwell time. In this embodiment, the value of A ranges from [1.4, 2.8], and the value of t ranges from [1.7, 7.5]. The larger W is, the smaller the trigger area can be, allowing users to focus more visual resources on the smaller area. At the same time, the response threshold can be shortened, making the interaction more efficient. The smaller the W value, the larger the trigger area needs to be in high-speed or sharp-curve scenarios to reduce the difficulty of focusing the eye, and the longer the response threshold needs to be to avoid accidental touches. The interaction trigger execution module is used to determine the validity of the eye movement behavior when the user's gaze enters the trigger area of ​​the dynamic model, and then match the corresponding preset confirmation modality according to the interaction task type parameter. A multimodal interaction method combining eye-tracking triggering and voice / head movement confirmation is adopted to avoid the risk of misoperation from single eye-tracking interaction. The specific steps are as follows: Step 2.1, Determining the validity of eye-tracking triggers: When the user's gaze enters the trigger area after dynamic modeling, it first proceeds through: , Determine the validity of this eye movement behavior; In the formula, E is the effectiveness index, and t s This refers to the actual time the user's gaze lingers within the trigger area; t refers to the response threshold in the current scenario; v0 is the gaze direction corresponding to the trigger area. Refers to the normalized value of the deviation in line of sight; The 10 in the figure is the reference upper limit for the deviation of the line of sight. It is a pre-set maximum acceptable line of sight deviation threshold. This value is an adjustable reference value and can be replaced according to the safety requirements of the driving scenario, the requirements for interaction accuracy, etc. For example, it can be adjusted to 8°, 12°, etc. The unit is degrees. It is used to normalize the actual deviation value of the line of sight. In this embodiment, the effectiveness decreases significantly when the deviation exceeds 10°; When E≥0.8, the eye-tracking operation is determined to meet the triggering conditions, and the system will continue to execute the subsequent interaction process; Otherwise, it will be considered a misclick and no action will be triggered; When E < 0.8, the eye-tracking operation is deemed invalid, and the system does not respond to the interaction request. Among them, 0.8 is a pre-set validity judgment threshold. This threshold is an adjustable reference value and can be replaced according to the accuracy requirements of the interaction and the safety standards of the driving scenario. For example, it can be adjusted to 0.75, 0.85, etc. Step 2.2, Priority matching for multimodal verification: Based on the interaction task type T, the corresponding preset confirmation modality is automatically selected, according to the following rules: When T=1, i.e. for high-priority interactive tasks: head movement confirmation is used, and the head movement angle parameter H={h1,h2} is defined, where h1 is the left and right deflection angle and h2 is the up and down deflection angle; The confirmation condition is that |h1| ≤ 5° and |h2| ≤ 3°; that is, if it is a nod or a shake of the head, then the confirmation is complete. In the confirmation conditions, 5° and 3° are both preset angle judgment thresholds used to constrain the range of specific parameters, such as representing the user's line of sight offset limit in the horizontal and vertical directions; The angle judgment threshold is an adjustable reference value, which can be replaced with other reasonable values ​​according to the device accuracy and the needs of the interaction scenario, such as adjusting it to |h1|≤6°, |h2|≤4°, etc. When T=2, that is, for medium-priority interactive tasks: short voice confirmation is used. The confirmation condition is that the voice input duration is between 0.3 and 0.8 seconds and the content is "yes" to execute, and "no" to cancel. If the response time exceeds the specified limit or the content is incorrect, no response will be provided. When T=3, i.e. for low-priority interactive tasks, eye-tracking secondary focusing confirmation is used. The confirmation condition is the second dwell time t of the gaze within the trigger area. s2 ≥0.5×t; Step 2.3, Execution and Feedback of Interactive Tasks: After successful triggering and confirmation, the corresponding interactive task is executed, and the user is notified through visual and tactile dual-modal feedback. The feedback parameters are as follows: Visual feedback: Display the completion animation within the triggered area. Animation duration: ; In the formula, 0.3 is the baseline coefficient set for the animation duration, in seconds. The smaller W is, the more scarce the visual resources are, the longer the animation duration is, and the more eye-catching the feedback effect is achieved. The baseline coefficient can be adjusted to 0.2, 0.4, etc., to adapt to different users' perception needs for feedback duration.

[0026] Haptic feedback: Achieved through steering wheel vibration; vibration frequency: In the formula, 2 is a proportionality coefficient for the vibration frequency. The higher the task priority, the more the coefficient 2 amplifies the frequency increase, making the vibration more noticeable. This proportionality coefficient can be adjusted to 1.5, 2.5, etc., to match the performance limit of the steering wheel vibration module. Vibration duration: In the formula, 0.2 is the duration ratio coefficient: it is related to the visual attention weight W. The larger W is, the more abundant the visual resources are, and the longer the vibration duration is; 0.1 is the baseline value of the duration, in seconds: to ensure that even if W is 0, the vibration still has a basic duration; both can be adjusted, such as 0.15, 0.12, etc., to balance the promptness of the feedback and the interference of driving.

[0027] Taking a user switching music on a straight highway at 110km / h as an example, i.e., T=3, the specific implementation process is as follows: Step A.1: Real-time collection of driving scenario parameters: The basic driving scenario parameters are assumed to be S={110,0,8000}, which means a speed of 110km / h, a straight road, and a light intensity of 8000 lux; V={0,1}, which means the user's gaze is directly ahead and the gaze movement frequency is 1 time / second; T=3, which is a low-priority task. Step A.2, Calculation of visual attention weights: ; Step A.3: Dynamically adjust the position of the trigger area:

[0028] The center coordinates of the trigger area are (x0, y0) = (2.15cm, 0cm), which means it is located 2.15cm directly in front of the focal point of the line of sight. Step A.4: Adjusting the size of the trigger area and the response threshold: Size of the trigger area: ; Step A.5, Determining the validity of eye-tracking triggers: Assuming the user's gaze enters the trigger area, the actual time t of their gaze remaining within the trigger area... s =3.5 seconds, line-of-sight deviation |v1−v0|=2°, then the effectiveness index is:

[0029] That is, when E=0.802≥0.8, the eye movement behavior is considered valid; Since T=3, eye-tracking secondary focusing is then used for confirmation. Assume the user's second dwell time is t. s2 =2 seconds; And t s2 =2≥0.5×3.49≈1.745 seconds, then the pass is confirmed; Step A.6: Execution and Feedback of Interactive Tasks Perform the music switching operation and notify the user through visual and haptic feedback. A completion animation will be displayed within the trigger area, with the animation duration as follows: Seconds, simultaneously achieved through steering wheel vibration, vibration frequency: Hz, vibration duration Second.

[0030] This embodiment achieves a balance between safety and efficiency in driving interaction by constructing a dynamic mapping model between visual attention and eye-tracking trigger areas in driving scenarios, combined with a multimodal confirmation mechanism. On the one hand, dynamically adjusting the position, size, and response threshold of the trigger area based on parameters such as vehicle speed and road curvature prevents the user's gaze from deviating significantly from the road, adapting to complex scenarios such as highways and sharp curves. On the other hand, employing a multimodal confirmation method of "eye movement + voice / head movement / secondary eye movement" reduces the risk of accidental touches in single eye-tracking interaction and enhances the perceptuality of interaction through visual and tactile feedback. Taking a high-speed music switching scenario as an example, this solution achieves low-burden, high-efficiency human-computer interaction while ensuring driving safety, adapting to the operational needs of tasks with different priorities.

[0031] As a second embodiment of the present invention: In specific implementation, compared with Embodiment 1, the technical solution of this embodiment differs from that of Embodiment 1 only in that this embodiment further includes: The trigger area update module is used to update the trigger area in real time when the driving scenario parameters change; In this implementation, driving scenario parameters change, such as entering a curve from a straight road and increasing the speed from 60km / h to 100km / h; The specific process is as follows: Step 3.1, Threshold for Determining Scene Changes: pass: To determine whether the scene has changed, calculate the parameter change ΔS={Δs1,Δs2,Δs3}. When Δs1≥20km / h, the speed scenario changes; When Δs2 ≥ 0.05 rad / m, the road curvature scenario is determined to have changed; When Δs3≥500lux, it is determined that the lighting scene has changed; Among them, s 1，新 This refers to the vehicle speed collected at the current moment, i.e., the latest vehicle speed data, s 1，旧 This refers to the vehicle speed collected at the previous moment, i.e., the previously recorded vehicle speed data; s 2，新 This refers to the road curvature measured at the current moment, s 2，旧 This refers to the road curvature collected at the previous moment; s 3，新 This refers to the ambient light intensity collected at the current moment, s 3，旧 This refers to the ambient light intensity collected at the previous moment; Step 3.2, Update frequency of the trigger region: Depending on the type of scene change, i.e. through: Determine the update frequency F of the trigger region; In the formula, 20 is the preset judgment benchmark corresponding to the change in vehicle speed, i.e., 20 km / h; 0.05 is the preset judgment benchmark corresponding to the change in road curvature, i.e., 0.05 rad / m; and 500 is the preset judgment benchmark corresponding to the change in illumination, i.e., 500 lux. The more drastic the scene changes, such as a sharp bend at high speed, the larger the F value, and the more frequently the trigger area is updated. In this embodiment, the maximum is 1 time / second. F ≤ 0.1 times / second indicates that the scene is stable, that is, it is updated once every 10 seconds, thereby reducing system power consumption. Step 3.3, Smooth transition of the update process: To avoid triggering sudden changes in the region that could cause visual discomfort to users, a linear transition algorithm is used for updates, and the total transition time t is defined. t =0.5 seconds, t t This is a fixed value that determines how quickly the trigger area moves from the old position to the new position; pass: Calculate the center coordinates (x, y) of the trigger area during the transition process. t ,y t ); Among them, t p This refers to the elapsed time during the transition process, i.e., the time elapsed from the start of the transition from the triggering region to the current moment, in seconds, and 0 ≤ t. p ≤t t ; At the same time, the size A of the trigger area t and response threshold t t The same linear transition algorithm logic is used for updates to ensure the continuity of the interactive experience.

[0032] This embodiment adds a real-time update module for the trigger area. For dynamic changes in driving scenario parameters (such as sudden increases in vehicle speed or road curves), it accurately identifies scene transitions by setting a scene change judgment threshold and adjusts the update frequency of the trigger area according to the severity of the change. This ensures timely adaptation of the interactive area when the scene changes abruptly, while reducing the update frequency to save power when the scene is stable. Simultaneously, a linear transition algorithm is used to achieve smooth updates of the trigger area, avoiding visual discomfort to the user caused by sudden changes in the area and ensuring the continuity of the interactive experience. This solution further enhances the system's adaptability to dynamic driving environments, ensuring that the interactive area always matches the real-time scene, improving the safety and smoothness of interaction under complex road conditions.

[0033] As an embodiment of the present invention: In specific implementation, compared with Embodiment 1 and Embodiment 2, the technical solution of this embodiment is to combine the solutions of Embodiment 1 and Embodiment 2.

[0034] This embodiment combines the dynamic triggering region modeling and multimodal confirmation mechanism of Embodiment 1 with the scenario-based real-time update and smooth transition scheme of Embodiment 2, achieving full-scenario coverage of "static adaptation + dynamic adjustment". On the one hand, it retains the core advantages of attention weight calculation, triggering region adaptation, and multimodal confirmation in the basic scenario; on the other hand, it adds the ability to update in real time and smoothly transition after scenario changes, which not only adapts to the interaction needs of the initial driving scenario, but also dynamically responds to sudden changes in road conditions and vehicle speed, taking into account the safety, efficiency, and continuity of interaction. This combined solution covers the entire "stable-changing" scenario during driving, greatly improving the system's universality and user experience.

[0035] It should be stated that all user data collected in this application was collected with the user's consent and authorization, and the use of user data is legal and compliant, and the use and processing of user data comply with the relevant laws, regulations and standards of the relevant regions.

[0036] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0037] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0038] The above formulas are all dimensionless calculations. Dimensionless calculation involves introducing a reference benchmark, such as the maximum, minimum, standard deviation, or theoretical extreme value of a physical quantity, to transform the original physical quantity into a dimensionless relative value. This value is usually mapped to a specific interval, such as [0,1] or [-1,1], which eliminates the influence of units while preserving the relative size relationship of the physical quantities. The formula is derived from software simulation based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0040] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal human-computer intelligent interaction system based on eye tracking, characterized in that, include: The eye-tracking region modeling module is used to collect basic driving scenario parameters, including a set of vehicle driving parameters, a set of user visual feature parameters, and interaction task type parameters. The vehicle driving parameter set includes the current vehicle speed, road curvature, and ambient light intensity; the user visual feature parameter set includes the angle between the user's gaze focus and the road in front of the vehicle, and the frequency of the user's gaze movement within a specified unit of time; the interaction task type parameter indicates the priority of the interaction task to be executed. Then, based on the basic driving scenario parameters, the visual attention weight is determined; then, based on the visual attention weight and the user's visual feature parameter set, the center coordinates of the trigger area are determined, and then the area of ​​the trigger area and the response threshold are determined. The interaction trigger execution module first determines the validity of the eye movement behavior when the user's gaze enters the trigger area after dynamic modeling, and then matches the corresponding preset confirmation modality according to the interaction task type parameter. The trigger area update module is used to update the trigger area in real time when driving scenario parameters change.

2. The multimodal human-computer intelligent interaction system based on eye tracking according to claim 1, characterized in that: The visual attention weights are calculated as follows: First, the vehicle speed, road curvature, and user eye movement frequency are normalized separately. Then, they are weighted and summed according to their respective weight proportions to obtain the visual attention weight value, which is used to represent the proportion of visual resources that users allocate to the interactive interface. Among them, when normalizing driving speed, 120km / h is the upper limit for high speed, with a weight ratio of 0.6; when normalizing road curvature, 0.1rad / m is the upper limit for sharp curvature, with a weight ratio of 0.3; when normalizing line-of-sight movement frequency, 5 times / second is the upper limit for high-frequency movement, with a weight ratio of 0.

1.

3. The multimodal human-computer intelligent interaction system based on eye tracking according to claim 2, characterized in that: The center coordinates, area, and response threshold of the trigger region are determined as follows; The center coordinates of the trigger area are determined by using the projection coordinates of the user's current gaze focus on the interactive screen as a reference, combined with the visual attention weight value to determine the distance from the gaze focus, and then determining the direction of the trigger area relative to the gaze focus based on the cosine and sine values ​​corresponding to the angle between the user's gaze focus and the road in front of the vehicle, and finally calculating the center coordinates; where the center of the screen is the origin; the distance to the gaze focus is the visual attention weight value multiplied by 5; the direction of the trigger area relative to the gaze focus is consistent with the user's line of sight; The size of the trigger area is calculated by multiplying the visual attention weight value by 2 and then adding 1. The response threshold is calculated by dividing 1.5 by the visual attention weight value.

4. The multimodal human-computer intelligent interaction system based on eye tracking according to claim 3, characterized in that: The validity of eye-tracking behavior is determined as follows: pass To determine the validity of the eye movement behavior; In the formula, E is the effectiveness index, and t s This refers to the actual time the user's gaze lingers within the trigger area; t refers to the response threshold in the current scenario; v1 refers to the angle between the user's line of sight and the road in front of the vehicle. When the user's line of sight is directly in front, v1=0; v0 is the line of sight direction corresponding to the trigger area. Refers to the normalized value of the deviation in line of sight; When E≥0.8, the eye-tracking behavior is considered valid; otherwise, it is considered a false touch and no operation is triggered.

5. A multimodal human-computer intelligent interaction system based on eye tracking according to claim 4, characterized in that: The interactive task type parameter takes values ​​of {1,2,3}, where 1 represents high priority, 2 represents medium priority, and 3 represents low priority.

6. A multimodal human-computer intelligent interaction system based on eye tracking according to claim 5, characterized in that: The matching method for modal confirmation is as follows: For high-priority interactive tasks: head movement confirmation is used, and the confirmation conditions are that the positive value of the left and right deflection angle is less than or equal to 5° and the positive value of the up and down deflection angle is less than or equal to 3°; that is, if it is a nod or a shake of the head, the confirmation is completed. For medium-priority interactive tasks, short voice confirmation is used. The confirmation conditions are that the voice input duration is between 0.3 and 0.8 seconds and the content is "yes" to execute, and "no" to cancel. For low-priority interactive tasks, eye-tracking secondary focusing confirmation is used. The confirmation condition is that the time the gaze stays in the trigger area twice is greater than or equal to half of the response threshold in the current scene.

7. A multimodal human-computer intelligent interaction system based on eye tracking according to claim 6, characterized in that: In the modal confirmation matching method, for medium-priority interactive tasks, no response will be given if the confirmation conditions exceed the time limit or the content does not match.

8. A multimodal human-computer intelligent interaction system based on eye tracking according to claim 6, characterized in that: After the interaction triggering and execution module completes the effective triggering of eye movement behavior and modal confirmation, it executes the corresponding interaction task and informs the user through visual feedback and tactile feedback; visual feedback refers to displaying the completed animation in the trigger area; Haptic feedback is achieved through steering wheel vibration.

9. A multimodal human-computer intelligent interaction system based on eye tracking according to claim 1, characterized in that: The process for triggering a region update is as follows: First, determine the parameter changes that determine whether the driving scenario has changed. The parameter changes that indicate a change in the scenario include changes in vehicle speed, road curvature, and ambient light intensity. These are the absolute values ​​of the differences between the vehicle speed, road curvature, and ambient light intensity collected at the current moment and the vehicle speed, road curvature, and ambient light intensity collected at the previous moment, respectively. Next, divide the vehicle speed change by 20, the road curvature change by 0.05, and the illumination change by 500 respectively. Then add these three ratios together and finally divide 1 by the sum to obtain the update frequency of the trigger area. Here, 20 is the preset judgment criterion for vehicle speed change; 0.05 is the preset judgment criterion for road curvature change; and 500 is the preset judgment criterion for illumination change. Then, a linear transition algorithm was used to determine the center coordinates of the trigger area during the transition process; In this transition process, the center x-coordinate of the trigger area is equal to the old center x-coordinate plus the difference between the new and old center x-coordinates, multiplied by the ratio of the transition time already consumed to the total transition time; the center x-coordinate of the trigger area during the transition process is equal to the old center y-coordinate plus the difference between the new and old center y-coordinates, multiplied by the ratio of the transition time already consumed to the total transition time; and the total transition time is a fixed value. Meanwhile, the size of the trigger area and the response threshold are updated using the same linear transition algorithm logic.

10. A multimodal human-computer intelligent interaction system based on eye tracking according to claim 9, characterized in that: The method for determining whether the scene has changed is as follows: When the change in vehicle speed is greater than or equal to 20 km / h, the speed scenario is determined to have changed. When the change in road curvature is greater than or equal to 0.05 rad / m, it is determined that the road curvature scenario has changed. When the change in ambient light intensity is greater than or equal to 500 lux, the lighting scene is considered to have changed.

Images