A method for situational awareness interaction based on eye movement information in an airborne cabin environment

Abstract

The application relates to the field of human-computer interaction, in particular to a situation perception interaction method based on eye movement information in an airborne cabin environment. The method comprises the following steps: monitoring user eye movement information in real time; when it is monitored that current user eye movement information triggers a situation perception condition, predicting a next interaction action of the user according to historical user eye movement information and corresponding historical actions; when the next action meets a key action definition, giving a prompt of whether the user executes the task; when the next action does not meet the key action definition, executing the next interaction action.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, and in particular to a context-aware interaction method based on eye-tracking information in an airborne cockpit environment. Background Technology

[0002] As a complex human-machine interface, the airborne cockpit places high demands on the skills and abilities of pilots. To alleviate pilot workload and improve human-machine interaction efficiency, perceiving and predicting user interaction intentions and actions in the airborne cockpit environment is of great significance for achieving intelligent human-machine collaborative interaction.

[0003] In airborne cockpit environments, human-computer interaction has traditionally relied on single-channel or multi-channel fusion methods such as physical buttons, touch, voice, and gestures. However, eye-tracking information has not been fully utilized in natural airborne interaction, significantly limiting the efficiency of cockpit interaction and the improvement of the user experience. Compared to traditional interaction methods (such as touch, voice, and gestures), eye-tracking information can quickly provide the gaze area or target range, thus rapidly reflecting the user's intent. However, existing interaction methods based on explicit eye-tracking (such as gaze, eye gestures, and blinking) are prone to causing visual fatigue and significantly increasing cognitive load. Furthermore, errors caused by natural eye tremors and perceptual mechanisms make explicit eye-tracking interaction difficult to support fine-grained interactive behaviors, thus hindering its effective application in environments like airborne cockpits where interaction space is limited, interactive elements are concentrated, and safety requirements are extremely high.

[0004] To effectively improve interaction efficiency and ensure safety in the airborne cockpit environment, the system can utilize eye-tracking information detected during natural interaction to infer and predict the user's interaction intent and next action, and then assist the user in completing the interaction task based on the prediction results. This effectively leverages the real-time advantages of eye-tracking interaction technology while effectively reducing visual fatigue and cognitive load during explicit interaction. Furthermore, through human-computer collaborative interaction design, the system can effectively ensure the operational safety of interaction tasks.

[0005] Chinese patent application CN202010827022.0 discloses an eye-tracking interaction method, system, and device based on eye-tracking technology. This method uses user eye movement behavior data to train a machine learning algorithm, filters, processes, and analyzes the data to obtain eye movement behavior patterns, and establishes a model of the user's subjective eye-tracking interaction intention. However, this method relies solely on eye movement information to determine user intention, which introduces a degree of noise interference. Furthermore, it does not utilize the correlation information between the user's historical actions, and therefore cannot update the model based on the user's latest action feedback during action prediction. It also does not involve human-computer collaborative interaction, posing a significant risk of misoperation when applied in an airborne cockpit environment.

[0006] Chinese patent application CN201910094616.2 discloses a multimodal deep fusion airborne cockpit human-computer interaction method. This method establishes a joint semantic expression of voice and touch commands based on user intent, performs multimodal fusion using various cockpit modal characteristics, automatically analyzes external sensor data to generate corresponding control commands, and then adaptively adjusts multimodal parameters based on user commands and sensor data, finally feeding back the generated multimodal control commands to the user for confirmation. This invention uses a voice and touch fusion approach for human-computer interaction, which lacks advantages in real-time performance and visual attention representation capabilities compared to eye-tracking interaction methods. Furthermore, this invention focuses on generating interaction commands under multimodal fusion conditions, which differs from the problem addressed by this invention.

[0007] Chinese patent application CN201810869753.4 discloses a high-efficiency interactive system based on a closed cockpit, comprising a closed cockpit with cameras mounted on its outer sides in the front, rear, left, and right directions, and rear projectors mounted on its inner sides in the front, rear, left, and right directions. Each rear projector is connected to a camera in the same direction via a data transmission line. A fixed rotating seat is mounted at the center of the cockpit, with pressure sensor arrays mounted on the backrest and seat cushion. A touchscreen is positioned in front, and a gesture recognition device is located below the touchscreen. Gyroscopes are mounted on the user's head and torso. This invention focuses on the generation of interactive commands under the condition of gesture and touch fusion, which is different from the problem solved by this invention.

[0008] Existing Chinese patents do not specify how to effectively utilize eye-tracking information, especially implicit eye-tracking information, in an airborne cockpit environment to achieve context-aware interaction. How to better improve interaction efficiency while ensuring operational safety during interaction tasks remains an unsolved problem in the field of human-computer interaction in airborne cockpit environments. Summary of the Invention

[0009] Purpose of the invention: This invention provides a context-aware interaction method based on eye-tracking information in an airborne cockpit environment, which can improve interaction efficiency and reduce user visual fatigue and cognitive load by leveraging the real-time advantage of eye-tracking information; at the same time, it ensures the operational safety of interaction tasks through human-computer collaborative interaction design.

[0010] Technical solution:

[0011] A context-aware interaction method based on eye-tracking information in an airborne cockpit environment includes:

[0012] Real-time monitoring of user eye movement information;

[0013] When the current user eye movement information is detected and the context-aware condition is triggered, the user's next interaction action is predicted based on the historical user eye movement information and corresponding historical actions.

[0014] If the next action meets the definition of a key action, prompt the user whether to perform the task.

[0015] If the next action does not meet the definition of the key action, execute the next interactive action.

[0016] The method of predicting the user's next interaction action based on historical user eye movement information and corresponding historical actions includes:

[0017] The current user eye movement information is input into the context-aware model to predict the user's next interaction action; the context-aware model is trained using machine learning with historical actions and historical user eye movement information as samples.

[0018] The method of predicting the user's next interaction action based on historical user eye movement information and corresponding historical actions includes:

[0019] Using probabilistic statistical methods, based on the correspondence between historical actions and historical user eye movement information, the action with the highest probability corresponding to the current user's eye movement information is analyzed from historical actions, and the action with the highest probability is used as the predicted next interaction action of the user.

[0020] Probability and statistical methods employ joint probability distributions and multivariate conditional probabilities.

[0021] Using probabilistic statistical methods, based on the correspondence between historical actions and historical user eye-tracking information, the action with the highest probability corresponding to the current user's eye-tracking information is analyzed from historical actions, including:

[0022] Obtain the state transition matrix, which records the joint probability distribution of w consecutive action combinations in the history. There are n possible actions, and each action is allowed to be repeated; w is an integer greater than 2.

[0023] Obtain the emission matrix, which records the joint distribution probability of each historical action with different types of historical eye-tracking information;

[0024] Obtain the current eye movement information and the corresponding probabilities for all next actions from the emission matrix;

[0025] Obtain the w-1 consecutive actions closest to the current prediction; obtain all combinations and probabilities of the previous w-1 actions being the aforementioned w-1 consecutive actions from the state transition matrix; obtain the probabilities of each action from the state transition matrix;

[0026] Based on the above information about the current eye movement, we determine the probability of all possible next actions and their probabilities, all combinations and their probabilities, and the probability of each action. We then determine the probability of the current user's eye movement information corresponding to the nth action (wth action) based on the known previous (w-1) actions. Finally, we select the action with the highest probability as the predicted next action.

[0027] The method further includes:

[0028] Receive user input commands;

[0029] If the user instruction indicates that the predicted action is not executed or is canceled, and other actions are executed instead, then the execution of the predicted action is stopped, other actions are executed, and the probabilities related to other actions in the emission matrix and state transition matrix are corrected. Other actions are any action other than the predicted action among n actions.

[0030] The probabilities related to other actions in the emission matrix and state transition matrix are corrected, including:

[0031] In the state transition matrix, a correction coefficient is added to increase the probability of the current combination of consecutive w actions. The combination of consecutive w actions includes the previous w-1 actions being the consecutive w-1 actions and the w-th action being the other actions. All combinations that are different from the current combination of consecutive w actions by only one action are queried. The correction coefficient is evenly distributed to each combination that is different by only one action to obtain the average score. The original probability of each combination that is different by only one action is subtracted from the average score to obtain the state transition matrix to be used next.

[0032] In the emission matrix, a correction coefficient is added to increase the probability of the current prediction combination, which includes the current user eye movement information and the other actions. The correction coefficient is then evenly distributed among each combination containing the current user eye movement information or the other actions to obtain an average value. The average value is then subtracted from the original probability of each combination containing the current user eye movement information or the other actions to obtain the emission matrix to be used next.

[0033] User eye movement information is obtained in real time using an eye tracker or other eye-tracking devices during the interaction process. This includes, but is not limited to, the two-dimensional or three-dimensional coordinates of the gaze point, the gaze area, the gaze trajectory, eye movement behavior, the effective range, and the pupil diameter. Eye movement behavior includes, but is not limited to, saccades, staring, and blinking.

[0034] The context-aware condition is that the change in the gaze point and gaze area between the current and previous moments is greater than a preset range.

[0035] A computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the method.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] 1) The context-aware model in this invention considers the user's historical actions and current eye movement information when using machine learning or probability and statistics methods. Specifically, it considers the relationship between any three actions of the user, thus obtaining a more flexible relational data matrix and effectively ensuring that the model can infer the user's interaction intent in action scenarios with different relational types.

[0038] 2) The context-aware model in this invention uses the user's implicit eye movement information. During the interaction, the user does not need to perform any predetermined eye movement operation. This not only effectively utilizes the real-time characteristics of eye movement interaction, but also reduces the visual fatigue and cognitive load of the user during the interaction process.

[0039] 3) The human-machine collaborative interaction design in this invention introduces a key action discrimination process. That is, in the airborne cockpit environment, interactive actions with low safety requirements and risk levels are executed by the system, while interactive actions with high safety requirements and risk levels are decided and executed by the user, thereby realizing human-machine collaborative interaction and ensuring the operational safety of interactive tasks. Attached Figure Description

[0040] Figure 1 Flowchart for an implementation where the predicted action is not a critical action.

[0041] Figure 2 Flowchart for an implementation example where the predicted action is a key action.

[0042] Figure 3a This is a schematic diagram showing the detection of an eye-tracking gaze range within the "stacked target" region during an embodiment operation where the predicted action is not a critical action (partial magnification).

[0043] Figure 3b This is a schematic diagram illustrating how, in an example where the predicted action is not a critical action (local zoom-in), the context-aware model determines the user's interaction intent as "locking the target" and predicts the user's next action as "local zoom-in."

[0044] Figure 3c This is a schematic diagram illustrating how the system automatically "magnifies" the gaze area by 2 times during an embodiment where the predicted action is not a critical action (local magnification).

[0045] Figure 4a This is a schematic diagram illustrating the detection of an eye-tracking gaze range within the "magnified target" region during an embodiment of the operation where the predicted action is a key action (target lock).

[0046] Figure 4bThis is a schematic diagram illustrating how, in an example of predicting a key action (target lock), the context-aware model determines the user's interaction intent as "target lock" and predicts the user's next action as "target lock A".

[0047] Figure 4c This is a schematic diagram illustrating the system's "target locked" prompt during an implementation where the predicted action is a critical action (target lock). Detailed Implementation

[0048] This invention provides a context-aware interaction method based on eye-tracking information in an airborne cockpit environment. Figure 1-2 As shown, it includes:

[0049] 1) The eye tracker records the user's eye movement information in real time;

[0050] 2) After detecting the user's eye movement behavior, the context-aware condition is triggered. The context-aware model first infers the user's interaction intention and then predicts the user's next interaction action.

[0051] 3) The system determines whether the predicted action conforms to the definition of a critical action;

[0052] 4) If the predicted action is a critical action, the system will provide relevant prompts, and the user can decide whether to execute it;

[0053] 5) If the predicted action is not a critical action, the system executes the predicted action, and the user judges whether the execution result meets expectations; if it does not meet expectations, the user cancels the execution of the action; otherwise, the system enters the next round of interaction.

[0054] 6) Update the prediction context-aware model based on the user's judgment of the predicted action.

[0055] The context-aware model is obtained through the following steps:

[0056] a) Obtain user's historical actions and eye movement information;

[0057] b) Use machine learning or probability and statistics methods to learn the relationship parameters between historical actions and historical eye movement information;

[0058] c) Using the user's historical actions and current eye movement information as input, calculate and output the most likely action as the next interaction action;

[0059] d) After the user makes a judgment on the prediction result, update the relational parameters in b).

[0060] Furthermore, if there is a strong regularity between actions, machine learning algorithms such as LSTM or HMM can be given priority.

[0061] Furthermore, the probability and statistics related knowledge includes: approximate calculation of joint probability distribution and multivariate conditional probability.

[0062] Preferably, the context-aware prediction condition setting refers to determining the distribution area of ​​the user's eye movement gaze points within a certain period of time by recording eye movement trajectories. If a change in the distribution area of ​​the user's eye movement gaze points is detected, the behavior prediction condition is triggered.

[0063] Preferably, in the context-aware user behavior prediction and interaction method, the key action judgment refers to the system judging whether the predicted action belongs to a pre-set set of key actions. The set of key actions refers to the set of pilot interaction actions divided and classified according to safety requirements and risk levels in the airborne cockpit environment.

[0064] Preferably, in the context-aware user behavior prediction and interaction method, the airborne cockpit environment includes an airborne touchscreen display, an eye tracker, an aircraft joystick, a memory, a processor, and other interactive devices. The memory stores a computer program, and the processor is configured to run the computer to execute the method described above.

[0065] Preferably, in the context-aware user behavior prediction and interaction method, the user eye movement recording refers to acquiring the user's eye movement information in real time during the interaction process using an eye tracker or other eye tracking device.

[0066] This invention discloses a context-awareness and interaction method based on eye-tracking information in an airborne cockpit environment. First, a context-awareness model is constructed using probabilistic algorithms. Then, taking the "target locking" task during maneuvering as an example, the interaction method of this invention is specifically explained. It should be noted that the described embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0067] In this invention, the combination of w consecutive actions includes three actions, with the intentional action x at time i+1. i+1 The intended action at time i is x. i The intended action at time i-1 is x. i-1 Intentional action x i and x i-1 For all historical actions, the intentional action x i+1 The predicted next move.

[0068] The construction method of the context-aware model is as follows:

[0069] 1) Let the pilot's eye movement information at time i+1 be e. i+1 The intended action at time i is x. i The intended action at time i-1 is x. i-1Let i = 1, 2, ..., n, representing the interactive action under a certain interactive intention condition. For example, x i The intention action representing "locking onto the target and zooming in locally" physically means "the user's interaction intention is to lock onto the target, and the interaction action is to zoom in locally." The model predicts the optimal intention action x′ at time i|1. i+1 , such that the conditional probability P(x) i+1 |e i+1 x i x i-1 (To obtain the maximum value)

[0070] 2) Let the state transition matrix be T. (n×n×n) It records historical actions x i x i-1 With action x i+1 The joint probability distribution; its element value t i-1，i，+1 =P(x i-1 x i x i+1 ) represents action x i-1 x i x i+1 The probability of being executed consecutively (regardless of the order) is:

[0071] 3) Let the emission matrix be S (n×n) It records the action x at time i. i With eye-tracking information e i The joint probability distribution, whose element value u i =P(e i x i ) indicates that the user performed the action x. i Eye movement information is e i The probability is ∑ e P(e i x i )=∑ x P(e i x i =1; Let the empirical vector be E = {P(x1), P(x2), ..., P(x...}}. n It records the probability of each action x occurring during the interaction, and has

[0072] 4) Assume that the eye movement information at time i+1 is g i+1 Action x at time i i Action x at time i-1 i-1 Independent. According to the law of total probability, that is...

[0073]

[0074] Equivalent to

[0075]

[0076] Eye tracking information at time i+1 e i+1 Only with predicted action x i+1 Related to user actions in the first two time periods x i x i-1 Irrelevant, that is

[0077]

[0078] For P(e i+1 |x i+1 Using the law of total probability, we can find P(x) i+1 |e i+1 x i x i-1 ) can be represented as

[0079]

[0080] Eye tracking information at time i+1 e i+1 User actions at moment i i User action x at time i-1 i-1 All information is known; for each time step i+1, different operations x need to be predicted. i+1 , P(g i+1 )P(x i x i-1 If ) is a fixed term, then:

[0081]

[0082] The probability P(x) is stored in a three-dimensional transition matrix T. i+1 x i x i-1 The two-dimensional emission matrix S stores the probability P(g) i+1 x i+1 ), probability P(x i+1 It can be obtained by adding the corresponding columns of the emission matrix.

[0083] 5) Update the prediction model based on user feedback on the predicted action, defining a revision coefficient δ, where δ ranges from (0, 0.1], and the action at time i-1 is x. o The action at time i is x p The i+1 eye-tracking information is e i+1 If the predicted action for i+1 is x q The user's actual action is x. r The state transition matrix T is transformed as follows, where t oprThe action at time i-1 before the feedback update is x. o The action at time i is x p The action at time i+1 is x r The probability of t; opr The action at time i-1 after the feedback update is x. o The action at time i is x p The action at time i+1 is x r The revised probability, which needs to be "strengthened" by increasing its value, is contributed equally by the remaining elements associated with it:

[0084] t opr =t opr +δ

[0085]

[0086]

[0087]

[0088] Among them, t tpr , t otr , t opr For t opr All other probability values ​​associated; t tpr This refers to the feedback before the update, excluding action x. o In addition, the remaining n-1 actions are related to action x. p Action x r The joint probability distribution; t otr This refers to the feedback before the update, excluding action x. p Apart from that, all other actions and action x o Action x r The joint probability distribution; t opt This refers to the feedback before the update, excluding action x. r Apart from that, all other actions and action x o Action x p The joint probability distribution formed by t. tpt * , t otr * , t opt * To complement the strengthening item t opr * All other probability values ​​after amortization of the association. t tpr * This refers to the feedback after the update, excluding action x of T. o In addition, the remaining n-1 actions are related to action x. p Action x rThe new joint distribution probability formed; t otr * This refers to the feedback after the update, excluding action x of T. p Apart from that, all other actions and action x o Action x r The new joint distribution probability formed; t opt * This refers to the feedback after the update, excluding action x of T. r Apart from that, all other actions and action x o Action x p The resulting joint probability distribution;

[0089] The emission matrix S is transformed as follows, s t+1，r It is the intention action x corresponding to the (i+1)th eye movement information of S before the feedback update. r The probability of s; i+1，r * To provide feedback on the intention action x corresponding to the (i+1)th eye movement information of S after the update. r The probability needs to be "enhanced" by increasing its value, with the change contributed equally by the remaining elements associated with it:

[0090] s i+1，r * =s i+1，r +δ

[0091]

[0092]

[0093] Among them, s t，r The feedback before the update, S, includes eye movement signals e. i+1 Apart from that, all other eye movement signals and actions x r The joint probability distribution; s i+1，t The feedback before the update is S, excluding action x. r Apart from that, all other movements and eye movement signals e i+1 The joint probability distribution;

[0094] s t，r * In addition to eye movement signals e i+1 Apart from that, all other eye movement signals and actions x r The joint probability distribution; s i+1，t * In addition to action x r Apart from that, all other movements and eye movement signals e i+1 The joint probability distribution;

[0095] If the predicted action is the same as the actual action, the emission matrix remains unchanged.

[0096] like Figures 3a-3c , Figures 4a-4c As shown, an example of an interaction method constructed for performing a "target locking" task during a maneuver is as follows:

[0097] 1) In the cockpit, the pilot controls the aircraft via a display screen and joystick, while an eye tracker records the user's eye movement information in real time. When the system detects that the pilot's eye gaze is within the "stacked target" area, the user's behavior triggers the context-aware condition, and the context-aware model begins to infer the user's interaction intention and the next interaction action;

[0098] 2) If the inference result is "target locked, local zoom-in", that is, in all actions x i+1 In (i = 1, 2, ..., n), the action representing "locking onto the target and zooming in locally" is... Enabling multivariate conditional probability In all P(x) i+1 e i+1 x i x i-1 The value in is the maximum value, where i = 1, 2, ..., n. According to the definition of a critical action, if the action is not critical (e.g., the action has low safety requirements), it can be executed automatically by the system. The system automatically executes the action, "enlarging" the target display area by 2 times; if the predicted action does not meet the user's expectations, the user cancels the execution result of the predicted action, restoring the display ratio of the target display area.

[0099] 3) If the prediction model infers that the user's next action is "lock target, lock target A", that is, in all user actions x i+1 In (i = 1, 2, ..., n), the action "Lock on target, lock on target A" is represented. Enabling multivariate conditional probability In all P(x) i+1 |e i+1 x i x i-1 The maximum value is represented in the range (i = 1, 2, ..., n). According to the definition of a critical action, if the action is critical (e.g., the action has high safety requirements), the pilot must decide whether to execute it. The system provides relevant prompts on the display screen, such as highlighting and flashing "Target A" to remind the pilot to lock onto it.

[0100] 4) In the embodiment where the prediction action is "target lock, local zoom-in", if the user's expectation is the same as the prediction, the emission matrix remains unchanged; otherwise, the user rejects the "target lock, local zoom-in" action. Actual action xr Therefore, the state transition matrix T needs to be transformed as follows:

[0101] t opr *-t opr +δ

[0102]

[0103]

[0104]

[0105] The emission matrix S is transformed as follows

[0106] s i+1，r * =s i+1，r +δ

[0107]

[0108]

[0109] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention should be determined by the claims.

Claims

1. A context-aware interaction method based on eye-tracking information in an airborne cockpit environment, characterized in that, include: Real-time monitoring of user eye movement information; When the current user eye movement information triggers a context-aware condition, the system predicts the user's next interaction action based on historical user eye movement information and corresponding historical actions. This includes: inputting the current user eye movement information into the context-aware model to predict the user's next interaction action; the context-aware model is trained using machine learning with historical actions and historical user eye movement information as samples; and using probabilistic statistical methods, based on the correspondence between historical actions and historical user eye movement information, analyzing historical actions to identify the action with the highest probability corresponding to the current user eye movement information, and using this action as the predicted next interaction action. Specifically, this involves obtaining a state transition matrix, which records the joint probability distribution of w consecutive action combinations in the historical actions. There are n possible actions, and each action is allowed to be repeated; w is greater than... An integer of 2; obtain the emission matrix, which records the joint distribution probability of each historical action and different types of historical eye movement information; obtain all next actions and probabilities corresponding to the current eye movement information from the emission matrix; obtain the nearest w-1 consecutive actions to the current prediction; obtain all combinations and probabilities of the previous w-1 actions being the aforementioned consecutive w-1 actions from the state transition matrix; obtain the probabilities of various actions from the state transition matrix; based on the above-mentioned all next actions and probabilities corresponding to the current eye movement information, all combinations and probabilities, and probabilities of various actions, determine the probability of the previous w-1 actions being known and the current user eye movement information corresponding to the nth type of w-th action; select the action with the highest probability as the predicted next action; wherein, the context-aware condition is that the change in the fixation point and fixation area between the current and previous moments is greater than a preset range; If the next action meets the definition of a key action, prompt the user whether to perform the action; If the next action does not conform to the definition of the critical action, execute the next interactive action. Receive user input commands; If the user instruction indicates that the predicted action is not executed or is canceled, and other actions are executed instead, then the execution of the predicted action is stopped, other actions are executed, and the probabilities related to other actions in the emission matrix and state transition matrix are corrected. Other actions are any action other than the predicted action among n actions.

2. The method according to claim 1, characterized in that, The construction method of the context-aware model is as follows: 1) Let the pilot's eye movement information at time i+1 be e. i+1 The intended action at time i is x. i The intended action at time i-1 is x. i-1 Let i = 1, 2, ..., n, representing the interaction action under a certain interaction intention condition; the model predicts the optimal intention action x' at time i+1. i+1 , such that the conditional probability P(x) i+1 |e i+1 x i x i-1 (To obtain the maximum value) 2) Let the state transition matrix be T. (n×n×n) It records historical actions x i x i-1 With action x i+1 The joint probability distribution; its element value t i-1，i，i+1 =P(x i-1 x i x i+1 ) represents action x i-1 x i x i+1 The probability of being executed consecutively is: 3) Let the emission matrix be S (n×n) It records the action x at time i. i With eye-tracking information e i The joint probability distribution, whose element value u i =P(e i ,x i ) indicates that the user performed the action x. i Eye movement information is e i The probability is ∑ e P(e i ,x i )=∑ x P(e i ,x i =1; Let the empirical vector be E = {P(x1), P(x2), ..., P(x...}} n It records the probability of each action x occurring during the interaction, and has 4) Assume that the eye movement information at time i+1 is e i+1 Action x at time i i Action x at time i-1 i-1 Independence: According to the law of total probability, that is Equivalent to Eye movement information at time i-1 e i+1 Only with predicted action x i+1 Related to user actions x in the first two time periods i ,x i-1 Irrelevant, that is For P(e i+1 |x i+1 Using the law of total probability, we can find p(x) i+1 |e i+1 ,x i ,x i-1 ) can be represented as Eye tracking information at time i+1 e i+1 User actions at moment i i User action x at time i-1 i-1 All information is known; for each time step i+1, different operations x need to be predicted. i+1 , P(e i+1 )P(x i x i-1 If ) is a fixed term, then: The probability P(x) is stored in a three-dimensional transition matrix T. i+1 x i x i-1 The two-dimensional emission matrix S stores the probability P(θ). i+1 x i+1 ), probability P(x i+1 It can be obtained by adding the corresponding columns of the emission matrix; 5) Update the prediction model based on user feedback on the predicted action, defining a revision coefficient δ, where δ ranges from (0, 0.1], and the action at time i-1 is x. o The action at time i is x p The i+1 eye-tracking information is e i+1 If the predicted action for i+1 is x q The user's actual action is x. r The state transition matrix T is transformed as follows, where t opr The action at time i-1 before the feedback update is x. o The action at time i is x p The action at time i+1 is x r The probability of t; opr * The action at time i-1 after the feedback update is x. o The action at time i is x p The action at time i+1 is x r The revised probability, which needs to be "strengthened" by increasing its value, is contributed equally by the remaining elements associated with it: t tpr * =t tpr +δ Among them, t tpr , t otr , t opr For t opr All other probability values ​​associated; t tpr This refers to the feedback before the update, excluding action x. o In addition, the remaining n-1 actions are related to action x. p Action x r The joint probability distribution; t otr This refers to the feedback before the update, excluding action x. p Apart from that, all other actions and action x o Action x r The joint probability distribution; t opt This refers to the feedback before the update, excluding action x. r Apart from that, all other actions and action x o Action x p The joint probability distribution; t tpr * , t otr * , t opt * To complement the strengthening item t opr * All other probability values ​​after amortization of the association; t tpr * This refers to the feedback after the update, excluding action x of T. o In addition, the remaining n-1 actions are related to action x. p Action x r The new joint distribution probability formed; t otr * This refers to the feedback after the update, excluding action x of T. p Apart from that, all other actions and action x o Action x r The new joint distribution probability formed; t opt * This refers to the feedback after the update, excluding action x of T. r Apart from that, all other actions and action x o Action x p The resulting joint probability distribution; The emission matrix S is transformed as follows, s i+1，r It is the intention action x corresponding to the (i+1)th eye movement information of S before the feedback update. r The probability of s; i+1，r To provide feedback on the intention action x corresponding to the (i+1)th eye movement information of S after the update. r The probability needs to be "enhanced" by increasing its value, with the change contributed equally by the remaining elements associated with it: s t+1，r * =s i+1，r +d Among them, s t，r The feedback before the update, S, includes eye movement signals e. i+1 Apart from that, all other eye movement signals and actions x r The joint probability distribution; s i+1，t The feedback before the update is S, excluding action x. r Apart from that, all other movements are related to eye movement signals θ i+1 The joint probability distribution; s t，r * In addition to the action eye movement signal θ i+1 Apart from that, all other eye movement signals and actions x r The joint probability distribution; s i+1，t * In addition to action x r Apart from that, all other movements and eye movement signals e i+1 The joint probability distribution; If the predicted action is the same as the actual action, the emission matrix remains unchanged.

3. The method according to claim 1, characterized in that, The probabilities related to other actions in the emission matrix and state transition matrix are corrected, including: In the state transition matrix, a correction coefficient is added to increase the probability of the current combination of consecutive w actions. The combination of consecutive w actions includes the previous w-1 actions being the consecutive w-1 actions and the w-th action being the other actions. All combinations that are different from the current combination of consecutive w actions by only one action are queried. The correction coefficient is evenly distributed to each combination that is different by only one action to obtain the average score. The original probability of each combination that is different by only one action is subtracted from the average score to obtain the state transition matrix to be used next. In the emission matrix, a correction coefficient is added to increase the probability of the current prediction combination, which includes the current user eye movement information and the other actions. The correction coefficient is then evenly distributed among each combination containing the current user eye movement information or the other actions to obtain an average value. The average value is then subtracted from the original probability of each combination containing the current user eye movement information or the other actions to obtain the emission matrix to be used next.

4. The method according to claim 1, characterized in that, User eye movement information is obtained in real time using an eye tracker or other eye-tracking devices during the interaction process. This includes, but is not limited to, the two-dimensional or three-dimensional coordinates of the gaze point, the gaze area, the gaze trajectory, eye movement behavior, the effective range, and the pupil diameter. Eye movement behavior includes, but is not limited to, saccades, staring, and blinking.

5. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the method described in claims 1-4.

Images