Multi-mode user intention recognition method and system based on eye movement tracking

Abstract

The invention provides a multi-mode user intention identification method and system based on eye movement tracking, the method identifies a gazing object based on an eye tracker, and combines modes such as text and image to estimate the user intention to execute a specific task, the eye movement tracking is that pupil positions are positioned through an event camera to obtain coordinates, and the gazing point of eyes is calculated; according to the method, the user intention is identified and estimated, then the gaze information graph is constructed through picture element segmentation, the user intention is identified and estimated, the robustness and accuracy of user intention identification are enhanced, the system can more intelligently adapt to complex requirements of different users in different scenes, and therefore more personalized service experience is provided.

Description

Technical Field

[0001] The present invention relates to a method for identifying user intention, and in particular to a multimodal method and system for identifying user intention based on eye tracking. Background Art

[0002] Gaze tracking and recognition technology determines the position of human eyes by analyzing and processing eye movement data. This technology involves the processing method of eye movement data. Researchers have proposed various algorithm models to identify the gaze point position in eye movement data. These models include machine learning-based methods such as support vector machines, random forests, and neural networks, as well as deep learning-based methods such as convolutional neural networks and recurrent neural networks. These models improve the accuracy and stability of gaze point recognition through training and learning.

[0003] Gaze point recognition technology has been applied in many fields, including user interface design, medical research, driver research, market research, and educational research. It has dozens of applications in augmented reality and virtual reality (AR / VR), such as concave rendering or physiologically accurate rendering, interactive programs that respond to eye movements by allowing users to select targets with their eyes, etc. These applications and others (such as laser eye surgery) benefit from fast and accurate tracking of eyes that basically fill the sensor's field of view (i.e., "near eye" tracking). At the same time, in the field of advertising and marketing, this technology helps to evaluate consumers' attention allocation to specific advertising elements (such as brand logos, text information, product images, etc.), optimize advertising layout and content, and provide personalized user experience.

[0004] Accuracy and latency in eye tracking systems are often at odds with each other. High-end eye tracking systems address this issue by using high-speed cameras, proprietary protocols, and custom readout interfaces to maximize bandwidth. But the result is often that they are bulky and power-hungry. These complexities are inevitable due to the huge amount of data generated by high-speed, high-resolution cameras. But in near-eye gaze tracking, most of this data is redundant. Only the pupil moves, while most of the image does not change. In addition, existing user intent recognition methods based on pure text and pure two-dimensional images have certain limitations and fail to fully exploit the user's multi-source interaction information. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal user intention recognition method and system based on eye tracking, which improves the speed and power consumption, while breaking through the limitations of existing methods, fully mining the multi-source interactive information of users, and enhancing the robustness and accuracy of user intention recognition.

[0006] The technical solution to achieve the purpose of the present invention is:

[0007] A multimodal user intention recognition method based on eye tracking, comprising:

[0008] Step 1: divide the eye into two parts, namely, the pupil and the upper eyelid, according to its structure, and construct an eye model including a pupil model, an upper eyelid model and a light spot model in combination with the reflected light spot of the infrared light source on the user's eyeball;

[0009] Step 2, obtaining eye event data and grayscale frame data from the event camera, constructing a candidate point set, and fitting the eye model through the candidate point set to determine the eye model parameters and the pupil center position;

[0010] Step 3, obtaining the coordinates of the gaze point in the two-dimensional image where the user is gazing;

[0011] Step 4, repeat steps 2-3, obtain eye model parameters, pupil center position and gaze point coordinates, build a gaze point dataset, and train a gaze point estimation model;

[0012] Step 5, obtaining multimodal data of the user's gaze in the two-dimensional picture according to the gaze point estimation model, the Yolo-world target recognition model, and the StrucTexT pre-training model, and constructing a gaze information graph based on the multimodal data;

[0013] Step 6: Repeat step 6 to build a gaze information graph dataset based on the intended actions of interest, and train the intent recognition model;

[0014] Step 7: Build a gaze information graph in real time and use the trained intent recognition model to identify the user's intended actions.

[0015] Furthermore, the pupil model is an ellipse, and any point (x, y) on the ellipse satisfies the quadratic curve equation:

[0016]

[0017] Among them, the parameters a>0, c>0, and a≠c represent the size of the ellipse, which is inversely proportional to the square of the main axis of the ellipse; the parameter b represents the direction of the ellipse, and when b≠0, the ellipse axis is not parallel to the coordinate axis; d and e are parameters that affect the position of the center point of the ellipse; the parameter f represents the offset between the ellipse and the plane, represents the parameter set of the pupil model ellipse E, .

[0018] Furthermore, the upper eyelid model is a parabola, and any point (x, y) on the parabola satisfies the quadratic curve equation:

[0019]

[0020] Among them, the parameter u≠0 represents the opening direction of the parabola, v is the parameter that affects the position of the symmetry axis of the parabola, and w determines the intersection of the parabola and the y-axis. Represents the parabola P parameter set: .

[0021] Furthermore, the light spot model is a circle, and any point (x, y) on the circle satisfies the quadratic curve equation:

[0022]

[0023] Among them, the parameters g and h are related to the horizontal and vertical coordinates of the center of the circle, r is related to the radius of the circle, and ; Represents a circular C parameter set: .

[0024] Furthermore, the step 2 specifically includes: Step 2.1, considering the pixels at the edge of the pupil in the grayscale frame image as a set of candidate points , binarize the grayscale frame, and obtain the candidate point set by edge detection and recognition through the DoG operator for:

[0025]

[0026]

[0027] Among them, θ is the threshold, The standard deviation is The Gaussian kernel of The standard deviation is Gaussian kernel of

[0028] Step 2.2, set the threshold δ, and obtain the candidate point set within the range of δ from the ellipse boundary for:

[0029]

[0030] in, yes Projection point on the ellipse;

[0031] Step 2.3, based on the candidate point set , use the least squares method to fit the ellipse E representing the pupil to determine the pupil model parameters and obtain the pupil center position ;

[0032] Step 2.4: similarly to steps 2.1 to 2.3, the parameters of the upper eyelid model and the light spot model are determined by fitting.

[0033] Furthermore, the gaze point estimation model is based on the MLP model, with 3 hidden layers, ReLU as the activation function, Adam as the optimizer, and mean square error as the loss function.

[0034] Furthermore, the step 5 specifically includes:

[0035] Step 5.1, divide the two-dimensional image into N element nodes according to the content boundary. For each element node, use the Yolo-world model to identify the image features in the two-dimensional image, use the StrucTexT pre-trained model to identify the text features in the image, and use the image features and text features as node features;

[0036] Step 5.2, use the gaze point estimation model trained in step 4 to obtain the gaze point coordinates. If the gaze point coordinates are located in the set element node A area, the gaze point is considered to be element node A. The time when the gaze point coordinates enter the element node A area is recorded as t1, and the time when the gaze point coordinates leave the element node area A is recorded as t2, then the gaze duration t=t2-t1 is obtained. If the gaze point coordinates move from the element node A area to the element node area B area, the gaze point moves in the direction from A to B.

[0037] Step 5.3, construct a gaze information graph including node features, gaze point duration and movement direction.

[0038] Furthermore, the process of the intention recognition model recognizing the user's intended action is:

[0039] For each element node, the extracted text features are converted into a fixed-length semantic vector using a Bert encoder, and the extracted image features are converted into a fixed-length semantic vector using a one-hot encoder. All semantic vectors in the element block are concatenated and reduced to a fixed element semantic vector using a linear layer. ,in Represents the number of the element block;

[0040] The node movement direction is determined according to the gaze movement direction between element nodes, and the intention vector is obtained using the graph embedding model.

[0041] Furthermore, the cross entropy loss is used in the training of the intent recognition model, which is:

[0042]

[0043] in, is the probability distribution, represents the probability that the sample belongs to the i-th type of intention action, is the representation of the sample label. When the sample belongs to the i-th type of intention action ,otherwise , C is the sample intention action label.

[0044] A multimodal user intention recognition system based on eye tracking, comprising:

[0045] The eye model building unit divides the eye into two parts, namely, the pupil and the upper eyelid, according to its structure, and builds an eye model including a pupil model, an upper eyelid model and a light spot model in combination with the reflected light spot of the infrared light source on the user's eyeball;

[0046] A parameter determination unit, which obtains eye event data and grayscale frame data from an event camera sensor, constructs a candidate point set, and performs eye model fitting through the candidate point set to determine eye model parameters and pupil center position;

[0047] A gaze point coordinate acquisition unit, which acquires the coordinates of the gaze point in the two-dimensional image that the user is gazing at;

[0048] A gaze point estimation model training unit obtains eye model parameters, pupil center position and gaze point coordinates, builds a gaze point dataset, and trains a gaze point estimation model;

[0049] A gaze information graph construction unit obtains multimodal data of the user's gaze in the two-dimensional picture according to the gaze point estimation model, the Yolo-world target recognition model, and the StrucTexT pre-training model, and constructs a gaze information graph based on the multimodal data;

[0050] The intention recognition model training unit obtains the gaze information graph of the intended action of interest, constructs the gaze information graph dataset, and trains the intention recognition model;

[0051] The user intention recognition unit builds a gaze information map in real time and uses the trained intention recognition model to recognize user intention actions.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. Compared with other user intention recognition methods, this application adopts a fusion strategy of multimodal data such as comprehensive eye movement events, text information, image information, etc., which can capture and understand the user's true intention more comprehensively and accurately. The comprehensive application of this multimodal data not only enhances the robustness and accuracy of user intention recognition, but also enables the system to more intelligently adapt to the complex needs of different users and different scenarios, thereby providing a more personalized service experience;

[0054] 2. Compared with existing eye tracking methods, this application uses an event camera that can adaptively sample when the eye moves, generating a series of pixel-by-pixel data packets with timestamps, including pixel position, sampling time, etc., called events; in eye tracking, movement is sparse in time and space, and by only sampling and processing relevant information, speed and power consumption are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0057] Combination Figure 1 , this embodiment provides a multimodal user intention recognition method based on eye tracking, including:

[0058] Step 1: Divide the eye into two parts according to its structure: pupil and upper eyelid, and build an eye model based on the reflected light spot of the infrared light source on the user's eyeball. Specifically include:

[0059] Step 1.1, define an ellipse E representing the pupil, and for any point (x, y) on the ellipse, the quadratic curve equation is:

[0060]

[0061] Among them, a>0, c>0, and a≠c, representing the size of the ellipse, which is inversely proportional to the square of the main axis of the ellipse; b represents the direction of the ellipse, and when b≠0, the ellipse axis is not parallel to the coordinate axis; d and e affect the position of the center point of the ellipse; f represents the offset between the ellipse and the plane; Represents the parameter set of the ellipse E:

[0062]

[0063] Step 1.2, define a parabola P representing the upper eyelid, and for any point (x, y) on the parabola, the quadratic curve equation is satisfied:

[0064]

[0065] Among them, u≠0, representing the opening direction of the parabola; v affects the position of the symmetry axis of the parabola; w determines the intersection of the parabola and the y-axis; Represents the parabola P parameter set:

[0066]

[0067] Step 1.3, define a circle C representing the light spot, and for any point (x, y) on the circle, the quadratic curve equation is satisfied:

[0068]

[0069] Among them, g and h are related to the horizontal and vertical coordinates of the center of the circle, r is related to the radius of the circle, and ; Represents a circular C parameter set:

[0070]

[0071] Step 1.4, Combination , , , construct the eye parameter model:

[0072] ={a,b,c,d,e,f,u,v,w,g,h,r}

[0073] Step 2: Obtain eye event data (pixel changes, optical flow changes, etc.) and grayscale frame data from the event camera, construct a candidate point set D, and fit the pupil ellipse E and pupil center , including:

[0074] Step 2.1: Consider the pixels near the pupil edge in the grayscale frame image as a set of candidate points When the pupil is illuminated by an infrared light source, it appears as a dark spot in the grayscale frame. The grayscale frame is binarized and the DoG (Difference of Gaussian) operator is used for edge detection to identify the candidate point set.

[0075]

[0076]

[0077] Where θ is the threshold, The standard deviation is The Gaussian kernel of The standard deviation is Gaussian kernel

[0078] Step 2.2, set the threshold δ, and obtain the candidate point set within the range of δ from the currently estimated ellipse boundary , in, yes The projected point on the ellipse.

[0079]

[0080] Step 2.3, based on the candidate point set , use the least squares method to fit the ellipse E representing the pupil, is the set of fitted ellipse parameters and obtains the pupil center position .

[0081]

[0082] Step 3: Repeat step 2 to construct a candidate point set, fit the upper eyelid parabola and the spot circle respectively, and combine the ellipse parameters to obtain the fitted eye model parameters. .

[0083] Step 4: The front-view camera obtains the user's gaze on the two-dimensional image and marks the gaze point coordinates in the two-dimensional image as .

[0084] Step 5: construct a gaze point dataset and train a gaze point estimation model. The gaze point estimation model can adopt an existing learning model, such as an MLP model. The training process specifically includes:

[0085] Step 5.1, repeat steps 2 to 4 to obtain a series of eye model parameters, pupil center position, and gaze point coordinate data to construct a gaze point dataset.

[0086] Step 5.2, select 3 hidden layers, ReLU as the activation function, Adam as the optimizer, build the gaze point estimation model, use the mean square error as the loss function, and use the gaze point dataset to train the gaze point estimation model

[0087] Step 6: Obtain the multimodal data of the user's gaze in the two-dimensional picture, namely, the element block type, size, and image / text features, based on the gaze point estimation model, Yolo-world target recognition model, and StrucTexT pre-training model, and construct a gaze information graph. Specifically, it includes:

[0088] Step 6.1, divide the two-dimensional picture into N element nodes according to the content boundary. For each element node, use the Yolo-world model to identify the image features in the two-dimensional picture, use the StrucTexT pre-trained model to identify the text features in the picture, and use the image features and text features as node features.

[0089] Step 6.2, obtain the gaze point coordinates according to the gaze point estimation model obtained in step 5. If the gaze point coordinates are located in the element node A area, it is considered that the gaze is on the element node A. The time when the gaze point coordinates enter the element node A is recorded as t1, and the time when the gaze point coordinates leave the element node A is recorded as t2, and the gaze duration t-t2-t1 is obtained.

[0090] Step 6.3, construct a gaze information graph, where the graph nodes represent the gaze point elements, the edges represent the movement relationship of the gaze point between the elements, and the edge attributes include the gaze point movement direction and gaze duration. Specifically:

[0091] The graph nodes represent the gaze point elements, and the node attributes include the node features extracted in step 6.1; the edges represent the movement relationship of the gaze point between the elements (if the gaze point coordinate moves from element node A to element node B, the gaze point movement direction is from A to B, and a directed edge A->B is established between element node A and element node B), and the edge attributes include the gaze duration t recorded in step 6.2.

[0092] Step 7: In the target application scenario (such as VR games), collect the gaze information graph dataset and train the intent recognition model. Specifically include:

[0093] Step 7.1, define the intended actions of interest (e.g., buy equipment, quit the game, etc.), repeat step 6 to build the gaze information graph dataset

[0094] Step 7.2: For each element node, convert the extracted text features into a fixed-length semantic vector using a Bert encoder (a bidirectional encoder from Transformers), convert the extracted image features into a fixed-length semantic vector using a one-hot encoder, concatenate all semantic vectors in the element block, and use a linear layer to reduce the dimension into a fixed element semantic vector. ,in Represents the number of the element block.

[0095] In step 7.3, the node movement direction is determined according to the gaze movement direction between the element nodes, and the intention vector is obtained using the graph embedding model. The cross entropy loss is used during training to measure the difference between the predicted result and the actual intention action category to obtain the intention recognition model.

[0096]

[0097] in, is the probability distribution, represents the probability that the sample belongs to the i-th type of intention action, is the representation of the sample label. When the sample belongs to the i-th type of intention action ,otherwise , C is the sample intention action label.

[0098] Step 8: In the target application scenario (such as VR games), build a gaze information map in real time and use the intention recognition model to identify the user's intended actions.

[0099] A multimodal user intention recognition system based on eye tracking, comprising:

[0100] The eye model building unit divides the eye into two parts, namely, the pupil and the upper eyelid, according to its structure, and builds an eye model including a pupil model, an upper eyelid model and a light spot model in combination with the reflected light spot of the infrared light source on the user's eyeball;

[0101] A parameter determination unit, which obtains eye event data and grayscale frame data from an event camera sensor, constructs a candidate point set, and performs eye model fitting through the candidate point set to determine eye model parameters and pupil center position;

[0102] A gaze point coordinate acquisition unit, which acquires the coordinates of the gaze point in the two-dimensional image that the user is gazing at;

[0103] A gaze point estimation model training unit obtains eye model parameters, pupil center position and gaze point coordinates, builds a gaze point dataset, and trains a gaze point estimation model;

[0104] A gaze information graph construction unit obtains multimodal data of the user's gaze in the two-dimensional picture according to the gaze point estimation model, the Yolo-world target recognition model, and the StrucTexT pre-training model, and constructs a gaze information graph based on the multimodal data;

[0105] The intention recognition model training unit obtains the gaze information graph of the intended action of interest, constructs the gaze information graph dataset, and trains the intention recognition model;

[0106] The user intention recognition unit builds a gaze information map in real time and uses the trained intention recognition model to recognize user intention actions.

[0107] This application is based on VR games, and an experimental comparison is conducted with the recognition method using only gaze point information and using gaze point information and image information, as shown in Table 1, wherein "gaze point" means using only gaze point information, "gaze point + image" means using gaze point information and image information, and "gaze point + image + text" means that this application uses gaze point information, image information and text information. The experimental results show that multimodal data plays an important role in improving intent recognition.

[0108] Table 1 Comparison of intent recognition accuracy

[0109]

[0110] The present invention proposes a method for identifying gaze objects based on an eye tracker, and can estimate user intentions in combination with text, images and other modalities to perform specific tasks. Eye tracking locates the pupil position through an event camera to obtain coordinates, calculates the eye's gaze point or gaze point, and then estimates the user's intention through image element segmentation and semantic recognition. This not only enhances the robustness and accuracy of user intention recognition, but also enables the system to more intelligently adapt to the complex needs of different users and different scenarios, thereby providing a more personalized service experience.

[0111] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multimodal user intention recognition method based on eye tracking, characterized in that: include: Step 1: divide the eye into two parts, namely, the pupil and the upper eyelid, according to its structure, and construct an eye model including a pupil model, an upper eyelid model and a light spot model in combination with the reflected light spot of the infrared light source on the user's eyeball; Step 2, obtaining eye event data and grayscale frame data from the event camera, constructing a candidate point set, and fitting the eye model through the candidate point set to determine the eye model parameters and the pupil center position; Step 3, obtaining the coordinates of the gaze point in the two-dimensional image where the user is gazing; Step 4, repeat steps 2-3, obtain eye model parameters, pupil center position and gaze point coordinates, build a gaze point dataset, and train a gaze point estimation model; Step 5, obtaining multimodal data of the user's gaze in the two-dimensional picture according to the gaze point estimation model, the Yolo-world target recognition model, and the StrucTexT pre-training model, and constructing a gaze information graph based on the multimodal data; Step 6: Repeat step 6 to build a gaze information graph dataset based on the intended actions of interest, and train the intent recognition model; Step 7: Build a gaze information graph in real time and use the trained intent recognition model to identify the user's intended actions.

2. The multimodal user intention recognition method based on eye tracking according to claim 1, characterized in that: The pupil model is an ellipse, and any point (x, y) on the ellipse satisfies the quadratic curve equation: ； Among them, the parameters a>0, c>0, and a≠c represent the size of the ellipse, which is inversely proportional to the square of the main axis of the ellipse; the parameter b represents the direction of the ellipse, and when b≠0, the ellipse axis is not parallel to the coordinate axis; d and e are parameters that affect the position of the center point of the ellipse; the parameter f represents the offset between the ellipse and the plane, represents the parameter set of the pupil model ellipse E, .

3. The multimodal user intention recognition method based on eye tracking according to claim 1, characterized in that: The upper eyelid model is a parabola, and any point (x, y) on the parabola satisfies the quadratic curve equation: ； Among them, the parameter u≠0 represents the opening direction of the parabola, v is the parameter that affects the position of the symmetry axis of the parabola, and w determines the intersection of the parabola and the y-axis. Represents the parabola P parameter set: .

4. The multimodal user intention recognition method based on eye tracking according to claim 1, characterized in that: The light spot model is a circle, and any point (x, y) on the circle satisfies the quadratic curve equation: ； Among them, the parameters g and h are related to the horizontal and vertical coordinates of the center of the circle, r is related to the radius of the circle, and ; Represents a circular C parameter set: .

5. The multimodal user intention recognition method based on eye tracking according to claim 1, characterized in that: The step 2 specifically includes: Step 2.1: Consider the pixels at the edge of the pupil in the grayscale frame image as a set of candidate points , binarize the grayscale frame, and obtain the candidate point set by edge detection and recognition through the DoG operator for: ； ； Among them, θ is the threshold, The standard deviation is The Gaussian kernel of The standard deviation is Gaussian kernel of Step 2.2, set the threshold δ, and obtain the candidate point set within the range of δ from the ellipse boundary for: ； in, yes Projection point on the ellipse; Step 2.3, based on the candidate point set , use the least squares method to fit the ellipse E representing the pupil to determine the pupil model parameters and obtain the pupil center position ; Step 2.4: similarly to steps 2.1 to 2.3, the parameters of the upper eyelid model and the light spot model are determined by fitting.

6. The multimodal user intention recognition method based on eye tracking according to claim 1, characterized in that: The gaze point estimation model is based on the MLP model with three hidden layers, ReLU as the activation function, Adam as the optimizer, and mean square error as the loss function.

7. The multimodal user intention recognition method based on eye tracking according to claim 1, characterized in that: The step 5 specifically includes: Step 5.1, divide the two-dimensional image into N element nodes according to the content boundary. For each element node, use the Yolo-world model to identify the image features in the two-dimensional image, use the StrucTexT pre-trained model to identify the text features in the image, and use the image features and text features as node features; Step 5.2, use the gaze point estimation model trained in step 4 to obtain the gaze point coordinates. If the gaze point coordinates are located in the set element node A area, the gaze point is considered to be element node A. The time when the gaze point coordinates enter the element node A area is recorded as t1, and the time when the gaze point coordinates leave the element node area A is recorded as t2, then the gaze duration t=t2-t1 is obtained. If the gaze point coordinates move from the element node A area to the element node area B area, the gaze point moves in the direction from A to B. Step 5.3, construct a gaze information graph including node features, gaze point duration and movement direction.

8. The multimodal user intention recognition method based on eye tracking according to claim 1, characterized in that: The process of the intention recognition model identifying the user's intended action is as follows: For each element node, the extracted text features are converted into a fixed-length semantic vector through a bidirectional encoder, the extracted image features are converted into a fixed-length semantic vector using a one-hot encoder, all semantic vectors in the element block are concatenated, and the linear layer is used to reduce the dimension into a fixed element semantic vector. ,in Represents the number of the element block; The node movement direction is determined according to the gaze movement direction between element nodes, and the intention vector is obtained using the graph embedding model.

9. The multimodal user intention recognition method based on eye tracking according to claim 8, characterized in that: The cross entropy loss is used in the training of the intent recognition model: ； in, is the probability distribution, represents the probability that the sample belongs to the i-th type of intention action, is the representation of the sample label. When the sample belongs to the i-th type of intention action ,otherwise , C is the sample intention action label.

10. A multimodal user intention recognition system based on eye tracking that implements the method described in any one of claims 1 to 9, characterized in that: include: The eye model building unit divides the eye into two parts, namely, the pupil and the upper eyelid, according to its structure, and builds an eye model including a pupil model, an upper eyelid model and a light spot model in combination with the reflected light spot of the infrared light source on the user's eyeball; A parameter determination unit, which obtains eye event data and grayscale frame data from an event camera sensor, constructs a candidate point set, and performs eye model fitting through the candidate point set to determine eye model parameters and pupil center position; A gaze point coordinate acquisition unit, which acquires the coordinates of the gaze point in the two-dimensional image that the user is gazing at; A gaze point estimation model training unit obtains eye model parameters, pupil center position and gaze point coordinates, builds a gaze point dataset, and trains a gaze point estimation model; A gaze information graph construction unit obtains multimodal data of the user's gaze in the two-dimensional picture according to the gaze point estimation model, the Yolo-world target recognition model, and the StrucTexT pre-training model, and constructs a gaze information graph based on the multimodal data; The intention recognition model training unit obtains the gaze information graph of the intended action of interest, constructs the gaze information graph dataset, and trains the intention recognition model; The user intention recognition unit builds a gaze information map in real time and uses the trained intention recognition model to recognize user intention actions.

Images