Eye movement interaction object extraction method and device under bumpy condition, computer equipment, storage medium and computer program product

Abstract

The invention provides an eye movement interaction object extraction method and device under a bumpy condition, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a real-time image frame acquired in a bumpy environment, and acquiring an eye movement point sequence of a plurality of time scales generated when a user watches a target in the real-time image frame under the bumpy condition; inputting the real-time image frame into an image encoder in the target segmentation model to obtain image encoding features, and inputting the eye movement point sequences of the plurality of time scales into an eye movement prompt encoder in the target segmentation model to obtain eye movement prompt features; inputting the image coding feature and the eye movement prompting feature into a target mask decoder in a target segmentation model to obtain a target segmentation mask; and extracting the eye movement interaction object from the real-time image frame based on the segmentation mask. By adopting the method, the target object intended to be interacted by the sight line of the user can be stably and accurately extracted under the jolting and shaking conditions.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a method, apparatus, computer device, storage medium and computer program product for extracting eye-tracking interactive objects under bumpy conditions. Background Technology

[0002] In dynamic interactive scenarios such as in-vehicle navigation and drone control, the user's gaze is one of the most direct and natural channels for expressing intent.

[0003] In real-world scenarios such as vehicle movement and equipment movement, users inevitably experience continuous physical bumps and vibrations. This physical or equipment movement causes noise and jitter in the acquired eye-tracking signals. Existing object segmentation methods typically use the coordinates of a single eye-tracking point as a static cue input to a general segmentation model to achieve object segmentation. However, this approach struggles to distinguish between noise interference and genuine shifts in user attention when the input eye-tracking point randomly drifts or jitters due to external bumps. This leads to problems such as segmentation result drift, target loss, or incorrect object extraction.

[0004] Therefore, traditional technologies have the problem of being unable to stably and accurately extract the target object that the user intends to interact with under bumpy and shaking conditions. Summary of the Invention

[0005] Based on this, the purpose of this application is to at least solve one of the above-mentioned technical defects, in particular the technical defect that the prior art cannot stably and accurately extract the target object that the user intends to interact with under bumpy and shaking conditions. This application provides a method, apparatus, computer device, computer-readable storage medium and computer program product for eye-tracking interaction object extraction under bumpy and shaking conditions that can stably and accurately extract the target object that the user intends to interact with under bumpy and shaking conditions.

[0006] Firstly, this application provides a method for extracting eye-tracking interactive objects under turbulent conditions, the method comprising:

[0007] Acquire real-time image frames captured in a bumpy environment, and acquire eye-tracking point sequences at multiple time scales generated when a user gazes at a target in a real-time image frame under bumpy conditions.

[0008] Real-time image frames are input into the image encoder in the target segmentation model to obtain image encoding features, and eye movement point sequences at multiple time scales are input into the eye movement cue encoder in the target segmentation model to obtain eye movement cue features.

[0009] Image encoding features and eye-tracking cue features are input into the target mask decoder in the target segmentation model to obtain the target segmentation mask;

[0010] Based on segmentation masks, eye-tracking interaction objects are extracted from real-time image frames.

[0011] In an exemplary embodiment, the eye-tracking cue encoder includes a temporal feature extraction unit, a spatial feature extraction unit, and a spatiotemporal feature fusion unit; the temporal feature extraction unit includes multiple temporal feature extraction sub-units corresponding to multiple time scales; the eye-tracking cue sequence at multiple time scales is input into the eye-tracking cue encoder in the target segmentation model to obtain eye-tracking cue features, including:

[0012] The eye-tracking point sequence at each time scale is input into the spatial feature extraction unit to obtain the spatial feature sequence at each time scale;

[0013] The eye-tracking point sequence and spatial feature sequence at each time scale are input into the temporal feature extraction subunit at each time scale to obtain the temporal feature sequence at each time scale.

[0014] The temporal and spatial feature sequences at each time scale are input into the spatiotemporal feature fusion unit to obtain the spatiotemporal feature sequence at each time scale.

[0015] The spatiotemporal feature sequences at various time scales are fused to obtain a multi-time-scale spatiotemporal fusion feature sequence, which serves as an eye-tracking cue feature.

[0016] In an exemplary embodiment, the eye-tracking point sequence and spatial feature sequence at each time scale are input into the temporal feature extraction subunit at each time scale to obtain the temporal feature sequence at each time scale, including:

[0017] Based on the time information of the eye fixation point coordinates in the eye movement point sequence at each time scale, a time step embedding sequence corresponding to each time scale is generated;

[0018] The time step embedding sequence and spatial feature sequence of each time scale are input into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale.

[0019] In an exemplary embodiment, the image encoder includes a data preprocessing layer and a visual feature extraction layer; real-time image frames are input into the image encoder in the target segmentation model to obtain image encoded features, including:

[0020] Real-time image frames are input into the data preprocessing layer to obtain processed image frames;

[0021] The processed image frame is input into the visual feature extraction layer to obtain visual features, which are used as image coding features.

[0022] In an exemplary embodiment, prior to the step of inputting real-time image frames into the image encoder of the target segmentation model to obtain image encoded features, the method further includes:

[0023] Obtain the training dataset; the training dataset includes image frame samples collected in a static environment, eye-tracking point sequence samples of the image frame samples, and segmentation mask labels;

[0024] Noise is injected into the eye movement point sequence samples to generate noisy eye movement point sequences;

[0025] Image frame samples and noisy eye-tracking point sequences are input into a preset initial target segmentation model to obtain a predicted segmentation mask for the image frame samples; the preset initial target segmentation model includes a preset eye-tracking cue encoder, a preset image encoder, and a preset target mask decoder.

[0026] With the goal of minimizing the difference between the predicted segmentation mask and the segmentation mask label of the image frame sample, the initial target segmentation model is iteratively trained until the preset training conditions are met, thus obtaining the final target segmentation model.

[0027] In one exemplary embodiment, noise is injected into eye-tracking point sequence samples to generate noisy eye-tracking point sequences, including:

[0028] Obtain mixed noise sequences using a pre-built mixed noise model;

[0029] Noisy eye-tracking sequence is generated based on mixed noise sequence and eye-tracking point sequence samples.

[0030] Secondly, this application provides an eye-tracking interactive object extraction device under turbulent conditions, the device comprising:

[0031] The acquisition module is used to acquire real-time image frames collected in a bumpy environment, and to acquire eye-tracking point sequences at multiple time scales generated when the user gazes at a target in the real-time image frame under bumpy conditions.

[0032] The input module is used to input real-time image frames into the image encoder in the target segmentation model to obtain image encoding features, and to input eye movement point sequences at multiple time scales into the eye movement cue encoder in the target segmentation model to obtain eye movement cue features;

[0033] The segmentation module is used to input image-encoded features and eye-tracking cue features into the target mask decoder in the target segmentation model to obtain the target segmentation mask;

[0034] The extraction module is used to extract eye-tracking interaction objects from real-time image frames based on segmentation masks.

[0035] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0036] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0037] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0038] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0039] The method, apparatus, computer device, storage medium, and computer program product for extracting eye-tracking interactive objects under turbulent conditions provided in this application acquire real-time image frames collected in a turbulent environment, and acquire eye-tracking point sequences at multiple time scales generated when a user gazes at a target in the real-time image frames under turbulent conditions; input the real-time image frames into an image encoder in a target segmentation model to obtain image encoding features, and input the eye-tracking point sequences at multiple time scales into an eye-tracking cue encoder in the target segmentation model to obtain eye-tracking cue features; input the image encoding features and eye-tracking cue features into a target mask decoder in the target segmentation model to obtain a target segmentation mask; and extract eye-tracking interactive objects from the real-time image frames based on the segmentation mask; thus, by constructing a fusion... This target segmentation model, which integrates an image encoder, an eye-tracking cue encoder, and a target mask decoder, takes multiple time-scale eye-tracking point sequences as input. It addresses the challenge of unstable and inaccurate extraction of the target object intended for user interaction due to high noise and ambiguous intent in single instantaneous eye-tracking signals under turbulent conditions. Multi-scale temporal analysis is used to smooth and resolve eye-tracking jitter noise caused by turbulence. The model understands the stability and persistence of user gaze from short-term, medium-term, and long-term dimensions. Combined with the visual context provided by image encoding features, the model can accurately translate noisy gaze sequence into precise image spatial locations and object boundaries. This allows for robust target locking even during severe shaking, providing reliable input for subsequent interactions. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is an application environment diagram of an eye-tracking interactive object extraction method under bumpy conditions in one embodiment;

[0042] Figure 2 This is a flowchart illustrating a method for extracting eye-tracking interactive objects under turbulent conditions in one embodiment.

[0043] Figure 3 This is an architecture diagram of an eye-tracking interactive object extraction system under bumpy conditions in one embodiment.

[0044] Figure 4 This is a flowchart illustrating a method for extracting eye-tracking interactive objects under turbulent conditions in another embodiment;

[0045] Figure 5 This is a structural block diagram of an eye-tracking interactive object extraction device under bumpy conditions in one embodiment;

[0046] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] In recent years, eye-tracking technology has gradually become a research hotspot in the field of human-computer interaction due to its natural and non-invasive interactive characteristics. By capturing the user's point of gaze (PoG), the system can directly locate the user's focus of attention and can be applied to scenarios such as medical diagnosis, driving assistance, and intelligent navigation. However, existing eye-tracking interaction technologies have limitations: on the one hand, traditional eye-tracking interaction can only provide discrete pixel-level coordinates and cannot directly obtain the complete outline or semantic information of the object being watched by the user, making it difficult to meet complex interaction needs (such as target selection and dynamic tracking); on the other hand, in real dynamic scenarios such as in-vehicle and drone control, users often experience eye point jitter due to device vibration or body movement, and existing general segmentation models based on single-frame cues lack robustness to this.

[0049] Existing technologies, such as the GazeSAM system, combine eye tracking with the Segment Anything Model (SAM) for real-time annotation of medical images. Its core uses static PoGs (Profiles on the Frame) as input for SAM cues. While this is highly efficient in stable environments, it doesn't account for the temporal noise of PoGs in dynamic scenes, leading to a sharp drop in segmentation accuracy under vibration conditions. Another example is SAM meetsGaze, which uses passive eye-tracking data to generate segmentation cues. While it outperforms traditional methods in annotation speed, its noise model is designed only for laboratory environments and doesn't incorporate time-series modeling, failing to suppress error accumulation caused by continuous shaking. Furthermore, existing technologies suffer from significant data deficiencies: publicly available datasets (such as COCO and VIPSeg) lack multimodal data for dynamic vibration scenarios, making it difficult for model training and evaluation to accurately reflect real-world needs. Simultaneously, mainstream segmentation models (such as SAM) only support single-point or bounding box inputs in their cue encoders, failing to optimize for the spatiotemporal characteristics of eye-tracking sequences and struggling to capture the continuous changes in user attention.

[0050] In summary, the core problems of existing technologies can be summarized as follows: insufficient robustness to dynamic jitter, existing methods rely on static PoG input and do not effectively model the spatiotemporal correlation of eye-tracking point sequences, resulting in a significant decrease in segmentation accuracy in vibration scenarios; lack of utilization of temporal features, traditional cue encoders lack modeling of the temporal dependence of eye-tracking trajectories, cannot distinguish between instantaneous jitter and real attention shifts, and are susceptible to noise interference; and data-model mismatch, existing datasets do not cover multi-level physical vibration parameters, model training lacks support for real dynamic scenarios, and generalization ability is limited.

[0051] The technical solution of this application is applicable to real-time target segmentation and interaction tasks in complex dynamic scenarios such as augmented reality (AR), virtual reality (VR), and vehicle and drone control. The eye-tracking interaction object extraction method under bumpy conditions provided in the embodiments of this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 acquires real-time image frames captured in a bumpy environment, and acquires multiple time-scale eye-tracking point sequences generated when a user gazes at a target in the real-time image frame under bumpy conditions. Server 104 inputs the real-time image frames into the image encoder in the target segmentation model to obtain image encoding features, and inputs the multiple time-scale eye-tracking point sequences into the eye-tracking cue encoder in the target segmentation model to obtain eye-tracking cue features. Server 104 inputs the image encoding features and eye-tracking cue features into the target mask decoder in the target segmentation model to obtain the target segmentation mask. Based on the segmentation mask, server 104 extracts the eye-tracking interaction object from the real-time image frame. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0052] In one exemplary embodiment, such as Figure 2 As shown, a method for extracting eye-tracking interactive objects under turbulent conditions is provided, and this method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S202 to S208. Wherein:

[0053] Step S202: Acquire real-time image frames captured in a bumpy environment, and acquire eye-tracking point sequences at multiple time scales generated when the user gazes at a target in the real-time image frames under bumpy conditions.

[0054] Among them, real-time image frames acquired under bumpy conditions refer to scene images captured by cameras under unstable physical conditions such as vibration and shaking at the current moment. These images may contain motion blur or jitter artifacts.

[0055] Among them, the eye-tracking point sequence generated when a user gazes at a target under turbulent conditions refers to a series of gaze point coordinate data that change over time when the user's eyes are focused on a target on a screen or in a real scene, as the user's head and body move with the carrier (such as a vehicle or airplane) and the user's head and body move with the carrier (such as a vehicle or airplane).

[0056] Among them, eye-tracking point sequences at multiple time scales refer to the time-series sequences of different granularities obtained by analyzing and segmenting the original eye-tracking fixation point coordinate data within the same time period according to different time window lengths (such as 5 frames, 20 frames, and 50 frames) to capture multi-level information from instantaneous intent to sustained attention.

[0057] In practical applications, a short window of 5 frames is extremely sensitive to instantaneous high-frequency vibration noise and is used to capture abnormal jumps; a medium window of 20 frames is used to capture stable gaze trajectories; and a long window of 50 frames is used to capture the long-term drift trend of the user's gaze point.

[0058] Optionally, the server acquires real-time image frames of the current environment via a camera, and simultaneously collects the user's eye gaze data via an eye tracker. The server preprocesses the raw eye-tracking data to generate eye-tracking point sequences at multiple time scales.

[0059] Step S204: Input the real-time image frame into the image encoder in the target segmentation model to obtain image encoding features; and input the eye movement point sequence at multiple time scales into the eye movement cue encoder in the target segmentation model to obtain eye movement cue features.

[0060] Among them, the target segmentation model is a deep learning network designed to combine visual and eye-tracking signals to segment specific targets from images that the user intends to interact with.

[0061] The image encoder is a neural network module in the target segmentation model responsible for extracting multi-level visual features from the input image.

[0062] Among them, the eye-tracking cue encoder is a feature extraction module in the target segmentation model specifically designed to encode user gaze intent and spatial cues from noisy eye-tracking temporal data.

[0063] Optionally, the server sends real-time image frames to the image encoder branch of the target segmentation model to extract its image encoding features. At the same time, it sends multi-scale eye movement point sequences to another branch, namely the eye movement cue encoder, to extract eye movement cue features that can characterize the user's gaze intention.

[0064] Step S206: Input the image encoding features and eye-tracking cue features into the target mask decoder in the target segmentation model to obtain the target segmentation mask.

[0065] Among them, the target mask decoder is a neural network module in the target segmentation model responsible for fusing image coding features and eye-tracking cue features, and predicting the target region pixel by pixel.

[0066] The segmentation mask is a binary matrix with the same resolution as the input image, where pixels with a value of 1 represent regions identified as target objects and pixels with a value of 0 represent the background.

[0067] Optionally, the server inputs the image encoding features and eye-tracking cue features together into a target mask decoder, which outputs a segmentation mask of the target object.

[0068] Step S208: Extract eye-tracking interaction objects from real-time image frames based on segmentation masks.

[0069] Among them, the eye-tracking interaction object refers to the specific target object region (such as image block or outline) that is ultimately extracted from the real-time image frame based on the segmentation mask and corresponds to the user's gaze intention.

[0070] Optionally, the server can crop or mark the corresponding region from the original real-time image frame based on the segmentation mask, and output it as the final eye-tracking interactive object.

[0071] In the aforementioned method for extracting eye-tracking interaction objects under turbulent conditions, real-time image frames acquired in a turbulent environment are obtained, along with eye-tracking point sequences at multiple time scales generated when the user gazes at a target in the real-time image frames under turbulent conditions. The real-time image frames are input into an image encoder in a target segmentation model to obtain image-coded features, and the eye-tracking point sequences at multiple time scales are input into an eye-tracking cue encoder in the target segmentation model to obtain eye-tracking cue features. The image-coded features and eye-tracking cue features are input into a target mask decoder in the target segmentation model to obtain a target segmentation mask. Based on the segmentation mask, eye-tracking interaction objects are extracted from the real-time image frames. Thus, a method is constructed that fuses an image encoder, an eye-tracking cue encoder, and a target mask. The decoder's target segmentation model, inputting eye movement point sequences at multiple time scales, solves the problem that under turbulent conditions, single instantaneous eye movement signals are noisy and have ambiguous intent, making it impossible to stably and accurately extract the target object that the user intends to interact with. It uses multi-scale temporal analysis to smooth and resolve eye movement jitter noise caused by turbulence, and understands the user's gaze stability and intent persistence from multiple dimensions such as short time, medium time, and long time. Combined with the visual context provided by image encoding features, the model can accurately translate noisy gaze point sequences into precise image spatial locations and object boundaries, so that even in violent shaking, it can reliably lock the target that the user's true intention is pointing to, thus providing reliable input for subsequent interactions (such as gaze-click and gaze manipulation).

[0072] In an exemplary embodiment, the eye-tracking cue encoder includes a temporal feature extraction unit, a spatial feature extraction unit, and a spatiotemporal feature fusion unit. The temporal feature extraction unit includes multiple temporal feature extraction subunits corresponding to multiple time scales. The process of inputting eye-tracking point sequences from multiple time scales into the eye-tracking cue encoder in the target segmentation model to obtain eye-tracking cue features includes: inputting the eye-tracking point sequence of each time scale into the spatial feature extraction unit to obtain a spatial feature sequence of each time scale; inputting the eye-tracking point sequence and spatial feature sequence of each time scale into the temporal feature extraction subunit of each time scale to obtain a temporal feature sequence of each time scale; inputting the temporal feature sequence and spatial feature sequence of each time scale into the spatiotemporal feature fusion unit to obtain a spatiotemporal feature sequence of each time scale; and fusing the spatiotemporal feature sequences of each time scale to obtain a multi-time-scale spatiotemporal fused feature sequence, which serves as the eye-tracking cue feature.

[0073] The temporal feature extraction unit is a neural network component specifically designed to extract the temporal dynamic patterns (such as the trend, stability, and periodicity of gaze trajectory) from eye-tracking point sequences with temporal information. It is equipped with independent sub-units for different time scales.

[0074] The spatial feature extraction unit is a neural network component used to map the original two-dimensional eye-tracking point coordinates to a high-dimensional feature space and capture the relative positional relationships between points.

[0075] The spatiotemporal feature fusion unit is a neural network component used to effectively combine the extracted temporal features with spatial features to generate a joint representation (spatiotemporal feature sequence) that simultaneously contains the information of when and where the gaze occurs.

[0076] Optionally, for each timescale (e.g., short, medium, long), the server first uses a spatial feature extraction unit (e.g., a multilayer perceptron or a small convolutional network) to convert the two-dimensional coordinates of each time point into a high-dimensional vector, forming a spatial feature sequence for that scale. Next, the original eye-tracking gaze coordinate data sequence and the corresponding spatial feature sequence for each scale are input into a dedicated temporal feature extraction subunit (e.g., a recurrent neural network (RNN) or a temporal convolutional network (TCN)) to learn the evolution of eye movements over time, obtaining a temporal feature sequence. Then, a spatiotemporal feature fusion unit (e.g., using an attention mechanism or feature concatenation followed by convolution) fuses the temporal and spatial feature sequences to generate a unified spatiotemporal feature sequence for that scale. Finally, the spatiotemporal feature sequences obtained from all timescales are fused using a weighted average or a scale-importance-based attention mechanism to form the final multi-timescale spatiotemporal fusion feature sequence, which serves as the output of the entire eye-tracking cue encoder.

[0077] In this embodiment, a refined eye-tracking cue encoder incorporating temporal, spatial, and spatiotemporal fusion units is designed. An architecture employing parallel processing and fusion across multiple time scales is adopted to achieve in-depth analysis of multi-level and multi-dimensional features of jerk eye-tracking signals. This solves the problem that simply treating jerk eye-tracking data as a sequence of coordinate points fails to effectively separate the true gaze intent signal from the noise signal introduced by jitter. Short time scales can capture immediate intent and high-frequency jitter, while long time scales can reflect stable gaze trends. Spatiotemporal fusion ensures the consistency of eye-tracking cues in time and space, enhancing the encoder's robustness to noise and its ability to represent true intent.

[0078] In an exemplary embodiment, the eye-tracking point sequence and spatial feature sequence of each time scale are input into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale, including: generating a time step embedding sequence corresponding to each time scale based on the time information of the eye-tracking fixation point coordinates in the eye-tracking point sequence of each time scale; and inputting the time step embedding sequence and spatial feature sequence of each time scale into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale.

[0079] Among them, the time step embedding sequence is obtained by encoding the time step corresponding to each eye movement data point or its sequential position in the sequence through a learnable embedding layer or fixed position encoding. This enables the model to clearly know the temporal position information of each eye movement point, thereby better understanding the order and interval of the sequence.

[0080] Optionally, for the data input to each temporal feature extraction subunit, the server not only provides the coordinate data of the eye-tracking gaze point and the extracted spatial feature sequence, but also generates an additional time step embedding sequence. Specifically, based on the timestamp of the eye-tracking gaze point coordinate acquisition or its relative position index in the current time scale sequence (such as the 1st point, the 2nd point, etc.), a vector representation (i.e., time step embedding) is generated for each time point through a trainable lookup table or a preset mathematical function. Subsequently, the spatial feature sequence representing spatial information is combined with the time step embedding sequence representing temporal position to form enhanced input features, which are then fed into the temporal feature extraction subunit (such as LSTM, GRU, or Transformer encoder) for deep temporal modeling.

[0081] In this application, the input of the temporal feature extraction layer at each time scale is the spatial feature sequence and the time step embedding sequence of that time scale, thereby realizing the extraction of temporal features based on point information (spatial features) and temporal information (time step embedding) and realizing the modeling of temporal dependencies.

[0082] In this embodiment, by introducing a time-step embedding sequence as input to the temporal model, the model is forced to consider and utilize precise temporal sequence and interval information when learning eye movement patterns. Under turbulent conditions, the distribution of eye movement points may become chaotic due to jitter, but their sequential order and relative intervals on the time axis (such as rapid saccades between two stable gazes, or prolonged sustained gazes) are key clues to reveal the user's true intentions and distinguish jitter noise. Providing time-step embedding enables the model to learn which preceding points should influence the user's gaze at a certain point in time, thereby more effectively extracting stable gaze patterns from chaotic sequences and improving the accuracy and robustness of temporal modeling.

[0083] In an exemplary embodiment, the image encoder includes a data preprocessing layer and a visual feature extraction layer; inputting real-time image frames into the image encoder in the target segmentation model to obtain image coding features includes: inputting real-time image frames into the data preprocessing layer to obtain processed image frames; and inputting the processed image frames into the visual feature extraction layer to obtain visual features, which are used as image coding features.

[0084] The data preprocessing layer is a network layer or processing module that performs normalization operations on the input real-time image frames, such as scaling the image size to a fixed size, normalizing pixel values, and converting color spaces. The aim is to adapt the input data to the training distribution of the subsequent network and improve training stability.

[0085] The visual feature extraction layer is the core of the image encoder. It is usually a deep convolutional neural network (such as ResNet, ViT or its variants) and is responsible for automatically extracting multi-level semantic feature maps from the processed image. These feature maps range from low-level edge textures to high-level object semantic information.

[0086] Optionally, the server first sends the acquired real-time image frames to a data preprocessing layer, which performs a series of predefined transformations and outputs a processed image frame with a uniform format and standardized numerical range. Subsequently, the processed image frame is sent to a visual feature extraction layer, which consists of multiple stacked convolutional blocks or Transformer blocks. During forward propagation, the layer progressively downsamples and abstracts image information, ultimately outputting a set of feature maps with rich semantic information (i.e., image-coded features). These feature maps typically have a lower spatial resolution than the original image, but more channels and contain key visual cues for target recognition and segmentation.

[0087] In this embodiment, by designing a standardized data preprocessing layer and a powerful visual feature extraction layer, it is ensured that even real-time image frames acquired in turbulent environments, which may be blurry or have uneven brightness, can be transformed into high-quality, robust image coding features. Data preprocessing reduces the variability of the input image, while the visual feature extraction layer, through its knowledge obtained from pre-training on large-scale image data, possesses strong feature representation capabilities and a certain degree of anti-perturbation ability, providing a stable and reliable visual context foundation for the target segmentation model. This enables eye-tracking cue features to be accurately interpreted and located in a clear and structured visual semantic space.

[0088] In an exemplary embodiment, before the step of inputting real-time image frames into the image encoder of the target segmentation model to obtain image encoded features, the method further includes: acquiring a training dataset; the training dataset includes image frame samples collected in a static environment, eye-tracking point sequence samples of the image frame samples, and segmentation mask labels; injecting noise into the eye-tracking point sequence samples to generate noisy eye-tracking point sequences; inputting the image frame samples and the noisy eye-tracking point sequences into a preset initial target segmentation model to obtain a predicted segmentation mask for the image frame samples; the preset initial target segmentation model includes a preset eye-tracking cue encoder, a preset image encoder, and a preset target mask decoder; iteratively training the initial target segmentation model with the goal of minimizing the difference between the predicted segmentation mask and the segmentation mask labels until a preset training condition is met to obtain the final target segmentation model.

[0089] The training dataset is a set of labeled data used for model learning, which includes image frame samples, corresponding eye-tracking point sequences, and precise segmentation masks of target objects in each image frame sample.

[0090] Among them, static environment acquisition refers to acquiring data under stable conditions without bumps, in order to ensure the purity of eye-tracking data and image quality and accurate annotation.

[0091] Noise injection is a data augmentation technique designed to simulate the effects of turbulent conditions on eye-tracking data. It generates noisy eye-tracking sequences by adding random perturbations (such as Gaussian noise or jitter simulation) that conform to specific statistical characteristics to clean eye-tracking sequence.

[0092] The preset initial target segmentation model refers to the untrained model architecture, whose parameters need to be learned from the training data.

[0093] The segmentation mask label is the true value corresponding to each image sample, which is the precise pixel-level segmentation map of the target object.

[0094] The predicted segmentation mask is the predicted value corresponding to the image sample output by the initial target segmentation model, that is, the predicted pixel-level segmentation map of the target object.

[0095] Optionally, the server first prepares or receives a high-quality training dataset. In each training iteration, a batch of samples is read, and noise is injected into each sample to perturb the eye-tracking point sequence, generating a noisy eye-tracking point sequence that simulates turbulence. Then, clean image frame samples and the noisy eye-tracking point sequence that simulates turbulence are input into the initial target segmentation model. After forward propagation, the model outputs a predicted segmentation mask. Then, the loss between the predicted segmentation mask and the true segmentation mask label is calculated. Subsequently, all trainable parameters in the model (including the eye-tracking cue encoder, image encoder, and target mask decoder) are updated through the backpropagation algorithm. This process is repeated until the model's performance on the validation set no longer improves or reaches the maximum number of iterations. At this point, training stops, and the final target segmentation model is obtained.

[0096] In this embodiment, noise is actively injected into clean eye-tracking sequence samples during the model training phase to simulate the interference of a bumpy environment. This exposes the model directly to various noise patterns during the learning process. This data-driven noise simulation training strategy forces the model's eye-tracking cue encoder and the entire segmentation network to learn to extract robust intent features from contaminated signals and to learn to rely on multi-scale temporal analysis and visual context to correct the bias caused by noise, thus solving the problem of performance degradation of the target segmentation model under real bumpy conditions.

[0097] In one exemplary embodiment, noise is injected into eye-tracking point sequence samples to generate a noisy eye-tracking point sequence, including: obtaining a mixed noise sequence using a pre-built mixed noise model; and generating a noisy eye-tracking point sequence based on the mixed noise sequence and the eye-tracking point sequence samples.

[0098] The pre-built hybrid noise model is a mathematical model or program used to generate noise data simulating various interferences encountered by eye-tracking signals in a real bumpy environment. It is typically not a single Gaussian noise, but a composite of multiple noise sources, such as: 1) high-frequency jitter noise: simulating tiny, rapid, involuntary eye movements caused by high-frequency vibrations of the carrier; 2) low-frequency drift noise: simulating systematic shifts in the gaze point caused by slow tilting and repositioning of the body or head; 3) impulse noise (blink artifacts): simulating abnormal points caused by loss or jumps in eye-tracking data during blinks. This model can be obtained through statistical modeling based on the analysis of eye-tracking data collected under real bumpy conditions.

[0099] Optionally, during training, when noise needs to be injected into a certain eye movement sequence sample, the server calls a hybrid noise model. This model dynamically generates a hybrid noise sequence of the same length as the original eye movement sequence based on preset parameters (such as jitter intensity, drift amplitude, and blink probability). Then, this hybrid noise sequence is added point by point to the original eye movement sequence sample (or fused according to a more complex function) to obtain a noisy eye movement sequence that simulates being collected under turbulence. By adjusting the parameters of the hybrid noise model, noise data with different turbulence intensities and different combinations of types can be generated, enriching the diversity of training data.

[0100] In this embodiment, data augmentation is performed using a hybrid noise model based on real physical laws, rather than simple random noise. This makes the generated training data more closely resemble the turbulence and interference characteristics of the real world. High-frequency jitter, low-frequency drift, and impulse noise have different mechanisms of influence on eye movement signals, and the model needs to learn to distinguish and process them. For example, the model may need to learn to ignore high-frequency jitter but partially compensate for low-frequency drift to maintain the continuity of the gaze area. By training under composite noise, the feature representation and segmentation strategy learned by the model have stronger physical realism and environmental adaptability, ensuring that stable and accurate interactive object extraction performance can still be maintained in complex and ever-changing real turbulence scenarios.

[0101] The system used in this application is composed of hardware integrated devices and software modules working together.

[0102] The hardware integration device mainly includes an eye-tracking data acquisition module, a computing and processing module, and a display module. The eye-tracking data acquisition module is responsible for capturing the user's eye gaze coordinates in real time at a high frequency (e.g., 90Hz). The computing and processing module consists of a high-performance computing unit equipped with a GPU, responsible for receiving eye-tracking data sequences, inertial measurement unit data sequences, and synchronized image data in real time via a high-bandwidth, low-latency wireless communication protocol. It utilizes the parallel computing capabilities of the GPU to run the core algorithms of this invention, including an eye-tracking cue encoder, an image encoder, and a target mask decoder, to complete complex spatiotemporal modeling and real-time image segmentation inference. The display module consists of a high-refresh-rate virtual reality or other display device, responsible for accurately overlaying the segmentation mask generated by the computing and processing module onto the corresponding real-time image frames. It also controls end-to-end interaction latency by optimizing the rendering pipeline and data transmission protocol, providing the user with a smooth interactive experience.

[0103] The software modules mainly include an eye-tracking data acquisition module, a data preprocessing module, an eye-tracking cue encoder module, an image encoder module, a target mask decoder module, and a post-processing module. These modules work together to achieve real-time processing of eye-tracking data and target segmentation. For ease of understanding by those skilled in the art, Figure 3An exemplary data processing flow is provided for the eye-tracking data acquisition module, data preprocessing module, eye-tracking cue encoder module, image encoder module, target mask decoder module, and post-processing module.

[0104] Figure 3 The data processing of the eye-tracking data acquisition module shown includes collecting eye-tracking point data of the user when viewing the target using an eye-tracking device, synchronously acquiring image data and head posture data of the target that the user is interested in through VR and other devices, and performing preprocessing such as standardization and scaling on the collected raw eye-tracking point data to ensure that the input meets the model requirements.

[0105] Figure 3 The eye-tracking cue encoder module shown, as one of the core improved modules of this application, is used to process and encode eye-tracking point data to generate high-quality cues to guide segmentation. Unlike existing technologies that only support single-point cues, this application redesigns it to receive eye-tracking point sequences at different time scales to capture anomalous jumps, stable gaze trajectories, and long-term drift trends. A spatiotemporal feature modeling mechanism is introduced, specifically: the eye-tracking point sequence at each time scale is input to the spatial feature extraction unit, and spatial feature modeling is achieved through position encoding to obtain the spatial feature sequence at each time scale; the eye-tracking point sequence and spatial feature sequence at each time scale are input to the temporal feature extraction subunit (temporal attention unit) at each time scale, and the temporal information is introduced to generate... A time step vector is generated and a multi-head self-attention mechanism is used to model temporal attention in order to capture the temporal dependencies of eye-tracking points and obtain the temporal feature sequence for each time scale. Based on the temporal information of the coordinates of each eye-tracking fixation point in the eye-tracking point sequence at each time scale, a time step embedding sequence corresponding to each time scale is generated. The time step embedding sequence and spatial feature sequence of each time scale are input into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale. The spatiotemporal feature sequences of each time scale are fused and the processed spatiotemporal information is deeply encoded through Transformer coding blocks. Finally, high-quality cue information for segmentation is output, resulting in a multi-time scale spatiotemporal fusion feature sequence, which serves as the eye-tracking cue feature.

[0106] Figure 3 The target mask decoder module shown performs target segmentation based on the aforementioned eye-tracking cue features and image-coded features. The specific steps include: using a multi-head self-attention mechanism to jointly model the eye-tracking cue features and image-coded features to extract spatiotemporal dependencies; fusing the processed eye-tracking cue features and image-coded features; and finally using a multilayer perceptron and other methods to process and generate a segmentation mask.

[0107] Figure 3The post-processing module shown will further optimize the generated segmentation mask by denoising, boundary repair, etc., to ensure accuracy and robustness.

[0108] During the inference and evaluation phases, the target segmentation model processes eye-tracking point sequences and image frames in real time to generate segmentation results, and can use metrics such as intersection-over-union ratio (IoU) to quantitatively evaluate segmentation performance.

[0109] In another embodiment, such as Figure 4 As shown, a method for extracting eye-tracking interactive objects under turbulent conditions is provided, and this method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0110] Step S402: Acquire real-time image frames captured in a bumpy environment, and acquire eye-tracking point sequences at multiple time scales generated when the user gazes at a target in the real-time image frames under bumpy conditions.

[0111] Step S404: Input the real-time image frame into the image encoder in the target segmentation model to obtain the image encoding features.

[0112] Step S406: Input the eye-tracking point sequence of each time scale into the spatial feature extraction unit to obtain the spatial feature sequence of each time scale.

[0113] Step S408: Input the eye-tracking point sequence and spatial feature sequence of each time scale into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale.

[0114] Step S410: Input the temporal feature sequence and spatial feature sequence of each time scale into the spatiotemporal feature fusion unit to obtain the spatiotemporal feature sequence of each time scale.

[0115] Step S412: The spatiotemporal feature sequences of each time scale are fused to obtain a multi-time scale spatiotemporal fusion feature sequence, which is used as an eye-tracking cue feature.

[0116] Step S414: Input the image encoding features and eye-tracking cue features into the target mask decoder in the target segmentation model to obtain the target segmentation mask.

[0117] Step S416: Extract eye-tracking interaction objects from real-time image frames based on segmentation masks.

[0118] It should be noted that the specific limitations of the above steps can be found in the above description of the specific limitations of an eye-tracking interactive object extraction method under turbulent conditions.

[0119] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0120] The following describes the eye-tracking interactive object extraction device under bumpy conditions provided in the embodiments of this application. The eye-tracking interactive object extraction device under bumpy conditions has the same inventive concept as the eye-tracking interactive object extraction method under bumpy conditions described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the eye-tracking interactive object extraction device under bumpy conditions provided below can be referred to the limitations of the eye-tracking interactive object extraction method under bumpy conditions described above. The eye-tracking interactive object extraction device under bumpy conditions described below and the eye-tracking interactive object extraction method under bumpy conditions described above can be referred to each other, and will not be repeated here.

[0121] In one exemplary embodiment, Figure 5 A schematic diagram of an eye-tracking interactive object extraction device under bumpy conditions provided in this application embodiment is shown below. Figure 5 As shown, the eye-tracking interactive object extraction device under bumpy conditions includes: an acquisition module 502, an input module 504, a segmentation module 506, and an extraction module 508, wherein:

[0122] The acquisition module 502 is used to acquire real-time image frames collected in a bumpy environment, and to acquire eye-tracking point sequences at multiple time scales generated when the user gazes at a target in the real-time image frame under bumpy conditions.

[0123] The input module 504 is used to input real-time image frames into the image encoder in the target segmentation model to obtain image encoding features, and to input eye movement point sequences at multiple time scales into the eye movement cue encoder in the target segmentation model to obtain eye movement cue features.

[0124] The segmentation module 506 is used to input image encoding features and eye-tracking cue features into the target mask decoder in the target segmentation model to obtain the segmentation mask of the target;

[0125] Extraction module 508 is used to extract eye-tracking interaction objects from real-time image frames based on segmentation masks.

[0126] In an exemplary embodiment, the eye-tracking cue encoder includes a temporal feature extraction unit, a spatial feature extraction unit, and a spatiotemporal feature fusion unit. The temporal feature extraction unit includes multiple temporal feature extraction subunits corresponding to multiple time scales. The input module 504 is specifically used to input the eye-tracking point sequence of each time scale into the spatial feature extraction unit to obtain the spatial feature sequence of each time scale; input the eye-tracking point sequence and spatial feature sequence of each time scale into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale; input the temporal feature sequence and spatial feature sequence of each time scale into the spatiotemporal feature fusion unit to obtain the spatiotemporal feature sequence of each time scale; and fuse the spatiotemporal feature sequences of each time scale to obtain a multi-time-scale spatiotemporal fusion feature sequence, which serves as the eye-tracking cue feature.

[0127] In an exemplary embodiment, the input module 504 is specifically used to generate a time step embedding sequence corresponding to each time scale based on the time information of the coordinates of each eye fixation point in the eye movement point sequence of each time scale; and input the time step embedding sequence and spatial feature sequence of each time scale into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale.

[0128] In an exemplary embodiment, the image encoder includes a data preprocessing layer and a visual feature extraction layer; the input module 504 is specifically used to input real-time image frames into the data preprocessing layer to obtain processed image frames; and to input the processed image frames into the visual feature extraction layer to obtain visual features, which are used as image encoding features.

[0129] In an exemplary embodiment, the apparatus further includes: a training module for acquiring a training dataset; the training dataset includes image frame samples acquired in a static environment, eye-tracking point sequence samples of the image frame samples, and segmentation mask labels; injecting noise into the eye-tracking point sequence samples to generate noisy eye-tracking point sequences; inputting the image frame samples and the noisy eye-tracking point sequences into a preset initial target segmentation model to obtain a predicted segmentation mask for the image frame samples; the preset initial target segmentation model includes a preset eye-tracking cue encoder, a preset image encoder, and a preset target mask decoder; iteratively training the initial target segmentation model with the goal of minimizing the difference between the predicted segmentation mask and the segmentation mask labels until a preset training condition is met to obtain the final target segmentation model.

[0130] In one exemplary embodiment, the training module is further configured to obtain a mixed noise sequence using a pre-built mixed noise model; and generate a noisy eye-tracking sequence based on the mixed noise sequence and eye-tracking point sequence samples.

[0131] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the eye-tracking interactive object extraction method under any of the turbulence conditions described in the above embodiments.

[0132] In one exemplary embodiment, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the eye-tracking interactive object extraction method under any of the above embodiments under turbulent conditions.

[0133] In one exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the eye-tracking interactive object extraction method under any of the above embodiments under turbulent conditions.

[0134] Indicatively, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the internal structure of a computer device 600 provided in an embodiment of this application. The computer device 600 can be provided as a server. (Refer to...) Figure 6 The computer device 600 includes a processing component 602, which further includes one or more processors, and memory resources represented by memory 601 for storing instructions, such as application programs, that can be executed by the processing component 602. The application programs stored in memory 601 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 602 is configured to execute instructions to perform the eye-tracking interactive object extraction method under turbulent conditions according to any of the above embodiments.

[0135] The computer device 600 may also include a power supply component 603 configured to perform power management of the computer device 600, a wired or wireless network interface 604 configured to connect the computer device 600 to a network, and an input / output (I / O) interface 605. The computer device 600 may operate on an operating system stored in memory 601, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0136] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0138] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0139] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting eye-tracking interactive objects under bumpy conditions, characterized in that, The method includes: Acquire real-time image frames captured in a bumpy environment, and acquire eye-tracking point sequences at multiple time scales generated when a user gazes at a target in the real-time image frames under the bumpy conditions. The real-time image frames are input into the image encoder in the target segmentation model to obtain image encoding features, and the eye movement point sequences at multiple time scales are input into the eye movement cue encoder in the target segmentation model to obtain eye movement cue features. The image encoding features and the eye-tracking cue features are input into the target mask decoder in the target segmentation model to obtain the segmentation mask of the target; Based on the segmentation mask, the eye-tracking interaction object is extracted from the real-time image frame.

2. The method according to claim 1, characterized in that, The eye-tracking cue encoder includes a temporal feature extraction unit, a spatial feature extraction unit, and a spatiotemporal feature fusion unit; the temporal feature extraction unit includes multiple temporal feature extraction sub-units corresponding to the time scale. The step of inputting the eye movement point sequences at multiple time scales into the eye movement cue encoder in the target segmentation model to obtain eye movement cue features includes: The eye-tracking point sequence for each time scale is input into the spatial feature extraction unit to obtain the spatial feature sequence for each time scale; The eye-tracking point sequence and spatial feature sequence of each time scale are input into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale; The temporal feature sequence and spatial feature sequence of each time scale are input into the spatiotemporal feature fusion unit to obtain the spatiotemporal feature sequence of each time scale; The spatiotemporal feature sequences of each time scale are fused to obtain a multi-time-scale spatiotemporal fusion feature sequence, which is used as the eye-tracking cue feature.

3. The method according to claim 2, characterized in that, The step of inputting the eye-tracking point sequence and spatial feature sequence of each time scale into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale includes: Based on the time information of the eye-tracking point coordinates in the eye-tracking point sequence of each time scale, a time step embedding sequence corresponding to each time scale is generated; The time step embedding sequence and spatial feature sequence of each time scale are input into the temporal feature extraction subunit of each time scale to obtain the temporal feature sequence of each time scale.

4. The method according to claim 1, characterized in that, The image encoder includes a data preprocessing layer and a visual feature extraction layer; The step of inputting the real-time image frame into the image encoder in the target segmentation model to obtain image encoded features includes: The real-time image frame is input into the data preprocessing layer to obtain the processed image frame; The processed image frame is input into the visual feature extraction layer to obtain visual features, which are used as the image encoding features.

5. The method according to claim 1, characterized in that, Before the step of inputting the real-time image frame into the image encoder of the target segmentation model to obtain image encoded features, the method further includes: Obtain the training dataset; the training dataset includes image frame samples collected in a static environment, eye-tracking point sequence samples of the image frame samples, and segmentation mask labels; Noise is injected into the eye movement point sequence samples to generate noisy eye movement point sequences; The image frame samples and the noisy eye-tracking point sequence are input into a preset initial target segmentation model to obtain a predicted segmentation mask for the image frame samples; the preset initial target segmentation model includes a preset eye-tracking cue encoder, a preset image encoder, and a preset target mask decoder. With the goal of minimizing the difference between the predicted segmentation mask and the segmentation mask label, the initial target segmentation model is iteratively trained until the preset training conditions are met, thus obtaining the final target segmentation model.

6. The method according to claim 5, characterized in that, The step of injecting noise into the eye-tracking point sequence sample to generate a noisy eye-tracking point sequence includes: Obtain mixed noise sequences using a pre-built mixed noise model; The noisy eye-tracking sequence is generated based on the mixed noise sequence and the eye-tracking point sequence sample.

7. A device for extracting eye-tracking interactive objects under bumpy conditions, characterized in that, The device includes: The acquisition module is used to acquire real-time image frames collected under bumpy conditions, and to acquire eye-tracking point sequences at multiple time scales generated when the user looks at a target in the real-time image frame under the bumpy conditions. The input module is used to input the real-time image frame into the image encoder in the target segmentation model to obtain image encoding features, and to input the eye movement point sequence at multiple time scales into the eye movement cue encoder in the target segmentation model to obtain eye movement cue features; The segmentation module is used to input the image encoding features and the eye-tracking cue features into the target mask decoder in the target segmentation model to obtain the segmentation mask of the target; An extraction module is used to extract eye-tracking interaction objects from the real-time image frame based on the segmentation mask.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images