Text plotting system and plotting method of mixed digital paper based on smooth tracking
Through a hybrid digital paper text drawing system based on smooth tracking, using a multi-channel interaction method combined with eye-touch control, the problem of difficult to deal with text drawing at view angles of less than 1.8° is solved in the prior art, and an efficient and natural interaction effect is achieved. In particular, the completion time is significantly shortened in long text plotting tasks.
Patent Information
- Application Number
- CN202411140022.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-29
AI Technical Summary
The existing text plotting system is difficult to deal with text plotting at a viewing angle of less than 1.8°, and lacks efficient eye-controlled interaction technology, especially in scenarios of hand dysfunction and interactive intention.
Adopting a hybrid digital paper text drawing system based on smooth tracking, collecting eye movement data and text information, identifying gaze point positions, activate interactive areas, and achieving efficient labeling of small texts through a multi-channel interaction method combined with eye touch painted.
It improves interaction accuracy and nature, reduces information load, and increases interaction robustness, especially in long text drawing tasks, which significantly shortens the task completion time.
Smart Images

Figure CN119045660A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the problem of eye-controlled small target selection, and in particular to a text plotting flow method and system for hybrid digital paper based on smooth tracking. Background Art
[0002] Currently, almost no text marking systems use eye control technology, especially when the text size is less than 1.8° visual angle. This is because traditional gaze-activated eye control interaction technology requires a larger interactive object size. However, eye control technology has advantages in scenarios with hand dysfunction and using interactive intentions. Therefore, the interaction process to solve the problem of small text marking and the system framework for hybrid digital paper text selection have become an important research direction. Summary of the invention
[0003] The technical problem to be solved by the present invention is to provide a text marking system based on smooth tracking hybrid digital paper for realizing efficient marking of small texts in view of the deficiencies in the above-mentioned prior art.
[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0005] The present invention first provides a text marking system for hybrid digital paper based on smooth tracking, comprising:
[0006] The subsystem for collecting eye movement data and calibrating gaze point parameters, the subsystem for collecting text information and generating interactive objects, the subsystem for detecting the correlation between eye movement and interactive object movement, the subsystem for identifying gaze interactive actions based on gaze point position, the subsystem for activating the interactive area module, and the subsystem for selecting the target space position;
[0007] The subsystem for collecting eye movement data and calibrating gaze point parameters is used to collect eye images and extract gaze point information from the collected eye images;
[0008] A subsystem for collecting text information and generating interactive objects, which is used to collect and recognize text information, and generate interactive objects based on the recognized text information;
[0009] A subsystem for detecting the correlation between eye movement and interactive object movement, used for calculating the movement correlation between the eye movement timing data obtained from the subsystem for collecting eye movement data and calibrating gaze point parameters and the movement timing data of each interactive object in the interactive object set obtained from the subsystem for collecting text information and generating interactive objects;
[0010] A gaze interaction action recognition subsystem based on gaze point position is used to extract gaze interaction actions from the time-series gaze point data obtained by the eye movement data collection and gaze point parameter calibration subsystem;
[0011] The interactive area activation module subsystem is used to generate an interactive area and activate interactive objects in the interactive area when the gaze interaction action recognition subsystem based on the gaze point position recognizes the gaze interaction action;
[0012] The target space position subsystem is selected to determine the interaction instruction based on the motion correlation data of the eye movement detection and interaction object motion correlation subsystem, change the state of the interaction object, and plot the target text.
[0013] A subsystem for collecting eye movement data and calibrating gaze point parameters includes: an eye camera module, a digital information display module, a gaze point detection module and a first microprocessor module;
[0014] An eye camera module, used for collecting eye images;
[0015] A gaze point detection module, used to extract gaze point information from the eye image collected by the eye camera module;
[0016] A digital information display module, used for presenting a calibration target when calibrating gaze point parameters;
[0017] The first microprocessor module is used to store, sample and output the gaze point information extracted by the gaze point detection module.
[0018] A subsystem for collecting text information and generating interactive objects includes: an environment camera module, a space positioning module, a text detection module, an interactive object generation module, an interactive object controller module and a second microprocessor module;
[0019] An environment camera module is used to collect environment images, including text images in the environment and surrounding environment images;
[0020] A spatial positioning module, used to calculate the spatial coordinates of the head-mounted display and text based on the environment image;
[0021] The text detection module is used to extract text information from text images and find the coordinates of all space positions in the text;
[0022] An interactive object generation module, used to generate an instance of an interactive object at a specified location;
[0023] A first interactive object controller module, used to control the movement of the interactive object and collect the real-time position of the interactive object;
[0024] The second microprocessor module includes a timer and a sampler, and is used to assist the functions of the interactive object controller module.
[0025] A subsystem for detecting the correlation between eye movement and interactive object movement, including: a gaze point detection module, an interactive object controller module, a movement correlation detection module, an interactive object state control module, and an interactive object state locking module;
[0026] A gaze point detection module is used to obtain the real-time coordinate points of gaze;
[0027] a second interactive object controller module for controlling the movement of each activated interactive object;
[0028] A motion correlation detection module, used to detect the motion correlation between the temporal gaze point coordinates of the eye and the temporal point coordinates of the moving slider in the interactive object;
[0029] An interactive object state control module is used to control the state performance of the interactive object;
[0030] The interactive object state locking module sets a threshold according to the motion correlation, locks the state of the interactive object when it reaches the preset state, and sets the state unlocking time. When the unlocking time limit is reached, the locked state of the interactive object is unlocked.
[0031] A gaze interaction action subsystem is provided for recognizing gaze point positions according to gaze point positions, including: a gaze point detection module and a gaze recognition module;
[0032] A gaze point detection module, used to provide the gaze point position in real time;
[0033] A gaze recognition module is used to recognize gaze interaction actions from the time-series gaze point position data and give the gaze point position;
[0034] The interactive area activation module subsystem includes an interactive object search module and an interactive area generation module;
[0035] An interactive object search module, used to search for multiple nearest interactive objects according to the gaze point position;
[0036] The interactive area generation module is used to generate an interactive area that wraps the outer contour of the interactive object searched according to the interactive object searched.
[0037] The target space position selection subsystem includes: a selection instruction monitoring module, an interactive object state control module, and a text book marking control module;
[0038] Select the command monitoring module to monitor interactive actions at all times;
[0039] An interactive object state control module, used to control the interactive object to change from a locked state to a triggered state;
[0040] The text marking control module is used to mark the target text.
[0041] The present invention also provides a text marking method for hybrid digital paper based on smooth tracking, comprising the following steps:
[0042] Step 1: Based on the subsystem for collecting text information and generating interactive objects and the subsystem for collecting eye movement data and calibrating gaze point parameters, text recognition is performed and interactive objects are prefabricated;
[0043] Step 2: Based on the gaze point position, the gaze interaction action subsystem and the interaction area activation module subsystem are identified to monitor the gaze point position in real time and activate the interaction area;
[0044] Step 3: Based on the subsystem for detecting the correlation between eye movement and interactive object movement and the subsystem for selecting the target space position, the target space position is selected through eye control actions;
[0045] Step 4: Based on the selected target space position subsystem, plot the target text.
[0046] Step 1 includes:
[0047] Step 11: Collect eye movement data and calibrate gaze point parameters:
[0048] Step 12: Collect text information and generate interactive objects.
[0049] Step 2 includes:
[0050] Step 21: Identify gaze interaction actions according to gaze point positions;
[0051] Step 22: When a gaze action is detected, the interactive area and the interactive objects contained in the interactive area are activated.
[0052] Step 3 includes:
[0053] Step 31: Detect the correlation between eye movement and movement of the interactive object;
[0054] Step 32: Locking the state of the interactive object according to the motion correlation value of the interactive object;
[0055] Step 33: Select the target space position by pressing keys or other commands.
[0056] Compared with the prior art, the present invention has the following beneficial effects:
[0057] The interaction accuracy is higher. The text marking system of this solution uses eye control technology and can mark text with a size smaller than 1.8° viewing angle. Since the technical solution uses a smooth tracking eye control interaction recognition method, it can process position-independent interaction instructions. Compared with existing eye control interaction technologies such as gaze, the present invention improves the interaction accuracy of the eye control text marking system.
[0058] The interaction method is more natural. The text marking method of this solution uses a phased operation, including key stages such as area activation, interaction intention locking, and text marking, which reduces the information load and increases the robustness of interaction. The text marking method of this solution uses a multi-channel interaction method that combines eye and touch. The eyes locate and lock the interaction target, and the touch is used for confirmation, which increases the naturalness of the interaction.
[0059] The interaction efficiency is high. Through ergonomic tests, it is found that compared with the traditional method of marking the eye scanning area, the present invention has a shorter task completion time under the task conditions of long text (2-3 sentences, when the text is presented at a viewing angle of 1.8°, the length is 6-8 lines), large font size (viewing angle 1.8°), and long text with medium font size (viewing angle 1.05°). BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 : Operation flow chart;
[0061] Figure 2 :Interactive flowchart of text plotting on hybrid digital paper based on smooth tracking;
[0062] Figure 3 :User scenario diagram;
[0063] Figure 4 :System framework diagram;
[0064] Figure 5 : A diagram of the composition of a microprocessor module for collecting eye gaze points;
[0065] Figure 6 : Collect text information and generate a system structure diagram of interactive objects;
[0066] Figure 7 :Interaction object description diagram;
[0067] Figure 8 : Schematic diagram of text plotting interaction process;
[0068] Fig. 9 : Flowchart of the text plotting system verification experiment;
[0069] Fig.10 : Experimental interface for text plotting system verification;
[0070] Fig.11 : A bar chart comparing the task completion rate under different condition combinations;
[0071] Fig.12 : Box plot of the comparison of task completion time under different condition combinations;
[0072] Fig.13 : Bar graph of comparative interaction styles for the cognitive load questionnaire;
[0073] Fig.14 : A bar chart comparing the interaction methods of the user experience questionnaire;
[0074] Fig.15 : Line chart comparing interaction methods of the user experience questionnaire. DETAILED DESCRIPTION
[0075] The present invention is described in detail below in conjunction with the accompanying drawings:
[0076] The text plotting system of the hybrid digital paper based on smooth tracking of the present invention is as follows: Figure 4 As shown, including:
[0077] The subsystem includes a subsystem for collecting eye movement data and calibrating gaze point parameters, a subsystem for collecting text information and generating interactive objects, a subsystem for detecting the correlation between eye movement and interactive object movement, a subsystem for identifying gaze interactive actions based on gaze point position, an interactive area activation module subsystem, and a subsystem for selecting the target space position.
[0078] The subsystem for collecting eye movement data and calibrating gaze point parameters is used to collect eye images and extract gaze point information from the collected eye images;
[0079] The subsystem for collecting text information and generating interactive objects is used to collect and recognize text information, and generate interactive objects based on the recognized text information.
[0080] A subsystem for detecting the correlation between eye movement and interactive object movement, used for calculating the movement correlation between the eye movement timing data obtained from the subsystem for collecting eye movement data and calibrating gaze point parameters and the movement timing data of each interactive object in the interactive object set obtained from the subsystem for collecting text information and generating interactive objects;
[0081] A gaze interaction action recognition subsystem based on gaze point position is used to extract gaze interaction actions from the time-series gaze point data obtained by the subsystem for collecting eye movement data and calibrating gaze point parameters through a gaze recognition algorithm;
[0082] The interactive area activation module subsystem is used to generate an interactive area and activate interactive objects in the interactive area when the gaze interaction action recognition subsystem based on the gaze point position recognizes the gaze interaction action;
[0083] The target space position subsystem is selected to determine the interaction instruction based on the motion correlation data of the eye movement detection and interaction object motion correlation subsystem, change the state of the interaction object, and plot the target text.
[0084] A subsystem for collecting eye movement data and calibrating gaze point parameters includes: an eye camera module, a digital information display module, a gaze point detection module and a first microprocessor module;
[0085] An eye camera module, used for collecting eye images;
[0086] A gaze point detection module, used to extract gaze point information from the eye image collected by the eye camera module;
[0087] A digital information display module, used for presenting a calibration target when calibrating gaze point parameters;
[0088] The first microprocessor module is used to store, sample and output the gaze point information extracted by the gaze point detection module.
[0089] A subsystem for collecting text information and generating interactive objects includes: an environment camera module, a space positioning module, a text detection module, an interactive object generation module, an interactive object controller module and a second microprocessor module;
[0090] An environment camera module is used to collect environment images, including text images in the environment and surrounding environment images;
[0091] A spatial positioning module, used to calculate the spatial coordinates of the head-mounted display and text based on the environment image;
[0092] The text detection module is used to extract text information from text images and find the coordinates of all space positions in the text;
[0093] An interactive object generation module, used to generate an instance of an interactive object at a specified location;
[0094] The interactive object controller module is used to control the movement of the interactive object and collect the real-time position of the interactive object;
[0095] The second microprocessor module includes a timer and a sampler, and is used to assist the functions of the interactive object controller module.
[0096] A subsystem for detecting the correlation between eye movement and interactive object movement, including: a gaze point detection module, an interactive object controller module, a movement correlation detection module, an interactive object state control module, and an interactive object state locking module;
[0097] A gaze point detection module is used to obtain the real-time coordinate points of gaze;
[0098] Interactive object controller module, used for the movement of each activated interactive object in the space;
[0099] A motion correlation detection module, used to detect the motion correlation between the temporal gaze point coordinates of the eye and the temporal point coordinates of the moving slider in the interactive object;
[0100] An interactive object state control module is used to control the state performance of the interactive object;
[0101] The interactive object state locking module sets a threshold value according to the motion correlation, locks the state of the interactive object when it is reached, and sets a state unlocking time. When the unlocking time limit is reached, the locked state of the interactive object is unlocked.
[0102] A gaze interaction action subsystem is provided for recognizing gaze point positions according to gaze point positions, including: a gaze point detection module and a gaze recognition module;
[0103] A gaze point detection module, used to provide the gaze point position in real time;
[0104] A gaze recognition module is used to recognize gaze interaction actions from the time-series gaze point position data and give the gaze point position;
[0105] The interactive area activation module subsystem includes an interactive object search module and an interactive area generation module;
[0106] An interactive object search module, used to search for multiple nearest interactive objects according to the gaze point position;
[0107] The interactive area generating module is used to generate an interactive area that encloses the outer contour of the searched interactive object according to the searched interactive object.
[0108] The target space position selection subsystem includes: a selection instruction monitoring module, an interactive object state control module, and a text book marking control module;
[0109] Select the command monitoring module to monitor interactive actions at all times;
[0110] An interactive object state control module, used to control the interactive object to change from a locked state to a triggered state;
[0111] The text marking control module is used to mark the target text.
[0112] The first microprocessor module of the subsystem for collecting eye movement data and calibrating gaze point parameters and the second microprocessor module of the subsystem for collecting text information and generating interactive objects can be the same processor or different processors.
[0113] The interactive object controller module in the subsystem for collecting text information and generating interactive objects and the subsystem for detecting the correlation between eye movement and interactive object movement are the same module or two modules; if they are the same module, the functions to be implemented in each subsystem are different.
[0114] The subsystem for detecting the correlation between eye movement and interactive object movement and the gaze point detection module in the subsystem for identifying gaze interaction actions according to the gaze point position are the same module.
[0115] The microprocessor module is composed of Figure 5 As shown, it includes a signal memory, a timer, a signal sampler, and a gaze point calibrator. The signal memory stores the gaze point information extracted by the gaze point detection module. The timer counts at the beginning of signal acquisition. The signal sampler samples and outputs the signal stored in the signal memory according to the set sampling frequency and the value of the current timer. The gaze point calibrator calibrates the gaze point according to the target point position and gaze point information displayed by the digital information display module, and adjusts the parameters of the gaze point detection module.
[0116] An embodiment of the present invention is Figure 3 As shown, it includes: head-mounted display, physical paper, Bluetooth keyboard, etc.
[0117] The head-mounted display obtains eye gaze point data, trigger signals of hand triggers, and presents digital information to the user. The head-mounted display is the main carrier of the text plotting system based on smooth pursuit hybrid digital paper, which includes a subsystem for collecting eye movement data and calibrating gaze point parameters, a subsystem for collecting text information and generating interactive objects, a subsystem for detecting the correlation between eye movement and interactive object movement, a subsystem for identifying gaze interaction actions based on gaze point position, a subsystem for activating the interactive area module, and a subsystem for selecting the target space position.
[0118] Physical paper is a physical paper with a QR code. The head-mounted display's environmental camera module scans the QR code to locate the physical paper's spatial position, and overlays the digital information on the physical paper. The physical paper contains the QR code and the physical version of the text to be marked. The digital version of the text to be marked will overlap with the physical version of the text.
[0119] A Bluetooth keyboard is a way to implement a hand trigger, which sends interactive instructions to the head-mounted display by tapping your fingers. Other triggers for interactive instructions can be foot buttons, facial expressions, blinking, and other interactive actions.
[0120] The text plotting method of hybrid digital paper based on smooth tracking of the present invention comprises the following steps:
[0121] Step 1: Identify text and pre-create interactive objects, including:
[0122] Step 11: Collect eye movement data and calibrate gaze point parameters:
[0123] The system for collecting eye movement data and calibrating gaze point parameters includes: an eye camera module, a digital information display module, a gaze point detection module and a microprocessor module.
[0124] The eye camera module is used to collect eye images. Depending on the gaze point detection algorithm, it is decided whether to use a normal camera or an infrared camera. When the system is a head-mounted glasses, the eye camera module should be small enough to be placed on the glasses and move with the head. When the system is a desktop type, the eye camera module can be placed under the display and does not move with the head.
[0125] The digital information display module is used to present the calibration target when calibrating the gaze point parameters.
[0126] The gaze point detection module is used to extract the gaze point information from the collected eye images. The optional gaze point detection algorithms include pupil center corneal reflection (PCCR) method, deep learning method, geometric appearance based method, etc.
[0127] Here gaze is defined as the general eye observation behavior, and the gaze point is defined as the intersection of the line of sight and the object being pointed at during observation.
[0128] The microprocessor module is composed of Figure 5 As shown, it includes a signal memory, a timer, a signal sampler, and a gaze point calibrator. The signal memory stores the gaze point information extracted by the gaze point detection module. The timer counts at the beginning of signal acquisition. The signal sampler samples and outputs the signal stored in the signal memory according to the set sampling frequency and the value of the current timer. The gaze point calibrator calibrates the gaze point according to the target point position and gaze point information displayed by the digital information display module, and adjusts the parameters of the gaze point detection module.
[0129] The process of collecting eye movement data and calibrating gaze point parameters is as follows:
[0130] The eye camera module collects eye images and transmits the images to the gaze point detection module. The gaze point detection module extracts gaze point information from the collected eye images. Further, the microprocessor module samples and stores the gaze point information. In calibrating the gaze point, the user is required to gaze at 5 to 7 points of the digital information display module. The system will calibrate the gaze point according to the target point position displayed by the digital information display module and the collected gaze point information, and adjust the parameters of the gaze point detection module.
[0131] Step 12: Collect text information and generate interactive objects:
[0132] The system for collecting text information and generating interactive objects is as follows: Figure 6As shown, it includes: an environment camera module, a space positioning module, a text detection module, an interactive object generation module, an interactive object controller module and a microprocessor module.
[0133] The environment camera module is used to collect environment images, including text images and general environment images in the environment.
[0134] The spatial positioning module is used to calculate the spatial coordinates of the head-mounted display and text based on the environmental image.
[0135] The text detection module is used to extract text information from text images and find the coordinates of all space positions in the text.
[0136] The interactive object generation module is used to generate an instance of an interactive object at a specified location.
[0137] The interactive object controller module is used to control the movement of the interactive object and collect the real-time position of the interactive object.
[0138] The microprocessor module contains timers, samplers, and is used to assist the functions of the interactive object controller module.
[0139] Interactive objects refer to objects with interactive functions, such as interface controls, etc. Interactive objects include motion sliders and motion tracks, such as Figure 7 As shown in the figure. The motion slider is the moving target point to be tracked, and the motion track is the path of the motion slider. The motion slider and motion track can be displayed or not and how to display them according to the interface design. The motion parameters of the interactive object include phase, speed, acceleration, trajectory shape, etc.
[0140] The premise of phase is that the motion of the interactive object is periodic motion. Phase is the starting point of the periodic motion, which is equivalent to defining the position of the motion slider with the percentage of the period.
[0141] Speed is the speed at which the motion slider moves, or it can be the angular velocity in circular motion.
[0142] Acceleration is a parameter used when the motion slider is moving at a variable speed.
[0143] The trajectory shape is the shape of the motion track, which can be different shapes such as circle, rectangle, triangle, diamond, zigzag, V-shape, complex polygon, etc.
[0144] The control method of interactive objects should consider the strategy of eliminating motion ambiguity, in other words, avoiding the motion similarity between different moving objects. The control method of interactive objects can adopt the strategy of allowing different interactive objects to have different phases, speeds, accelerations, trajectory shapes and other parameters. The strategies of setting different parameters can be used alone or in combination. In addition, the strategy of eliminating motion ambiguity can be further optimized according to the position of the interactive objects.
[0145] The process of collecting text information and generating interactive objects is as follows: the environment camera module passes the collected general environment image to the space positioning module to obtain the space coordinates of the head-mounted display, and the environment camera module passes the collected text image to the text detection module to obtain text information. The interactive object generation module obtains the position coordinates of all text spaces from the text detection module, and generates instances of interactive objects at each position, and hides and displays these instances. The interactive object controller module and the microprocessor module are used to control the movement of the interactive object and collect the real-time position of the interactive object.
[0146] Step 2: Monitor the gaze point position in real time and activate the interactive area, including:
[0147] Step 21: Identify gaze interaction actions based on gaze point positions.
[0148] The system for identifying gaze interaction actions according to gaze point positions comprises: a gaze point detection module and a gaze recognition module.
[0149] The gaze point detection module is used to provide the gaze point position in real time.
[0150] The gaze recognition module is used to use a gaze recognition algorithm to recognize gaze interaction actions from the time-series gaze point position data and give the gaze point position.
[0151] Gaze refers to the eyes looking at a certain direction or object without moving, indicating that attention is focused on specific visual information. Gaze recognition algorithms include the eye velocity threshold algorithm (I-VT), the eye distance threshold algorithm (I-DT), the eye velocity distance threshold algorithm (I-VDT), etc.
[0152] Step 22: When a gaze action is detected, the interactive area and the interactive objects contained in the interactive area are activated.
[0153] The system of the interactive area activation module includes an interactive object search module and an interactive area generation module.
[0154] The interactive object search module is used to search for multiple interactive objects that are the nearest neighbors based on the gaze point position. The specific search quantity and search distance range are determined based on the text size and design goals. Generally, there are 8 interactive objects in the spaces between the upper and lower lines of text.
[0155] The interactive region generating module is used to generate the outer contours that enclose the interactive objects, namely, the interactive region, according to the interactive objects that are searched.
[0156] The interactive area refers to the interactive space range used to limit the interactive behavior of selecting the target space position, including the activation range, movement range, and interactive object set. The activation range refers to the minimum range that wraps the interactive object, the movement range refers to the movable range that the gaze point can fall on without affecting the activation state of the interactive area. This range should be larger than the activation range, and the interactive object set is the set of interactive objects in the interactive area.
[0157] The process of activating the interactive area according to the gaze point position is that the interactive object search module searches for a specified number of interactive objects within the specified range of the gaze point, and the interactive area generation module activates the interactive area and the interactive objects contained in the interactive area accordingly.
[0158] Step 3: Select the target space position through eye control actions such as gaze and smooth pursuit
[0159] Step 31: Detect eye movement and motion correlation of interactive objects
[0160] The system for detecting the correlation between eye movement and movement of interactive objects comprises: a gaze point detection module, an interactive object controller module, a movement correlation detection module, and an interactive object state control module.
[0161] The gaze point detection module is used to obtain the real-time coordinate points of the gaze.
[0162] The interactive object controller module is the same as the interactive object controller module in step 1, and is used to control the movement of each activated interactive object.
[0163] The motion correlation detection module is used to detect the motion correlation between the temporal gaze point coordinates of the eye and the temporal point coordinates of the moving slider in the interactive object. The motion correlation calculation can use the Pearson correlation coefficient algorithm with delay correction and the Euclidean distance algorithm with delay correction.
[0164] Description of the Pearson correlation coefficient algorithm with delay correction:
[0165] Input: Dataset data containing the x and y coordinates of the focus and interactive object (IO) points; sliding window size window_size; delay time delay_Time (in milliseconds)
[0166] Output: An array of minimum Pearson correlation coefficients for each window considering the delay, correlations
[0167] Here are the steps:
[0168] 1) The sample length of the initialized data set is n_samples.
[0169] 2) Initialize an empty list correlations.
[0170] 3) If delay_Time is equal to 0, set delaySample to 0. Otherwise, calculate delay sample delaySample=calculate_jump_delay_sample(delay_Time).
[0171] 4) For each starting index start from 0 to n_samples-window_size-delaySample+1:
[0172] ① Calculate the end index end = start + window_size.
[0173] ② Calculate the delayed start and end indexes of the focus data: start_dly = start + delaySample, end_dly = end + delaySample.
[0174] ③ Calculate the Pearson correlation coefficient of the delayed focus and the y coordinate of the current IO point: correlation_y.
[0175] ④ Calculate the Pearson correlation coefficient of the delayed focus and the x-coordinate of the current IO point: correlation_x.
[0176] ⑤Find the minimum of the two correlation coefficients: min_correlation.
[0177] ⑥Append min_correlation to the list correlations.
[0178] Returns: Convert the list correlations to an array and return it.
[0179] The algorithm traverses the dataset using a sliding window and introduces delay processing to simulate possible time differences in real situations. It calculates the Pearson correlation coefficient of the x and y coordinates of the focus and IO points after considering the delay, allowing for a more accurate detection of motion similarity. The algorithm finally outputs an array of the minimum Pearson correlation coefficients for each window after considering the delay.
[0180] Description of the Euclidean distance algorithm with delay correction:
[0181] Input: data set data containing focus and IO point coordinates; sliding window size window_size; delay time delay_Time (in milliseconds)
[0182] Output: The concentrated Euclidean distance array distances calculated for each window taking into account the delay
[0183] Here are the steps:
[0184] 1) The sample length of the initialized data set is n_samples.
[0185] 2) Initialize an empty list distances.
[0186] 3) If delay_Time is equal to 0, set delaySample to 0. Otherwise, calculate the number of delay samples according to delay_Time: delaySample=calculate_jump_delay_sample(delay_Time).
[0187] 4) For each starting index start from 0 to n_samples-window_size-delaySample+1:
[0188] ① Calculate the end index end = start + window_size.
[0189] ② Calculate the start and end indexes of the delay: start_dly = start + delaySample, end_dly = end + delaySample.
[0190] ③ Consider the delayed extraction of window data, which are the x and y coordinates of the focus and IO points: focus_data_x, focus_data_y of the delayed window, and io_data_x, io_data_y of the original window.
[0191] ④ Center the data by subtracting the mean: focus_centered_x, focus_centered_y, io_centered_x, io_centered_y.
[0192] ⑤ Calculate the Euclidean distance between the x-coordinate and y-coordinate of the focal point and the IO point after concentration: distance_x, distance_y.
[0193] ⑥ Combine the distances of x and y to calculate the total distance of the window and normalize it: distance.
[0194] ⑦Append the calculated distance to the list distances.
[0195] Returns: Convert the list distances to an array and return it.
[0196] This algorithm introduces delay processing in the process of sliding window traversal of the data set to simulate the time difference that may exist in actual situations. It first calculates the number of delayed samples based on the delay time, and then extracts the coordinate data of the focal point and IO point considering the delay for each window. Next, the data is concentrated by subtracting the average value, and the Euclidean distance between the concentrated focal point and IO point is calculated. Finally, the algorithm outputs the concentrated Euclidean distance array for each window considering the delay.
[0197] The interactive object state control module is used to control the state performance of the interactive object, such as the color of the motion slider, to inform the user of the current state of the interactive object. The interactive object states include resting state, hovering state, tracking state, tracking hovering state, locked state, and triggered state. The resting state means that the interactive object is neither gazed at nor tracked. The hovering state means that the interactive object is gazed at. The tracking state means that the interactive object is tracked. The tracking hovering state means that the interactive object is both tracked and gazed at. The locked state means that the tracking and gaze have reached the locked preset, and the state of the interactive object is locked. The triggered state means that the selection instruction is received and the interactive object is triggered.
[0198] The detection process of the motion correlation between eye movement and interactive objects is as follows: the gaze point detection module transmits the real-time gaze point to the motion correlation detection module, the motion correlation detection module calculates the motion correlation between each interactive object and the eye movement coordinates, and the interactive object state control module adjusts the real-time state of the interactive object according to the motion correlation.
[0199] Step 32: Lock the state of the interactive object according to the motion correlation value of the interactive object
[0200] The purpose of locking the interactive object state is to give the interactive action of selecting the target space time to react to the operation, thereby improving the accuracy of the operation. The interactive object state locking module can set a threshold based on the movement correlation to achieve the state of pre-setting the locked interactive object, and set the state unlocking time. When the unlocking time limit is reached, the locked state of the interactive object is unlocked.
[0201] Step 33: Select the target space position by pressing keys or other commands
[0202] The system for selecting the target space position includes: a selection instruction monitoring module and an interactive object state control module. The selection instruction can be an interactive action of different channels such as key operation, blinking, foot pedaling, teeth clicking, voice, etc. The selection instruction monitoring module is used to monitor the interactive action at all times. When the interactive object is in a locked state and a selection instruction is heard, the interactive object will be changed to a triggered state, that is, the target space position is selected. The interactive object state control module is responsible for controlling the process of changing the interactive object to a triggered state.
[0203] Step 4: Plot the target text
[0204] Marking the target text is achieved through the text marking control module.
[0205] Before marking the target text, the text marking control module will determine whether two spaces have been selected. If only one is selected, the interactive area is activated in step 2 and a small mark is added to the first selected target space. If two spaces are selected, the target text is marked.
[0206] The text marking control module may mark the target text by adding a line between the front and back positions of the target space. Specifically, it may be by adding a marking object or by adding an HTML tag to achieve the marking of the target text.
[0207] In order to describe the interaction process more clearly, a flowchart of the key steps is drawn, such as Figure 2 shown.
[0208] The interactive object generation control corresponds to step 1. The gray object in the text space in the figure is the generated interactive object, which is not displayed when the actual system is working.
[0209] The region activation corresponds to step 2. The red dashed box in the figure is the movement range of the activated interaction area, which is larger than the activation range and is the activity range of the gaze point. The blue circular pattern in the figure is the gaze point, which may not be displayed when the actual system is working.
[0210] The motion correlation detection corresponds to step 31. The red value above the interactive object in the figure is the value of the motion correlation. The system will calculate the motion correlation of each interactive object, but when the actual system is working, this value may not be displayed.
[0211] The interaction intention locking corresponds to step 32. The interactive object that turns yellow in the figure is the locked interactive object. When the interaction intention of the interactive object decreases below the threshold, the interactive object will return to the gray state.
[0212] The text plotting corresponds to step 4. The yellow semi-transparent rectangle in the figure is the marker for plotting text, and the text "It is very" is plotted.
[0213] In order to verify the system and method, a verification experiment of the text marking system was conducted to compare the performance of the eye-touch interaction method based on smooth tracking (SP) and the baseline method (BL) in different text marking tasks. The main purpose is to evaluate the performance of the system on the HoloLens 2 device for text marking tasks, with a special focus on accuracy, interaction efficiency, and cognitive load. By comparing the experimental results, this section will obtain a comprehensive evaluation of the system effect and applicability of the design and development, and provide guidance for further optimization of the text marking system.
[0214] The verification experiment sets two interactive modes, including the text marking method based on smooth tracking and the baseline method. The text marking method based on smooth tracking is a new interactive method proposed in this paper, which is written as SPMark or SP below. The baseline method is a marking method that uses eye movement points as a brush, which is written as BL below. The interactive process is as follows Figure 8 shown.
[0215] (1) Interaction process of SPMark method
[0216] The first step is to gaze at the header of the target text and activate the interactive objects in the area near the gaze point; track the movement slider on the interactive object, or gaze at the interactive object until the vertical line of the interactive object turns yellow.
[0217] In the second step, press the "down" key on the keyboard to select the locked yellow cursor, and a yellow vertical line will be placed at the cursor position as the first mark symbol of the target text to be selected.
[0218] The third step is to gaze at the end of the target text to activate the interactive object in the area near the gaze point; track the movement slider on the interactive object, or gaze at the interactive object until the vertical line of the interactive object turns yellow.
[0219] Step 4. Press the “down” key on the keyboard to select the locked yellow cursor and the target text will be marked.
[0220] Step 5. Press the "right" key to go to the next task.
[0221] (2) Interaction process of the baseline method
[0222] The first step is to stare at the beginning of the text and press the "down" key on the keyboard to start marking.
[0223] In the second step, the route is highlighted as the eyes scan it.
[0224] Step 3: Press the “down” key on the keyboard to end the plotting.
[0225] Step 4. Press the "right" key to go to the next task.
[0226] Experimental Procedure
[0227] Before the experiment began, participants received a brief introduction to the experimental objectives and procedures. Participants were required to sign a consent form to participate in the experiment and fill out a demographic questionnaire (such as age, gender, and virtual reality experience). They were then required to complete an eye movement calibration to ensure that the system could accurately track the participants' eye movements. Participants completed at least 6 training trials for SPMark and baseline respectively before the formal experiment began to familiarize themselves with the experimental process. The experimental tasks and interaction methods in the training phase were consistent with the formal experiment, but the experimental materials and the formal experiment materials were not repeated.
[0228] The experiment was divided into 2 parts, each with 4 phases and 6 trials. The two parts were the SPMark interaction experiment and the baseline interaction experiment. The order effect of the interaction was balanced by having half of the subjects take the SPMark part first and then the baseline part, and the other half take the baseline part first and then the SPMark part. Each phase was a repetition of the experimental conditions, including all combinations of the levels of the two independent variables, and the order of the combinations was randomized. A two-minute break was allowed after each phase. The following is an example of a single trial.
[0229] In the first step, a “+” sign appeared in the center of the screen. Participants were asked to fixate on the plus sign for 1000 milliseconds until it disappeared.
[0230] In the second step, the experimental operation window and experimental task information appeared on the screen. Participants browsed the experimental task information and completed the text plotting task in the experimental operation window by eye control and pressing the "down arrow" key on the keyboard. If the marking was wrong, the marking could be cleared at any time by pressing the "left arrow" key. Participants were asked to complete the highlighting task as quickly and accurately as possible.
[0231] In the third step, when participants thought they had completed the plotting task correctly, they could press the “right direction” key to move to the next task.
[0232] Step 4: At the end of the task, the screen was blank for 200 milliseconds, and the operation window and task information window did not display text. Then, the next task started until the entire experiment was completed.
[0233] The experimental steps for a single trial are as follows: Fig. 9 After completing each part of the experiment, participants filled out questionnaires on the interaction style (NASA-TLX and UEQ). After completing the experiment, participants received a structured interview. The entire experiment lasted about 30-40 minutes for each participant.
[0234] The experiment was conducted in the Human Factors Engineering Laboratory of the School of Mechanical Engineering of Southeast University. The laboratory controlled the ambient light, with moderate brightness and darkness. The ambient light at different times was close, which also reduced the rainbow effect of the HoloLens 2 device lens. The laboratory was a separate space with a quiet environment. The room temperature was kept at around 20 degrees Celsius, creating a good experimental environment for the subjects.
[0235] The pwr package in R language was used to analyze the sample size required for the study. The medium effect size f=0.25, the statistical test power 1-β=0.95, and the significance level ∝=0.05 were pre-set. The results showed that at least 18 subjects were needed for each group. In order to more clearly study the impact of the lock-in time independent variable on the experimental results, a total of 21 subjects were invited to this experiment. All the subjects were graduate students or undergraduates from the School of Mechanical Engineering of Southeast University, including 12 male subjects and 9 female subjects. The 21 subjects were all between 20 and 28 years old. All the personnel had good eyesight and no physical factors such as color blindness and color weakness that could easily cause reading difficulties. Before the experiment, it was confirmed that the subjects had had enough rest yesterday and had no eye fatigue or discomfort.
[0236] Experimental design
[0237] This experiment used a 2×2×3 within-subject design: two interaction conditions (SPMark and baseline), two text sizes (1.8°, 1.05°), and three subtasks (annotating words, sentences, and multiple sentences), and each subtask condition combination was performed four times. For ease of representation, the two levels of text size correspond to lowercase letters l and m, and the three levels of subtasks, i.e., text length, correspond to uppercase letters L, M, and S. These tasks reflect typical annotation activities on paper documents. The baseline is a direct plot of eye movement saccadic paths. Both SPMark and baseline use the same device for eye-touch interaction and text presentation. To reduce learning and order effects, this study used a random subtask order within the group and balanced task performance by alternating conditions. To reduce the learning effect caused by repeated reading of the same content, the materials of the experimental tasks were not repeated. There were a total of 2×2×3×4×20=960 data points, and a single experimental condition had 20×4=80 experimental data points.
[0238] Experimental materials
[0239] The text materials used in the experiment come from a standardized English reading assessment corpus. To ensure randomness, text segments of appropriate length were randomly selected from the corpus and then randomly sorted. Each text segment is between 315 and 480 characters long, which ensures that the length is between 9 and 12 lines when the text is presented at a 1.8° viewing angle.
[0240] The target selection texts of the experimental task include three lengths: short text is one word; medium text is one sentence, and when the text is presented at a 1.8° visual angle, the length of a sentence is 2-3 lines; long text is 2-3 sentences, and when the text is presented at a 1.8° visual angle, the length of the long text is 6-8 lines.
[0241] The 1.8° viewing angle for text size is derived from the research on text selection in virtual reality and augmented reality by Xu, W. and Liu, X., while the 1.05° viewing angle is derived from the typographic specifications of Apple's Human-Computer Interaction Guide. The specifications state that the text size of large titles in macOS is 26pt and the line height is 32pt. Here, the distance between the human eye and the screen is assumed to be 500mm, and the viewing angle is converted to 1.05° according to the conversion formula from pt to mm to viewing angle.
[0242] In the style parameters of text display, when the text is presented at a 1.8° visual angle, the number of characters per line is between 35 and 40 (including symbols and spaces) to improve reading efficiency. The horizontal size of the text paragraph is within 28.1° visual angle. Considering the consistency of the horizontal size of the control text, the number of characters per line of small-size text (1.05°) is limited to between 50 and 65. The line spacing is set to 0.5 times to avoid overlap when smoothly tracking the movement of the control. The character spacing and word spacing are the default values. The font used is Arial, which is a common sans-serif font that is versatile and easy to read. The text alignment is left-aligned, which ensures the consistency of the size of the space characters.
[0243] The text interaction window was 500mm away from the participants' eyes, and the test participants confirmed that they could read the text clearly without any difficulty. The 500mm setting was to simulate the distance of daily use of computer monitors. Placing the interactive panel at this distance is also conducive to hand-poking interaction of digital information. The interactive window uses a dark background to form a strong contrast with the text, thereby improving the reading experience.
[0244] The experimental interface consists of three parts: Fig.10 As shown. The box in the front is the operation area, which presents the text that can be plotted. The red "+" sign in the middle only appears at the beginning of the trial. The box on the left is the task prompt area with a 20° inclination. This part presents the text content to be plotted in each trial. The text content of the operation area and the task prompt area is exactly the same, and the layout is aligned. There are two operation buttons on the right side of the operation area, from top to bottom, they are the clear plotting button and the next task button. The functions of these two buttons are the same as those of the "left direction" and "right direction" buttons on the keyboard.
[0245] Experimental data analysis methods
[0246] The indicators of the evaluation system include efficiency, accuracy, cognitive load, and user experience. The indicator of efficiency is the task completion time. Specifically, the task completion time is the time from the presentation of the text to be annotated to the end of clicking the "right arrow" after the text is marked in the operation area. The indicator of accuracy is the correctness of task completion. The accuracy of the SPMark interaction method can be automatically obtained by selecting the result. The accuracy of the baseline method needs to be determined by significant user errors. This is determined by the researcher using the {0,0.5,1} scoring standard after the experiment. Cognitive load is obtained through the NASA-TLX questionnaire. User experience is obtained through the User Experience Questionnaire (UEQ). In addition, the behavior of the participants and the content of the interview are recorded.
[0247] Data analysis methods will include objective data accuracy (task completion rate), interaction efficiency (task completion time), and subjective data cognitive load and user experience. To ensure data accuracy, this study sets the task completion time to be calculated only for all samples that have not been modified and correctly completed the experimental task. In the data processing of task completion time, outlier processing measures were taken, including using the task completion time indicator to remove samples exceeding 3σ, and removing samples in the experimental blocks with task completion rates below 0.5 due to abnormal eye movement calibration. Data analysis will use statistical tests such as one-way analysis of variance and Bonferroni-corrected post hoc multiple comparisons to compare performance differences between different interaction methods and performance differences between different text tasks.
[0248] Experimental Results
[0249] In terms of task completion rate, under most task conditions, there is no significant difference in task completion rate between different interaction methods, such as Fig.11 This suggests that under different conditions considering text length and size, there may be no significant difference between the baseline method and the smooth tracking method in affecting users' task completion rate.
[0250] In terms of the task completion time indicator, Bonferroni-corrected post hoc multiple comparisons of paired samples found that under the conditions of long text large font size (L_l) and long text medium font size (L_m), there were significant differences in task completion time between the baseline interaction mode (BL) and smooth pursuit (SP), and the task completion time of SP was significantly shorter than that of BL (p<0.001, p<0.05). However, under the condition of Chinese text large font size (M_l), there was no significant difference in task completion time between BL and SP. Under the conditions of Chinese text medium font size (M_m), short text large font size (S_l), and short text medium font size (S_m), there were significant differences in task completion time between the baseline interaction mode (BL) and smooth pursuit (SP), and the task completion time of SP was significantly longer than that of BL (p<0.01, p<0.001, p<0.001). The comprehensive results are shown in Fig.12 In the figure, *** indicates p<0.001, ** indicates p<0.01, and * indicates p<0.05. These results show that the interaction method has a significant effect on the task completion time under different task conditions. In the long text plotting task, the smooth tracking method significantly shortened the task completion time compared with the baseline method.
[0251] In the cognitive load index, the scores of each sub-attribute of the cognitive load questionnaire were converted to a percentage system, and then a one-way ANOVA and Bonferroni-corrected post hoc multiple comparisons were used. The results are shown in Fig.13 The results showed that the smooth pursuit method (SP) was significantly lower than the baseline method (BL) in terms of mental demand, physical burden, and time demand (p<0.05). The smooth pursuit method (SP) was significantly higher than the baseline method (BL) in terms of task performance (p<0.05). There was no significant difference in the effort level and frustration index.
[0252] In the user experience index, the Short_UEQ_Data_Analysis_Tool was used to analyze the user experience questionnaire results to obtain the scores of the three sub-attributes of pragmatic quality, hedonic quality, and overall score. One-way ANOVA and Bonferroni-corrected post hoc multiple comparisons were used to analyze the results. Fig.14 The results showed that in the three attributes of pragmatic quality, hedonic quality, and overall score, the smooth pursuit method (SP) was significantly higher than the baseline method (BL) (p<0.05). The pragmatic quality of the smooth pursuit method (SP) was in the above-average range, the hedonic quality was in the excellent range, and the overall score was in the good range, such as Fig.15 shown.
Claims
1. A text plotting system based on smooth pursuit hybrid digital paper, characterized in that: include: The subsystem for collecting eye movement data and calibrating gaze point parameters, the subsystem for collecting text information and generating interactive objects, the subsystem for detecting the correlation between eye movement and interactive object movement, the subsystem for identifying gaze interactive actions based on gaze point position, the subsystem for activating the interactive area module, and the subsystem for selecting the target space position; The subsystem for collecting eye movement data and calibrating gaze point parameters is used to collect eye images and extract gaze point information from the collected eye images; A subsystem for collecting text information and generating interactive objects, which is used to collect and recognize text information, and generate interactive objects based on the recognized text information; A subsystem for detecting the correlation between eye movement and interactive object movement, used for calculating the movement correlation between the eye movement timing data obtained from the subsystem for collecting eye movement data and calibrating gaze point parameters and the movement timing data of each interactive object in the interactive object set obtained from the subsystem for collecting text information and generating interactive objects; A gaze interaction action recognition subsystem based on gaze point position is used to extract gaze interaction actions from the time-series gaze point data obtained by the eye movement data collection and gaze point parameter calibration subsystem; The interactive area activation module subsystem is used to generate an interactive area and activate interactive objects in the interactive area when the gaze interaction action recognition subsystem based on the gaze point position recognizes the gaze interaction action; The target space position subsystem is selected to determine the interaction instruction based on the motion correlation data of the eye movement detection and interaction object motion correlation subsystem, change the state of the interaction object, and plot the target text.
2. The text plotting system based on smooth tracking hybrid digital paper according to claim 1, characterized in that: A subsystem for collecting eye movement data and calibrating gaze point parameters includes: an eye camera module, a digital information display module, a gaze point detection module and a first microprocessor module; An eye camera module, used for collecting eye images; A gaze point detection module, used to extract gaze point information from the eye image collected by the eye camera module; A digital information display module, used for presenting a calibration target when calibrating gaze point parameters; The first microprocessor module is used to store, sample and output the gaze point information extracted by the gaze point detection module.
3. The text plotting system based on smooth tracking hybrid digital paper according to claim 1, characterized in that: A subsystem for collecting text information and generating interactive objects includes: an environment camera module, a space positioning module, a text detection module, an interactive object generation module, an interactive object controller module and a second microprocessor module; An environment camera module is used to collect environment images, including text images in the environment and surrounding environment images; A spatial positioning module, used to calculate the spatial coordinates of the head-mounted display and text based on the environment image; The text detection module is used to extract text information from text images and find the coordinates of all space positions in the text; An interactive object generation module, used to generate an instance of an interactive object at a specified location; A first interactive object controller module, used to control the movement of the interactive object and collect the real-time position of the interactive object; The second microprocessor module includes a timer and a sampler, and is used to assist the functions of the interactive object controller module.
4. The text plotting system based on smooth tracking hybrid digital paper according to claim 1, characterized in that: A subsystem for detecting the correlation between eye movement and interactive object movement, including: a gaze point detection module, an interactive object controller module, a movement correlation detection module, an interactive object state control module, and an interactive object state locking module; A gaze point detection module is used to obtain the real-time coordinate points of gaze; a second interactive object controller module for controlling the movement of each activated interactive object; A motion correlation detection module, used to detect the motion correlation between the temporal gaze point coordinates of the eye and the temporal point coordinates of the moving slider in the interactive object; An interactive object state control module is used to control the state performance of the interactive object; The interactive object state locking module sets a threshold value according to the motion correlation, locks the state of the interactive object when it is reached, and sets a state unlocking time. When the unlocking time limit is reached, the locked state of the interactive object is unlocked.
5. The text plotting system based on smooth tracking hybrid digital paper according to claim 1, characterized in that: A gaze interaction action subsystem is provided for recognizing gaze point positions according to gaze point positions, including: a gaze point detection module and a gaze recognition module; A gaze point detection module, used to provide the gaze point position in real time; A gaze recognition module is used to recognize gaze interaction actions from the time-series gaze point position data and give the gaze point position; The interactive area activation module subsystem includes an interactive object search module and an interactive area generation module; An interactive object search module, used to search for multiple nearest interactive objects according to the gaze point position; The interactive area generating module is used to generate an interactive area that encloses the outer contour of the searched interactive object according to the searched interactive object.
6. The text plotting system based on smooth tracking hybrid digital paper according to claim 1, characterized in that: The target space position selection subsystem includes: a selection instruction monitoring module, an interactive object state control module, and a text book marking control module; Select the command monitoring module to monitor interactive actions at all times; An interactive object state control module, used to control the interactive object to change from a locked state to a triggered state; The text marking control module is used to mark the target text.
7. The text marking method of the text marking system based on smooth tracking hybrid digital paper according to any one of claims 1 to 6, characterized in that: The following steps are involved: Step 1: Based on the subsystem for collecting text information and generating interactive objects and the subsystem for collecting eye movement data and calibrating gaze point parameters, text recognition is performed and interactive objects are prefabricated; Step 2: Based on the gaze point position, the gaze interaction action subsystem and the interaction area activation module subsystem are identified to monitor the gaze point position in real time and activate the interaction area; Step 3: Based on the subsystem for detecting the correlation between eye movement and interactive object movement and the subsystem for selecting the target space position, the target space position is selected through eye control actions; Step 4: Based on the selected target space position subsystem, plot the target text.
8. The text plotting method according to claim 7, characterized in that: Step 1 includes: Step 11: Collect eye movement data and calibrate gaze point parameters: Step 12: Collect text information and generate interactive objects.
9. The text plotting method according to claim 7, characterized in that: Step 2 includes: Step 21: Identify gaze interaction actions according to gaze point positions; Step 22: When a gaze action is detected, the interactive area and the interactive objects contained in the interactive area are activated.
10. The text plotting method according to claim 7, characterized in that: Step 3 includes: Step 31: Detect the correlation between eye movement and movement of the interactive object; Step 32: Locking the state of the interactive object according to the motion correlation value of the interactive object; Step 33: Select the target space position by pressing keys or other commands.
