Eye tracking cross-device interaction method and apparatus

Abstract

The application provides an eye tracking cross-device interaction method and device, and relates to the technical field of human-computer interaction. The method comprises: performing eye tracking recognition on eye movement images, combining head posture data to output a line of sight direction angle, a gaze point coordinate and a gaze duration, and forming an eye tracking feature sequence; combining camera pose parameters, field of view overlap region parameters and screen layout geometric relationship parameters from multiple devices, constructing a three-dimensional gaze prediction model based on the eye tracking feature sequence, and predicting a target screen or interface element of user gaze in a multi-device collaborative scene; generating a gaze intention field as an interaction scheduling basis based on a prediction result output by the three-dimensional gaze prediction model, dynamically scheduling task resources across devices under a distributed architecture of the Hongmeng, and realizing content projection, task continuation and interface synchronization between different devices. The application can break through the limitations of two-dimensional plane recognition of gaze points, and extend gaze prediction to a spatial scene of cross-device interaction.

Description

Technical Field

[0001] This invention relates to the technical field of human-computer interaction, and specifically to an eye-tracking cross-device interaction method and apparatus. Background Technology

[0002] With the continuous improvement of the distributed capabilities of the HarmonyOS system, seamless collaboration between multiple devices has gradually become an important direction for upgrading user experience. Eye tracking, as a natural human-computer interaction method, has been applied in single-device scenarios such as reading assistance, screen-off wake-up, and gaze input in recent years. However, most existing research is limited to the interaction control of a single terminal, lacking exploration of unified analysis of user gaze intent and task collaborative scheduling in a multi-terminal distributed environment. Therefore, how to deeply integrate eye tracking technology with distributed architecture to improve cross-device interactive experience has become a research and application topic worthy of attention.

[0003] Traditional single-device eye-tracking methods typically identify gaze points only within a two-dimensional plane, limiting prediction results to the coordinate system of a single screen and failing to accurately represent the user's true gaze intent in multi-device environments. This limitation prevents gaze prediction from transcending single-screen boundaries and hinders effective modeling of spatial relationships and interaction needs across devices, thus restricting the potential of eye-tracking in distributed interaction.

[0004] Therefore, a method is needed to overcome the limitations of recognizing gaze points in two-dimensional planes and extend gaze prediction to spatial scenarios with cross-device interaction. Summary of the Invention

[0005] This invention provides an eye-tracking cross-device interaction method and apparatus that can overcome the limitations of two-dimensional planar recognition of gaze points, enabling gaze prediction to be extended to spatial scenarios of cross-device interaction.

[0006] A first aspect of the present invention provides an eye-tracking cross-device interaction method, the method comprising: Collect user's eye-tracking images and head posture data; The eye movement images are used to identify eye trajectory, and the head posture data is combined to output the gaze direction angle, gaze point coordinates and gaze duration to form an eye trajectory feature sequence. By combining camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices, a three-dimensional gaze prediction model is constructed based on the eye trajectory feature sequence to predict the target screen or interface element that the user is looking at in a multi-device collaborative scenario. A gaze intent field is generated based on the prediction results output by the three-dimensional gaze prediction model; Using the gaze intent field as the basis for interaction scheduling, cross-device task resources are dynamically scheduled under the HarmonyOS distributed architecture to achieve content projection, task continuation and interface synchronization between different devices.

[0007] Based on the above technical solutions, preferably, the step of using the gaze intent field as the basis for interaction scheduling to dynamically schedule cross-device task resources under the HarmonyOS distributed architecture, thereby realizing content projection, task continuation, and interface synchronization between different devices, further includes: The smooth gaze intensity of each device within a continuous time window is calculated based on the gaze intention field; Calculate the corresponding expected gaze duration based on the smoothed gaze intensity; By combining the desired gaze duration with head movement stability parameters and device-related weights, an adaptive preset threshold is output. Using the desired gaze duration and the adaptive preset threshold as input, perform cross-device comparisons and output candidate focus devices; The candidate focus device outputs a stable focus device identifier based on the finite state machine that performs focus holding and release; Using the focus device identifier as input, a distributed focus token is generated and published to drive content projection, task continuation, and interface synchronization. Thus, when the user's gaze duration on the first device exceeds the adaptive preset threshold, the first device is set as the focus device, and the stability of the cross-device interaction link is maintained.

[0008] Based on the above technical solutions, preferably, after setting the first device as the focus device, the method further includes: When the user's eye movement is detected to shift from the first device to the second device, a task is triggered to flow across the devices between the first and second devices, and a synchronous update is performed at the interface element level to form an eye-tracking-driven cross-device interaction link.

[0009] Based on the above technical solutions, preferably, the step of performing eye trajectory recognition on the eye movement image and combining the head posture data to output the gaze direction angle, fixation point coordinates, and gaze duration to form an eye trajectory feature sequence specifically includes: Perform pupil center localization and eyelid detection on the eye movement image and head posture parameters, and output the pixel coordinates of the pupil center; Using the pupil center pixel coordinates and the camera intrinsic parameter matrix as input, calculate the first unit gaze vector in the camera coordinate system, and combine the head pose parameters to map the first unit gaze vector to the head coordinate system, and output the second unit gaze vector in the head coordinate system. The optical axis is output based on the second unit line-of-sight vector and the eye geometry model, and corrected to the visual axis based on the individualized Kappa angle, thus obtaining the unit direction vector of the visual axis in the world coordinate system. Using the unit direction vector of the visual axis and the equation of the device screen plane as input, the intersection of the visual axis and the screen plane is solved and mapped to the gaze point coordinates in the screen coordinate system; The angular velocity between adjacent frames is calculated based on the time series of the unit direction vector of the visual axis, and gaze and saccade segmentation is performed in combination with a preset angular velocity threshold to obtain the corresponding gaze duration. Based on the gaze direction angle, the coordinates of the gaze point, and the gaze duration, an eye trajectory feature sequence is constructed and output.

[0010] Based on the above technical solutions, preferably, the step of constructing a three-dimensional gaze prediction model based on the eye trajectory feature sequence by combining camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices specifically includes: By combining the eye trajectory feature sequence with the camera pose parameters of each device, a set of gaze rays in the world coordinate system is established; Based on the geometric relationship parameters between the set of gaze rays and the screen layout of each device, the intersection of the rays and the screen plane is calculated to obtain candidate gaze points. Based on the candidate gaze points and the corresponding screen boundaries, the geometric consistency error is calculated and converted into device-level likelihood data; For the device-level likelihood data and the field-of-view overlap region parameters between each device, construct a cross-device transfer prior and perform probability recursion to output the posterior probability distribution of each device; Using the posterior probability distribution and the geometric description of the internal interface elements of the device as input, the posterior probability at the interface element level is further calculated to obtain the prediction result for the target screen or the interface element.

[0011] Based on the above technical solutions, preferably, the step of generating a gaze intent field based on the prediction results output by the three-dimensional gaze prediction model specifically includes: The device posterior probability distribution of the target screen and the posterior probability of the interface elements are jointly represented to form a multi-level gaze probability set. Based on the multi-level gaze probability set and the spatial location of the candidate gaze points, combined with the screen layout geometry of each device, the discretized gaze probability is mapped to the potential gaze distribution in continuous space. A Gaussian kernel expansion operation is performed on the potential gaze distribution to form a smooth gaze potential in the spatial range; The gaze probability distribution at the device layer and the gaze probability distribution at the interface element layer are fused together by weighted superposition to generate a gaze intent field that describes the user's gaze trend.

[0012] Based on the above technical solutions, preferably, the step of using the gaze intent field as the basis for interaction scheduling, and dynamically scheduling cross-device task resources under the HarmonyOS distributed architecture to achieve content projection, task continuation, and interface consistency between different devices, specifically includes: The gaze intention field is integrated over the effective display area of ​​each device screen to obtain the corresponding device-level gaze potential energy, which is then normalized to form a device-level gaze potential energy distribution. Based on the device-level gaze potential distribution, a device-level scheduling score is generated by combining device relevance weights, task affinity values, and migration costs. Based on the device-level scheduling score, a cross-device resource allocation model is constructed to complete the mapping between tasks and devices under computational resource constraints and bandwidth constraints, and the task allocation results are output. Based on the task allocation result, a distributed focus token and execution instruction are generated. The distributed focus token includes a focus device identifier, a focus establishment timestamp, a task identifier, and a task status summary. After the distributed focus token is issued, the execution chain of content projection, task continuation and interface synchronization is driven, and the stability of the focus device is maintained through time smoothing mechanism and advantageous interval constraint.

[0013] In a second aspect of the invention, an eye-tracking cross-device interaction apparatus is provided, the apparatus being used to perform an eye-tracking cross-device interaction method as described in any of the foregoing embodiments, the apparatus comprising an acquisition module, a processing module, and an output module, wherein: The acquisition module is used to collect the user's eye movement images and head posture data; The processing module is used to perform eye trajectory recognition on the eye movement image, and combine the head posture data to output the gaze direction angle, gaze point coordinates and gaze duration to form an eye trajectory feature sequence. The processing module is used to combine camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices to construct a three-dimensional gaze prediction model based on the eye trajectory feature sequence, so as to predict the target screen or interface element that the user is looking at in a multi-device collaborative scenario. The processing module is used to generate a gaze intent field based on the prediction results output by the three-dimensional gaze prediction model. The output module is used to use the gaze intent field as the basis for interaction scheduling, and dynamically schedule cross-device task resources under the HarmonyOS distributed architecture to realize content projection, task continuation and interface synchronization between different devices.

[0014] In a third aspect of the invention, an electronic device is provided, including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the preceding embodiments.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium storing instructions that, when executed, perform the method as described in any of the preceding claims.

[0016] In summary, one or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: 1. This invention introduces camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices into a unified world coordinate system. It is no longer limited to two-dimensional gaze point recognition in a single-device screen coordinate system. Instead, it maps the eye trajectory feature sequence into a three-dimensional gaze ray set across devices. By combining the intersection relationship between the rays and the screen plane and the spatial constraints between multiple devices, a three-dimensional gaze prediction model is further constructed. This enables spatial-level analysis of the user's gaze target across different devices, allowing gaze prediction to break through the two-dimensional boundary of a single screen and extend to cross-device interactive scenarios with multi-terminal collaboration.

[0017] 2. Based on the gaze intention field, a joint determination mechanism of smooth gaze intensity, expected gaze duration, adaptive preset threshold and finite state machine is introduced to make the selection of focus device more in line with the user's real gaze habits, avoid misjudgment due to instantaneous eye movement, and thus achieve stable confirmation of focus device and continuous reliability of cross-device interaction link.

[0018] 3. When the user's gaze trajectory shifts from the first device to the second device, the task is automatically transferred across devices and the interface elements are updated synchronously. This allows for seamless switching during task execution, ensuring interface consistency and task state continuity, thereby achieving a truly cross-device interactive experience driven by eye movements.

[0019] 4. By combining gaze ray set, intersection of ray and screen plane, geometric consistency error calculation and cross-device probability recursion, the user's eye trajectory feature sequence is extended to a multi-device environment, realizing probabilistic prediction of target screen or interface elements, thereby breaking through the single-device limitation and ensuring the accuracy of gaze prediction in cross-device scenarios.

[0020] 5. The gaze probabilities of the device layer and the interface element layer are combined in multiple levels, and the probability distribution is smoothed by Gaussian kernel expansion. Finally, a gaze intent field is generated, which transforms the discrete prediction results into continuous spatial potential, providing an interpretable and stable expression of gaze trend for interactive scheduling, thereby improving the robustness of cross-device gaze-driven interaction.

[0021] 6. Based on the gaze intent field, the device-level gaze potential is calculated, and a scheduling score is formed by combining the task affinity value, device relevance weight, and migration cost. Then, distributed focus tokens are used to drive task allocation and execution, ensuring the stability and consistency of dynamic scheduling of content projection, task continuation, and interface synchronization, thereby achieving efficient utilization of cross-device resources and robust operation of the interaction link. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an eye-tracking cross-device interaction method disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of a module of an eye-tracking cross-device interaction device disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention.

[0023] Explanation of reference numerals in the attached drawings: 201, acquisition module; 202, processing module; 203, output module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] In the description of the embodiments of the present invention, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0026] In the description of the embodiments of the present invention, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0027] Existing eye-tracking methods are mostly limited to two-dimensional plane gaze point recognition on a single terminal, which cannot break through the boundaries of a single screen. They are also difficult to uniformly interpret user gaze intent and achieve task collaborative scheduling in a multi-terminal distributed environment, thus limiting the improvement of cross-device interaction experience. Therefore, there is an urgent need for a technical solution that combines the distributed capabilities of the HarmonyOS system to extend eye tracking to multi-device spatial scenarios in order to achieve seamless collaborative human-computer interaction.

[0028] This embodiment discloses an eye-tracking cross-device interaction method, referring to... Figure 1 This includes the following steps S110-S150: S110 collects the user's eye movement images and head posture data.

[0029] This invention discloses an eye-tracking cross-device interaction method applied to a server. The server includes, but is not limited to, electronic devices such as mobile phones, tablets, wearable devices, and PCs (Personal Computers), and can also be a backend server running an eye-tracking cross-device interaction method. The server can be implemented using a standalone server or a server cluster composed of multiple servers.

[0030] A unified time base is established under the HarmonyOS distributed soft bus. The master clock is used to perform PTP-level time synchronization on the slave clocks of each device, outputting a global timestamp sequence. Synchronization trigger links are established for the camera, infrared illumination, inertial measurement unit and depth sensor. Fixed frame rate, fixed exposure and infrared illumination intensity curves are set for each video stream to form a sampling sequence that corresponds one-to-one with timestamp and image frame, which serves as the starting point for subsequent geometric and temporal processing.

[0031] The joint calibration of multiple cameras and inertial measurement units is completed, and the intrinsic parameter matrix, distortion parameters and extrinsic pose of each camera are obtained. The rigid body transformation from the inertial measurement unit to the reference camera is obtained. A projection link from world coordinates to camera coordinates is established for each device, thereby forming a stitchable global geometric reference among multiple devices, so that subsequent head posture parameters can be expressed in the same reference system.

[0032] Steady-state imaging quality control is performed on the original video stream. Automatic gain upper limit and infrared supplementary light closed loop are used to ensure the dynamic range of pupil edge contrast and corneal highlights. Pixel-level resampling of line delay is performed in combination with rolling shutter correction parameters to reduce motion artifacts from interfering with subsequent pupil and iris edges, and output a steady-state eye-tracking image sequence.

[0033] Based on stable tracking of face bounding boxes, key point detection and affine correction are used to crop the regions of interest for both eyes, and normalized subframe sequences for the left and right eyes are constructed. Reflection specular suppression and iris enhancement filtering are performed on the subframe sequences, and median filtering and guided filtering are combined to remove salt-and-pepper noise and weak texture noise, resulting in a high signal-to-noise ratio eye-tracking image sequence suitable for geometric solutions.

[0034] The accelerometer and gyroscope outputs are combined, and short-term attitude increments are obtained using quaternion integration. Then, visual-inertial alignment is used to bind these short-term increments to the camera coordinate system and the world coordinate system. Quaternion updates are performed using: in, Let be the attitude quaternion at the current moment. The angular velocity vector of the gyroscope. The sampling period is This is quaternion multiplication. This is an exponential mapping from vector to quaternion; the update integrates the angular velocity of the inertial measurement unit as a rotational increment and maintains unit norm stability.

[0035] To compensate for the drift difference between inertial measurement unit (IMU) and visual estimation, an extended Kalman filter framework is introduced for joint attitude and bias estimation. State transition and observation are performed using the following methods: in, It is a state vector containing attitude quaternions, gyroscope bias, and accelerator bias. For inertial input, For visual observations from head keypoints and PnP constraints, The linearized Jacobian matrix is... To account for process noise and observation noise, smooth and drift-resistant head attitude parameters are obtained through joint updates.

[0036] To address the misalignment of multi-source timestamps, a sub-frame time realignment interpolation is constructed to time-align each eye-tracking image with the nearest inertial measurement unit (IMU) attitude solution. Samples with alignment errors less than a threshold are assigned high confidence weights, while samples with larger errors are corrected through spline interpolation, forming frame-by-frame paired eye-tracking image frames and head attitude parameter pairs.

[0037] Image quality metrics, including motion blur index, blink occlusion ratio, and reflected specular coverage, are calculated within each binocular region of interest in each frame to construct quality weights. in, As the quality weight of the current frame, For the normalized value of the fuzzy index, For occlusion ratio, For high light coverage, These are the weighting coefficients. The weight is a Sigmoid function; this weight is used for confidence fusion in the subsequent trajectory recognition stage.

[0038] World coordinate system normalization is performed on the head pose parameters, and the pose solutions of all devices are stitched together into a unified reference system through pre-calibrated extrinsic parameters to ensure geometric consistency across devices; simultaneously, maintenance is performed in a distributed buffer. Timestamp, eye-tracking frame, head pose parameters, quality weights An ordered queue ensures that subsequent eye trajectory recognition directly consumes structured and synchronized data.

[0039] S120 performs eye trajectory recognition on eye movement images, and combines head posture data to output the gaze direction angle, gaze point coordinates and gaze duration, forming an eye trajectory feature sequence.

[0040] In one possible implementation, the step of performing eye trajectory recognition on the eye movement image and combining the head posture data to output the gaze direction angle, fixation point coordinates, and gaze duration to form an eye trajectory feature sequence specifically includes: performing pupil center localization and eyelid detection on the eye movement image and head posture parameters, and outputting the pupil center pixel coordinates; using the pupil center pixel coordinates and the camera intrinsic parameter matrix as input, calculating a first unit gaze vector in the camera coordinate system, and combining the head posture parameters to map the first unit gaze vector to the head coordinate system, outputting a second unit gaze vector in the head coordinate system; and based on the second unit gaze vector... The optical axis is analyzed using the gaze vector and the eye's geometric model, and corrected to a visual axis based on an individualized Kappa angle, resulting in a unit direction vector of the visual axis in the world coordinate system. Using the unit direction vector of the visual axis and the device screen plane equation as input, the intersection of the visual axis and the screen plane is solved and mapped to the gaze point coordinates in the screen coordinate system. The angular velocity between adjacent frames is calculated based on the time series of the unit direction vector of the visual axis, and gaze and saccade segmentation is performed using a preset angular velocity threshold to obtain the corresponding gaze duration. Based on the gaze direction angle, the gaze point coordinates, and the gaze duration, an eye trajectory feature sequence is constructed and output.

[0041] Specifically, under the common timescale of eye-tracking images and head pose parameters, facial keypoint detection and affine correction are first performed on the original image to stably crop the binocular regions of interest. High-pass enhancement and guided filtering are then used to suppress low-contrast and weak-texture interference. Subsequently, within each binocular region of interest, Canny edge extraction and specular mask removal are performed to obtain a set of iris-pupil edge candidate points. Robust circle-ellipse hybrid fitting is used to locate the pupil center, obtaining the pupil center pixel coordinates and eyelid opening / closing state, where the eyelid opening / closing state is used for subsequent confidence weighting. The circle fitting uses the least squares method to solve for the pupil circle parameters. The objective function and closed-form update are as follows: in, Represents the pixel coordinates of the pupil center. Represents the fitted radius. Indicates the first The pixel coordinates of the edge points This indicates the number of edge points; this fitting, combined with RANSAC point set in-point filtering when occlusion exists, improves the robustness of the pupil center pixel coordinates.

[0042] After obtaining the pixel coordinates of the pupil center, distortion correction mapping is performed by combining the camera intrinsic matrix and distortion parameters. The homogeneous pixel coordinates are then back-projected into the first unit gaze vector in the camera coordinate system. This vector is then rigidly transformed to the head coordinate system based on the head pose parameters to obtain the second unit gaze vector. The back-projection and coordinate transformation are performed using the following formula: in, This represents the first unit line-of-sight vector in the camera coordinate system. Represents the camera intrinsic parameter matrix. Represents the pixel coordinates of the pupil center. Represents the L2 norm, This represents the second unit gaze vector in the head coordinate system. This represents the rotation matrix from the camera coordinate system to the head coordinate system; the orientation is expressed purely through normalization, and the gaze direction is aligned with the head pose parameters through rigid body rotation.

[0043] Within the head coordinate system, the second unit gaze vector is combined with the eyeball geometry model to first construct the optical axis, then obtain the visual axis through individualized Kappa angle rotation correction, and finally map it to the world coordinate system to obtain the unit direction vector of the visual axis in the world coordinate system. The optical axis is determined by the line connecting the center of the eyeball and the three-dimensional point of the pupil. The three-dimensional point of the pupil is obtained by offsetting along the gaze direction with a calibrated depth. Subsequently, a small Kappa angle rotation matrix is ​​applied and the world coordinate mapping is completed. in, This indicates the position of the camera's optical center in the head coordinate system. This indicates the pupil depth bias obtained from individual calibration. This represents the second unit line-of-sight vector. Indicates the three-dimensional position of the pupil. Indicates the location of the center of the eyeball. Represents the unit vector of the optical axis. This represents a small-angle rotation matrix constructed from horizontal and vertical Kappa angles. Represents the unit vector of the visual axis. This represents the rotation matrix from the head coordinate system to the world coordinate system. It represents the unit direction vector of the visual axis in the world coordinate system; by compensating for the systematic deviation between the optical axis and the visual axis through individualized Kappa angles, it ensures the accuracy of subsequent spatial intersection calculations.

[0044] In the world coordinate system, let the visual axis pass through the camera optical center in the world coordinate system and intersect with the device screen plane to obtain the three-dimensional intersection point. Then, obtain the gaze point coordinates in the screen coordinate system through screen homography mapping. Ray-plane intersection and two-dimensional mapping are calculated by the following formula: in, Indicates the position of the camera's optical center in the world coordinate system. This represents the unit direction vector of the visual axis in the world coordinate system. Indicates equipment The screen normal vector, Represents a known point on the screen plane. Represents the ray parameter solution. This represents the three-dimensional intersection point of the ray and the screen plane. The homography matrix representing the world-to-screen coordinates. It represents the coordinates of the gaze point in the screen coordinate system; it avoids numerical iteration instability by using analytical intersection and strictly places the spatial gaze point on the screen through homography mapping.

[0045] On a continuous visual axis sequence, angular velocity measurements between adjacent frames are constructed, and gaze and saccade segmentation is performed based on a preset angular velocity threshold. Simultaneously, the gaze duration is obtained by accumulating gaze frames within a sliding time window. The angular velocity measurement and time accumulation are implemented using the following formula: in, Indicates the angular velocity between the visual axes of adjacent frames. and This represents the unit vector of the visual axis between two adjacent frames. Indicates the time step between frames. Indicates the length of the sliding window. This indicates the preset angular velocity threshold. Indicates an indicator function, This represents the estimated gaze duration within the window; the inverse cosine of the vector inner product provides a scale-invariant measure of direction change, and threshold segmentation labels low-speed stable segments as gaze segments.

[0046] In each frame, the gaze direction angle is extracted from the visual axis in the world coordinate system. The gaze point coordinates in the screen coordinate system and the gaze duration within the sliding window constitute the in-frame coupling feature, which is then sequentially concatenated along the time axis to form an eye trajectory feature sequence. The direction angle is expressed using spherical coordinates, specifically: in, Indicates the polar angle (related to pitch). Indicates azimuth. express The three components, This represents the coordinates of the gaze point in the screen coordinate system. This indicates an estimate of the current gaze duration. This represents a single-frame eye trajectory feature vector; the ordered superposition of this feature vector over time constitutes an eye trajectory feature sequence, which directly supports subsequent 3D gaze prediction and gaze intent field generation.

[0047] S130 combines camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices to construct a 3D gaze prediction model based on eye trajectory feature sequences, in order to predict the target screen or interface element that the user is looking at in a multi-device collaborative scenario.

[0048] In one possible implementation, the step of constructing a 3D gaze prediction model based on the eye trajectory feature sequence by combining camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices specifically includes: establishing a set of gaze rays in the world coordinate system by combining the eye trajectory feature sequence and the camera pose parameters of each device; performing intersection calculations between the gaze rays and the screen plane according to the gaze ray set and the screen layout geometric relationship parameters of each device to obtain candidate gaze points; calculating the geometric consistency error based on the candidate gaze points and the corresponding screen boundaries and converting it into device-level likelihood data; constructing a cross-device transfer prior and performing probability recursion on the device-level likelihood data and the field-of-view overlap region parameters between each device, outputting the posterior probability distribution of each device; and further calculating the interface element-level posterior probability using the posterior probability distribution and the geometric description of the internal interface elements of the device as input to obtain the prediction result for the target screen or the interface element.

[0049] Specifically, when combining the eye trajectory feature sequence with the camera pose parameters of various devices, it is necessary to map the camera's extrinsic parameters (rotation matrix and translation vector) to a unified world coordinate system, and pair the visual axis unit direction vector in the world coordinate system of the trajectory features with the camera's optical center position to construct a set of gaze rays. Each gaze ray is represented as:

[0050] in, This indicates the position of the camera's optical center in the world coordinate system. Represents the unit direction vector of the visual axis. These are parameter variables. The gaze ray set records the geometric extension of the user's gaze in space at different times, ensuring the basis for subsequent calculations to find the intersection with the screen plane.

[0051] When the gaze ray set corresponds to the screen layout geometry parameters of each device, the plane equation of each screen is used: The intersection point of the ray and the plane is determined analytically. The formula for the intersection point is:

[0052] in, Represents the screen normal vector. This represents a reference point on the screen plane. This represents the intersection point of the ray and the plane. Obtaining candidate fove points provides a necessary condition for subsequent geometric consistency checks.

[0053] When a candidate gaze point matches the corresponding screen boundary, it is necessary to determine whether the gaze point is located within the effective display area of ​​the screen. Within this range, the minimum distance between the gaze point and the screen boundary is calculated as the geometric consistency error: Geometric consistency error is converted into device-level likelihood data using a Gaussian kernel: in, The homography matrix representing world-to-screen coordinates. Indicates the screen boundary. This represents the out-of-bounds cutoff constant. For noise scale, This is the function for determining interior points.

[0054] When fusing device-level likelihood data with field-of-view overlap parameters, the impact of inter-device field-of-view intersection on cross-device transfer needs to be considered. The inter-device field-of-view overlap region is defined as:

[0055] in, Indicates equipment The visible area The fixed angle is defined as the unit spherical angle. Based on this overlap weight, a transition prior is constructed:

[0056] Then, the posterior probability distribution of the device is calculated recursively: in, These are the prior weights. This recursive process ensures that gaze prediction between devices has temporal continuity and physical accessibility.

[0057] When combining the posterior probability distribution of the device with the geometric description of the device's internal interface elements, the calculation is further refined to the interface element level. The likelihood calculation formula for interface elements is:

[0058] After normalization, the posterior probability of the interface elements is obtained: in, Indicates the position of the gaze point in screen coordinates. Indicates the center position of the interface element. Indicates the salience weight of interface elements. This represents the scale parameter of the interface element. By outputting the prediction results for the target screen or interface element in this way, accurate modeling of gaze intent across devices can be achieved.

[0059] S140 generates a gaze intention field based on the prediction results output by the 3D gaze prediction model.

[0060] In one possible implementation, generating a gaze intent field based on the prediction results output by the 3D gaze prediction model specifically includes: jointly representing the device posterior probability distribution of the target screen with the posterior probability of the interface elements to form a multi-level gaze probability set; mapping the discretized gaze probabilities to a potential gaze distribution in continuous space based on the spatial location of the multi-level gaze probability set and candidate gaze points, combined with the screen layout geometry of each device; performing Gaussian kernel expansion on the potential gaze distribution to form a smooth gaze potential in the spatial range; and fusing the gaze probability distribution of the device layer and the gaze probability distribution of the interface element layer through a weighted superposition method to generate a gaze intent field describing the user's gaze trend.

[0061] Specifically, when jointly representing the device posterior probability distribution of the target screen and the posterior probabilities of interface elements, it is necessary to first represent the posterior probability distribution at the device layer. Posterior probability distribution of interface element layer A hierarchical combination is performed. The device-level posterior probability distribution represents the user's gaze tendency across multiple devices, while the interface element-level posterior probability distribution represents the user's gaze tendency among different interface elements within the same device. The joint representation can be constructed as a multi-level gaze probability set. This set maintains a global gaze distribution at the device level and a local fine distribution at the interface element level, thus taking into account both cross-device gaze and intra-device element gaze.

[0062] When generating a continuous spatial latent gaze distribution based on a multi-level gaze probability set and the spatial locations of candidate gaze points, it is necessary to map the discrete probability information to spatial points in the world coordinate system. Candidate gaze points are formed by the intersections of rays and the screen plane, denoted as […]. By combining the screen layout geometric parameters of each device (including screen plane equations, position, and orientation), each candidate gaze point is mapped to its corresponding posterior probability, resulting in a discretized spatial distribution. This mapping process can be represented as a latent gaze distribution function:

[0063] in, For the potential gaze distribution, Let be the Dirac function, representing the probability mass of placing the corresponding weight at the candidate fixation point location.

[0064] When performing Gaussian kernel expansion on the latent gaze distribution, the originally discretized distribution needs to be transformed into a continuous and smooth latent gaze field. This is achieved by applying a Gaussian kernel function around each candidate gaze point location, thus enabling local diffusion and smoothing of the probabilities. The expansion formula is as follows:

[0065] in, This represents the smoothed potential gaze distribution. The mean is Covariance is Gaussian kernel function, Indicates candidate fixation points, Controlling the diffusion range. This operation creates a continuous gaze potential distribution across the spatial range, effectively mitigating the jitter caused by discrete prediction points.

[0066] When fusing the device-level gaze probability distribution with the interface element-level gaze probability distribution, a weighted superposition method is used to ensure that gaze probabilities at different levels are uniformly represented in the gaze intent field. The fusion formula is as follows:

[0067] in, Indicates the field of intent to gaze. Indicates the center position of the device screen. Indicates the device layer smoothing parameters. Indicates the center position of the interface element. This represents the smoothing parameters for the UI element layer. This represents the fusion weight between the device layer and the interface element layer. The resulting gaze intent field not only reflects the user's global gaze trends across multiple devices, but also characterizes the local gaze hotspots of specific interface elements within a device, providing a continuous and interpretable spatial potential basis for subsequent cross-device task scheduling and interface interaction.

[0068] The S150 uses the gaze intent field as the basis for interaction scheduling, and dynamically schedules cross-device task resources under the HarmonyOS distributed architecture to achieve content projection, task continuation and interface synchronization between different devices.

[0069] In one possible implementation, the step of using the gaze intent field as the basis for interaction scheduling, dynamically scheduling cross-device task resources under the HarmonyOS distributed architecture to achieve content projection, task continuation, and interface synchronization between different devices, specifically includes: integrating the gaze intent field within the effective display area of ​​each device screen to obtain the corresponding device-level gaze potential energy and performing normalization processing to form a device-level gaze potential energy distribution; generating a device-level scheduling score based on the device-level gaze potential energy distribution, combined with device relevance weights, task affinity values, and migration costs; constructing a cross-device resource allocation model based on the device-level scheduling score, completing the mapping between tasks and devices under computational resource constraints and bandwidth constraints, and outputting task allocation results; generating a distributed focus token and execution instructions according to the task allocation results, wherein the distributed focus token includes a focus device identifier, a focus establishment timestamp, a task identifier, and a task status summary; after the distributed focus token is issued, driving the execution link of content projection, task continuation, and interface synchronization, and maintaining the stability of the focus device through a time smoothing mechanism and advantageous interval constraints.

[0070] Specifically, given that the gaze intent field has been determined, spatial integration and intensity accumulation are performed on the effective display area of ​​each device's screen to obtain device-level gaze potential energy, which is then normalized across all devices to form a device-level gaze potential energy distribution. The device-level gaze potential energy distribution has a one-to-one correspondence with the effective display area of ​​the screen, which is jointly defined by the screen's geometric boundaries and the occlusion conditions of the visible field. The device-level gaze potential energy distribution serves as the original metric for subsequent decisions, providing a unified benchmark for the relative priority between different devices, and is version-bound by timestamps and device identifiers to ensure cross-device consistency.

[0071] After the device-level gaze potential distribution is available, a comprehensive scoring framework is formed by introducing device relevance weights, task affinity values, and migration costs. Device relevance weights reflect the prior advantages brought by the previous focus device, the device where the current task is located, and spatially neighboring devices. Task affinity values ​​reflect the degree of matching between the task to be scheduled and the device's functions, decoding capabilities, and input methods. Migration costs characterize the comprehensive costs of state packaging, network transmission, content reconstruction, and user-perceived latency. The three factors, together with the device-level gaze potential distribution, generate a device-level scheduling score. The device-level scheduling score then enters a stability screening stage to suppress transient fluctuations, thereby obtaining a stable scoring sequence that can be used for resource allocation and focus arbitration.

[0072] After the device-level scheduling score stabilizes, a cross-device resource allocation model is constructed, mapping the set of tasks to be scheduled and the set of candidate devices into a binary allocation relationship, while simultaneously applying computational resource constraints and bandwidth constraints. The computational resource constraints cover the upper limit of computing power, memory usage, and concurrent thread capacity, while the bandwidth constraints cover link bandwidth, end-to-end round-trip latency, and jitter tolerance range. The resource allocation model maximizes overall utility and minimizes migration cost while satisfying the constraints, and outputs the task allocation results. The task allocation results include the host device, startup order, dependency graph, and fallback point for each task, and are accompanied by a consistent version number to ensure the atomicity of subsequent switching processes.

[0073] After the task allocation result is determined, a distributed focus token and execution instructions are generated. The distributed focus token contains the focus device identifier, focus establishment timestamp, task identifier, and task status summary, serving as the sole credential for cross-device arbitration and consistent commit. The execution instructions are encapsulated for three types of links: the content projection link specifies the content retrieval and rendering path, the task continuation link carries the execution context and recovery point, and the interface synchronization link carries the differential snapshot and order marker. The three types of execution instructions are broadcast on the distributed soft bus and are validated and executed sequentially by the subscribing devices based on the distributed focus token, ensuring that the received actions are traceable and rollbackable.

[0074] After the distributed focus token is issued and begins execution, the execution chain that drives content projection, task continuation, and interface synchronization enters the running state. At the same time, a time smoothing mechanism and a dominant interval constraint are enabled to maintain the stability of the focus device. The time smoothing mechanism performs exponential smoothing on the device-level scheduling score to filter short-term noise, and the dominant interval constraint requires that the current best device and the second best device maintain the minimum confidence gap to avoid frequent switching. Together with failure retry, acknowledgment threshold, and state renewal, these two mechanisms form a stable control closed loop, which keeps the focus device locked when the user's gaze intent is continuously concentrated and achieves smooth switching when the gaze intent shifts significantly and resource conditions are met, thereby ensuring the continuity, reliability, and low perceived latency of the cross-device interaction chain.

[0075] In one possible implementation, using the gaze intent field as the basis for interaction scheduling, and dynamically scheduling cross-device task resources under the HarmonyOS distributed architecture to achieve content projection, task continuation, and interface synchronization between different devices, specifically includes: calculating the smooth gaze intensity of each device within a continuous time window based on the gaze intent field; calculating the corresponding expected gaze duration based on the smooth gaze intensity; combining the expected gaze duration with head movement stability parameters and device relevance weights to output an adaptive preset threshold; performing cross-device comparison using the expected gaze duration and the adaptive preset threshold as input to output candidate focus devices; outputting a stable focus device identifier based on the candidate focus devices through a finite state machine of focus holding and releasing; and generating and publishing a distributed focus token using the focus device identifier as input to drive content projection, task continuation, and interface synchronization, thereby setting the first device as the focus device and maintaining the stability of the cross-device interaction link when it is detected that the user's gaze duration on the first device exceeds the adaptive preset threshold.

[0076] Specifically, when calculating the smoothed gaze intensity of each device within a continuous time window based on the gaze intent field, the gaze probability density of each device's screen area is first extracted from the gaze intent field and then weighted and accumulated within the continuous time window. The time window is constructed using a sliding window mechanism to ensure that short-term jitter does not directly affect the results. Subsequently, exponential smoothing or weighted averaging methods are used to smooth the accumulated values ​​to obtain the smoothed gaze intensity of each device. The smoothed gaze intensity reflects the stable gaze trend of the user within a certain time range, avoiding excessively frequent focus switching due to instantaneous eye movements.

[0077] When calculating the expected gaze duration based on smoothed gaze intensity, the smoothed gaze intensity is combined with the time axis. Segments below a threshold are suppressed, while segments consistently above the threshold are accumulated. The accumulation method uses frame-by-frame integration, that is, the effective frame duration corresponding to the gaze intensity is progressively accumulated within the time window to obtain the expected gaze duration. The expected gaze duration not only characterizes the user's concentration on a particular device but also quantifies the stability and persistence of the gaze, providing a quantitative basis for subsequent threshold determination and focus device confirmation.

[0078] When combining the expected gaze duration, head movement stability parameters, and device relevance weights to output an adaptive preset threshold, adjustments need to be made for stability differences across different scenarios. Head movement stability parameters, calculated using gyroscopes and accelerometers, reflect whether the user's head is moving significantly. The threshold is increased when the movement intensity is high to avoid erroneous switching due to ineffective eye movements. Device relevance weights reflect the closeness of the device to the current task; for example, the currently focused device, the device where the task is located, and spatially nearby devices receive higher weights, thus reducing the threshold required for switching. Finally, by combining these factors, a dynamically changing adaptive preset threshold is output, enabling the decision-making process to adapt to different users and different scenarios.

[0079] When comparing the expected gaze duration with an adaptive preset threshold, a cross-device comparison logic is established. Specifically, the expected gaze duration of all candidate devices is compared with their corresponding adaptive preset thresholds, and the device that meets the threshold conditions and has the greatest relative advantage is selected as the candidate focus device. Relative advantage is measured by the difference between the best and second-best devices, ensuring that handover only occurs when the difference is significant, thereby avoiding frequent focus oscillations caused by similar potential energy between devices. At this stage, the candidate focus device does not directly become the focus but enters the subsequent stability assessment stage.

[0080] When a candidate focus device passes through a finite state machine for focus holding and release, entry and exit conditions are set to correspond to the locked and released states, respectively. The entry condition requires the gaze duration of the candidate focus device to continuously exceed a threshold and remain for a certain time window, thus locking it as the focus device; the exit condition requires the gaze intensity to drop below the release threshold and remain there for a certain period of time, ensuring that the focus does not switch immediately due to momentary loss. The finite state machine consists of state nodes and transition conditions, which can guarantee the stability and anti-jitter properties of the focus device in the time dimension, and finally output a stable focus device identifier.

[0081] When generating and publishing distributed focus tokens, a stable focus device identifier is encapsulated as a unique credential, along with a focus establishment timestamp, task identifier, and task status summary. Distributed focus tokens are published via the HarmonyOS distributed soft bus, and all devices subscribe to them and perform subsequent operations based on the token. The content projection link transfers video or text streams to the focus device based on the focus device identifier, the task continuation link migrates the current computing task based on the task identifier, and the interface synchronization link maintains interface consistency based on the task status summary. Through the uniqueness and timing attributes of the distributed focus tokens, the continuity and consistency of cross-device interaction are guaranteed. This ensures that when the user's gaze duration on the first device exceeds an adaptive preset threshold, that device is set as the focus device, maintaining the stability of the cross-device interaction link.

[0082] In one possible implementation, after setting the first device as the focus device, the method further includes: when the user's eye trajectory is detected to have shifted from the first device to the second device, triggering a cross-device flow of tasks between the first device and the second device, and performing synchronous updates at the interface element level to form an eye-tracking driven cross-device interaction link.

[0083] Specifically, when the user's eye trajectory is detected to shift from the first device to the second device, the runtime monitoring module sets a time smoothing window and a dominant interval threshold based on the joint changes of the device-level posterior probability sequence and the interface element-level posterior probability sequence, and simultaneously determines the persistence of "leaving the first device" and the significance of "entering the second device". When the stable gaze cues of the first device fall below the release threshold and the stable gaze cues of the second device cross the lock threshold, and the minimum dwell time and minimum angular velocity constraints are met within the cross-device transfer window, it is determined that a cross-device gaze migration has occurred. The monitoring module also verifies whether the network latency, bandwidth margin, and available computing power of the second device meet the task acceptance baseline. If they do, it submits a transfer suggestion to the arbitration module, triggering the master-slave switch preparation process of the distributed focus token.

[0084] When a task is triggered to flow across devices between the first and second devices and synchronized at the interface element level, the arbitration module first serializes and packages the current task state of the first device, generating a task continuation descriptor containing the execution context, dependency graph, media location, and rollback point, and then revokes the old distributed focus token and issues a new one. After obtaining the task continuation descriptor, the second device performs readiness verification and replays the execution context, completes the decoding and reconstruction of content projection and rendering preheating, and simultaneously pulls a synchronized snapshot of the interface and performs differential merging according to the topological order of the interface element tree. It uses vector clocks or CRDT strategies to resolve concurrent conflicts and ensures consistency of interface elements in terms of hierarchy, attributes, and event binding. After receiving confirmation of the new distributed focus token, the first device enters a degraded state, retaining only a lightweight shadow session to support rollback until the second device completes the takeover and returns a consistency confirmation. If a decrease in bandwidth, an increase in packet loss, or a failure in state verification is detected at any stage, the task and interface are restored on the first device according to the rollback point, ensuring smooth migration, low interruption latency, and reversible switching of the cross-device interaction link under the user's line of sight.

[0085] This embodiment also discloses an eye-tracking cross-device interaction device, referring to... Figure 2 The device includes an acquisition module 201, a processing module 202, and an output module 203. It is used to execute any of the eye-tracking cross-device interaction methods described above, wherein: The acquisition module 201 is used to collect the user's eye movement images and head posture data.

[0086] The processing module 202 is used to perform eye trajectory recognition on eye movement images, and combine head posture data to output the gaze direction angle, gaze point coordinates and gaze duration to form an eye trajectory feature sequence.

[0087] The processing module 202 is used to combine camera pose parameters, field-of-view overlap region parameters and screen layout geometric relationship parameters from multiple devices to construct a three-dimensional gaze prediction model based on eye trajectory feature sequences, so as to predict the target screen or interface element that the user is looking at in a multi-device collaborative scenario.

[0088] Processing module 202 is used to generate a gaze intent field based on the prediction results output by the 3D gaze prediction model.

[0089] The output module 203 is used to use the gaze intent field as the basis for interaction scheduling, and dynamically schedule cross-device task resources under the HarmonyOS distributed architecture to realize content projection, task continuation and interface synchronization between different devices.

[0090] In one possible implementation, the processing module 202 is used to calculate the smooth gaze intensity of each device within a continuous time window based on the gaze intention field.

[0091] Processing module 202 is used to calculate the corresponding expected gaze duration based on the smooth gaze intensity.

[0092] The processing module 202 is used to combine the expected gaze duration with head movement stability parameters and device-related weights to output an adaptive preset threshold.

[0093] The output module 203 is used to perform cross-device comparisons and output candidate focus devices by taking the desired gaze duration and an adaptive preset threshold as inputs.

[0094] Output module 203 is used to output a stable focus device identifier based on the finite state machine of the candidate focus device after focus holding and release.

[0095] The processing module 202 is used to generate and publish a distributed focus token with the focus device identifier as input, drive content projection, task continuation and interface synchronization, so that when it is detected that the user's gaze duration on the first device exceeds the adaptive preset threshold, the first device is set as the focus device and the stability of the cross-device interaction link is maintained.

[0096] In one possible implementation, the output module 203 is used to trigger a cross-device flow of tasks between the first device and the second device when the user's eye trajectory is detected to have shifted from the first device to the second device, and to perform synchronous updates at the interface element level to form an eye-tracking-driven cross-device interaction link.

[0097] In one possible implementation, the output module 203 is used to perform pupil center localization and eyelid detection on the eye-tracking image and head posture parameters, and output the pupil center pixel coordinates.

[0098] The output module 203 is used to calculate the first unit gaze vector in the camera coordinate system by taking the pupil center pixel coordinates and the camera intrinsic parameter matrix as input, and to map the first unit gaze vector to the head coordinate system by combining the head pose parameters, and output the second unit gaze vector in the head coordinate system.

[0099] The processing module 202 is used to analyze and output the optical axis based on the second unit line-of-sight vector and the eye geometry model, and correct it to the visual axis based on the individualized Kappa angle to obtain the visual axis unit direction vector in the world coordinate system.

[0100] The processing module 202 is used to take the unit direction vector of the visual axis and the equation of the device screen plane as input, solve for the intersection of the visual axis and the screen plane, and map it into the gaze point coordinates in the screen coordinate system.

[0101] The processing module 202 is used to calculate the angular velocity between adjacent frames based on the time series of the visual axis unit direction vector, and perform gaze and saccade segmentation in combination with a preset angular velocity threshold to obtain the corresponding gaze duration.

[0102] The processing module 202 is used to construct and output an eye trajectory feature sequence based on the gaze direction angle, gaze point coordinates and gaze duration.

[0103] In one possible implementation, the processing module 202 is used to combine the eye trajectory feature sequence with the camera pose parameters of each device to establish a set of gaze rays in the world coordinate system.

[0104] The processing module 202 is used to perform intersection calculation between the ray and the screen plane based on the geometric relationship parameters between the gaze ray set and the screen layout of each device to obtain candidate gaze points.

[0105] The processing module 202 is used to calculate the geometric consistency error based on the candidate gaze point and the corresponding screen boundary and convert it into device-level likelihood data.

[0106] The processing module 202 is used to construct cross-device transfer priors and perform probability recursion on the device-level likelihood data and the field-of-view overlap region parameters between various devices, and output the posterior probability distribution of each device.

[0107] The processing module 202 is used as input the posterior probability distribution and the geometric description of the internal interface elements of the device to further calculate the posterior probability at the interface element level and obtain the prediction result for the target screen or interface element.

[0108] In one possible implementation, the processing module 202 is used to jointly represent the device posterior probability distribution of the target screen and the posterior probability of the interface elements to form a multi-level gaze probability set.

[0109] The processing module 202 is used to map the discretized gaze probability into a potential gaze distribution in continuous space based on the multi-level gaze probability set and the spatial location of the candidate gaze points, combined with the screen layout geometry of each device.

[0110] Processing module 202 is used to perform Gaussian kernel expansion operation on the potential gaze distribution so that the potential gaze distribution forms a smooth gaze potential in the spatial range.

[0111] The processing module 202 is used to fuse the gaze probability distribution of the device layer and the gaze probability distribution of the interface element layer through a weighted superposition method to generate a gaze intent field that describes the user's gaze trend.

[0112] In one possible implementation, the processing module 202 is used to integrate the gaze intention field within the effective display area of ​​each device screen to obtain the corresponding device-level gaze potential energy and perform normalization processing to form a device-level gaze potential energy distribution.

[0113] Processing module 202 is used to generate a device-level scheduling score based on the device-level gaze potential distribution, combined with device relevance weights, task affinity values, and migration costs.

[0114] The processing module 202 is used to construct a cross-device resource allocation model based on device-level scheduling scores, complete the mapping between tasks and devices under computing resource constraints and bandwidth constraints, and output the task allocation results.

[0115] The processing module 202 is used to generate a distributed focus token and execution instructions based on the task allocation results. The distributed focus token includes a focus device identifier, a focus establishment timestamp, a task identifier, and a task status summary.

[0116] The processing module 202 is used to drive the execution chain of content projection, task continuation and interface synchronization after the distributed focus token is issued, and maintains the stability of the focus device through time smoothing mechanism and advantageous interval constraint.

[0117] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0118] This embodiment also discloses an electronic device, as shown in the reference. Figure 3 The electronic device may include: at least one processor 301, at least one communication bus 302, user interface 303, network interface 304, and at least one memory 305.

[0119] The communication bus 302 is used to enable communication between these components.

[0120] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0121] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0122] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 305, and by calling data stored in memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications. The GPU is responsible for rendering and drawing the content required for display. The modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0123] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described above, etc. The data storage area may store data involved in the various method embodiments described above. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. As a computer storage medium, the memory 305 may include an operating system, a network communication module, a user interface 303 module, and an application for an eye-tracking cross-device interaction method.

[0124] exist Figure 3 In the illustrated electronic device, the user interface 303 is primarily used to provide an input interface for the user and to acquire user input data. The processor 301 can be used to call an application stored in the memory 305 that represents an eye-tracking cross-device interaction method. When executed by one or more processors 301, this causes the electronic device to perform one or more methods as described in the above embodiments.

[0125] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0126] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0127] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 305 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned memory 305 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.

[0131] The present invention also discloses a computer-readable storage medium storing instructions. When executed by one or more processors 301, these instructions cause an electronic device to perform one or more methods as described in the above embodiments.

[0132] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the specification and the disclosure of practical truths. This invention is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. An eye-tracking cross-device interaction method, characterized in that, The method includes: Collect user's eye-tracking images and head posture data; The eye movement images are used to identify eye trajectory, and the head posture data is combined to output the gaze direction angle, gaze point coordinates and gaze duration to form an eye trajectory feature sequence. By combining camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices, a three-dimensional gaze prediction model is constructed based on the eye trajectory feature sequence to predict the target screen or interface element that the user is looking at in a multi-device collaborative scenario. A gaze intent field is generated based on the prediction results output by the three-dimensional gaze prediction model; Using the gaze intent field as the basis for interaction scheduling, cross-device task resources are dynamically scheduled under the HarmonyOS distributed architecture to achieve content projection, task continuation and interface synchronization between different devices.

2. The eye-tracking cross-device interaction method according to claim 1, characterized in that, The method of using the gaze intent field as the basis for interaction scheduling, and dynamically scheduling cross-device task resources under the HarmonyOS distributed architecture to achieve content projection, task continuation, and interface synchronization between different devices, specifically also includes: The smooth gaze intensity of each device within a continuous time window is calculated based on the gaze intention field; Calculate the corresponding expected gaze duration based on the smoothed gaze intensity; By combining the desired gaze duration with head movement stability parameters and device-related weights, an adaptive preset threshold is output. Using the desired gaze duration and the adaptive preset threshold as input, perform cross-device comparisons and output candidate focus devices; The candidate focus device outputs a stable focus device identifier based on the finite state machine that performs focus holding and release; Using the focus device identifier as input, a distributed focus token is generated and published to drive content projection, task continuation, and interface synchronization. Thus, when the user's gaze duration on the first device exceeds the adaptive preset threshold, the first device is set as the focus device, and the stability of the cross-device interaction link is maintained.

3. The eye-tracking cross-device interaction method according to claim 2, characterized in that, After setting the first device as the focus device, the method further includes: When the user's eye movement is detected to shift from the first device to the second device, a task is triggered to flow across the devices between the first and second devices, and a synchronous update is performed at the interface element level to form an eye-tracking-driven cross-device interaction link.

4. The eye-tracking cross-device interaction method according to claim 1, characterized in that, The step of performing eye trajectory recognition on the eye movement images, and combining the head posture data to output the gaze direction angle, fixation point coordinates, and gaze duration, forms an eye trajectory feature sequence, specifically including: Perform pupil center localization and eyelid detection on the eye movement image and head posture parameters, and output the pixel coordinates of the pupil center; Using the pupil center pixel coordinates and the camera intrinsic parameter matrix as input, calculate the first unit gaze vector in the camera coordinate system, and combine the head pose parameters to map the first unit gaze vector to the head coordinate system, and output the second unit gaze vector in the head coordinate system. The optical axis is output based on the second unit line-of-sight vector and the eye geometry model, and corrected to the visual axis based on the individualized Kappa angle, thus obtaining the unit direction vector of the visual axis in the world coordinate system. Using the unit direction vector of the visual axis and the equation of the device screen plane as input, the intersection of the visual axis and the screen plane is solved and mapped to the gaze point coordinates in the screen coordinate system; The angular velocity between adjacent frames is calculated based on the time series of the unit direction vector of the visual axis, and gaze and saccade segmentation is performed in combination with a preset angular velocity threshold to obtain the corresponding gaze duration. Based on the gaze direction angle, the coordinates of the gaze point, and the gaze duration, an eye trajectory feature sequence is constructed and output.

5. The eye-tracking cross-device interaction method according to claim 1, characterized in that, The method of constructing a 3D gaze prediction model based on the eye trajectory feature sequence by combining camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices specifically includes: By combining the eye trajectory feature sequence with the camera pose parameters of each device, a set of gaze rays in the world coordinate system is established; Based on the geometric relationship parameters between the set of gaze rays and the screen layout of each device, the intersection of the rays and the screen plane is calculated to obtain candidate gaze points. Based on the candidate gaze points and the corresponding screen boundaries, the geometric consistency error is calculated and converted into device-level likelihood data; For the device-level likelihood data and the field-of-view overlap region parameters between each device, construct a cross-device transfer prior and perform probability recursion to output the posterior probability distribution of each device; Using the posterior probability distribution and the geometric description of the internal interface elements of the device as input, the posterior probability at the interface element level is further calculated to obtain the prediction result for the target screen or the interface element.

6. The eye-tracking cross-device interaction method according to claim 1, characterized in that, The generation of the gaze intent field based on the prediction results output by the three-dimensional gaze prediction model specifically includes: The device posterior probability distribution of the target screen and the posterior probability of the interface elements are jointly represented to form a multi-level gaze probability set. Based on the multi-level gaze probability set and the spatial location of the candidate gaze points, combined with the screen layout geometry of each device, the discretized gaze probability is mapped to the potential gaze distribution in continuous space. A Gaussian kernel expansion operation is performed on the potential gaze distribution to form a smooth gaze potential in the spatial range; The gaze probability distribution at the device layer and the gaze probability distribution at the interface element layer are fused together by weighted superposition to generate a gaze intent field that describes the user's gaze trend.

7. The eye-tracking cross-device interaction method according to claim 1, characterized in that, The method of using the gaze intent field as the basis for interaction scheduling, and dynamically scheduling cross-device task resources under the HarmonyOS distributed architecture to achieve content projection, task continuation, and interface consistency between different devices, specifically includes: The gaze intention field is integrated over the effective display area of ​​each device screen to obtain the corresponding device-level gaze potential energy, which is then normalized to form a device-level gaze potential energy distribution. Based on the device-level gaze potential distribution, a device-level scheduling score is generated by combining device relevance weights, task affinity values, and migration costs. Based on the device-level scheduling score, a cross-device resource allocation model is constructed to complete the mapping between tasks and devices under computational resource constraints and bandwidth constraints, and the task allocation results are output. Based on the task allocation result, a distributed focus token and execution instruction are generated. The distributed focus token includes a focus device identifier, a focus establishment timestamp, a task identifier, and a task status summary. After the distributed focus token is issued, the execution chain of content projection, task continuation and interface synchronization is driven, and the stability of the focus device is maintained through time smoothing mechanism and advantageous interval constraint.

8. An eye-tracking cross-device interaction device, characterized in that, The device is used to perform an eye-tracking cross-device interaction method as described in any one of claims 1-7, the device comprising an acquisition module, a processing module, and an output module, wherein: The acquisition module is used to collect the user's eye movement images and head posture data; The processing module is used to perform eye trajectory recognition on the eye movement image, and combine the head posture data to output the gaze direction angle, gaze point coordinates and gaze duration to form an eye trajectory feature sequence. The processing module is used to combine camera pose parameters, field-of-view overlap region parameters, and screen layout geometric relationship parameters from multiple devices to construct a three-dimensional gaze prediction model based on the eye trajectory feature sequence, so as to predict the target screen or interface element that the user is looking at in a multi-device collaborative scenario. The processing module is used to generate a gaze intent field based on the prediction results output by the three-dimensional gaze prediction model. The output module is used to use the gaze intent field as the basis for interaction scheduling, and dynamically schedule cross-device task resources under the HarmonyOS distributed architecture to realize content projection, task continuation and interface synchronization between different devices.

9. An electronic device, characterized in that, The device includes a processor, a communication bus, a user interface, a network interface, and a memory. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The communication bus is used to enable communication between the components within the electronic device. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

Images